---
## Optimizations
- **FlashHead LM Head** - lightweight replacement for the dense LM head, significantly improving throughput.
- **vLLM Plugin Integration** - compatible with **vLLM (0.14.0+)** via the [`flash-head`](https://github.com/embedl/flash-head) plugin.
---
## Performance
### Token Generation Speed (RTX 3500 Ada, batch size = 1)
| **Precision** | **Tokens/sec** | **Speedup vs BF16** |
|----------------|----------------|----------------------|
| BF16 baseline | 148 | 1.0× |
| **FlashHead (Embedl)** | **178** | **1.20×** |
| W4A16 baseline | 243 | 1.64x× |
| **FlashHead W4A16 (Embedl)** | **336** | **2.27×** |
FlashHead improves end-to-end speed by **1.38×** over state-of-the-art, while maintaining full accuracy parity.
**Measurement setup:** vLLM 0.10.2, batch_size=1, prompt length=32, max_new_tokens=128, 10 warm-up runs, averaged over 100 runs.
---
## Accuracy (Parity with Baseline)
| **Method** | **MMLU-Pro** | **IFEval** | **BBH** | **TruthfulQA** | **GSM8K** |
|-------------|---------------|--------------|-------------|----------------|--------------|
| **Baseline** | 0.15 | 0.55 | 0.38 | 0.31 | 0.42 |
| **FlashHead** | 0.15 | 0.49 | 0.38 | 0.31 | 0.39 |
FlashHead closely matches baseline accuracy.
---
## Installation
```bash
pip install flash-head
```
The [`flash-head`](https://github.com/embedl/flash-head) vLLM plugin is required. It activates automatically at startup.
---
## Usage Examples
**Note (vLLM context length):** `max_model_len=131072` may fail on GPUs without enough free VRAM for the KV cache. If you see a KV cache memory error, lower `max_model_len` (or increase `gpu_memory_utilization`).
### vLLM Inference
```python
from vllm import LLM, SamplingParams
model_id = "embedl/gemma-3-1b-it-FlashHead"
if __name__ == "__main__":
sampling = SamplingParams(max_tokens=128, temperature=0.0)
llm = LLM(model=model_id, trust_remote_code=True, max_model_len=131072)
prompt = "Write a haiku about coffee."
output = llm.generate([prompt], sampling)
print(output[0].outputs[0].text)
```
---
## Limitations
- Requires **vLLM >= 0.14.0**
- Currently optimized for **NVIDIA RTX GPUs**
---
## Roadmap
Planned improvements:
- Advanced mixed precision quantization
- Huggingface transformers generation
- vLLM CLI benchmarking for detailed latency evaluation
- `lm-eval-harness` integration for detailed accuracy evaluation
- Upstream support in **Transformers** and **vLLM**
- Compatibility with **GGUF**, **MLC**, **Llama.cpp**, **Ollama**, etc.
- Broader model coverage (larger models, VLMs, VLAs)
---
## License
- **Upstream:** Gemma Terms of Use.
- **Optimized Components:** Embedl Models Community Licence v1.0 *(no redistribution)*
---
## Contact
**Enterprise & Commercial Inquiries**
[models@embedl.com](mailto:models@embedl.com)
**Technical Issues & Early Access**
[https://github.com/embedl/flash-head](https://github.com/embedl/flash-head)
**More Information & Model Releases**
[https://embedl.com](https://embedl.com)
---
### Partner & Developer Opportunities
If you are evaluating on-device inference, building products on SLMs, or exploring custom model optimization, reach out for:
- Embedl SDK - AI optimization tools & profiling
- Embedl HUB - benchmarking platform
- Engineering support for on-prem/edge deployments
- Migration guidance (Llama / Qwen / Gemma)
- Early access & partner co-marketing opportunities
Contact: [models@embedl.com](mailto:models@embedl.com)