A small language model that decides when to retrieve.
weights · open sourceA QLoRA fine-tune of Qwen3.5-2B that emits Self-RAG reflection tokens, deciding when to retrieve and judging relevance and usefulness, on-device.
highlights
- 01End-to-end pipeline: synthetic critique data, QLoRA training, LoRA merge, quantization, vLLM serving and lm-evaluation-harness benchmarking.
- 02Reported MMLU 62.0, HellaSwag 64.1 and ARC-C 42.8, with 82.5% reflection-token accuracy, and benchmarked on-device on Apple M3.
- 03Weights in full precision and GGUF (Q4_K_M), with a 38-page technical report.
evaluation & scope
These are reported results from the model release. Reflection-token accuracy measures the retrieval-control tokens, not overall answer correctness. Consult the technical report for the evaluation setup and comparison baselines.
built with
- python
- pytorch
- peft
- qlora
- vllm
- lm-eval-harness
- llama.cpp
Co-authored with Naga Sri Arvapalli.