Pre-Tokenizer Latent Memory Routing.
open sourceA research study that lets retrieved memory decide where a BLT-style byte-level language model places its patch boundaries, using FAISS retrieval similarity to modulate the segmentation threshold. It is compared against a no-memory baseline and a post-segmentation retrieval control.
highlights
- 01Pre-registered pass/fail criteria, multi-seed controlled evaluation and a documented failure log.
- 02A contrastive memory encoder trained across three iterations, the last with multi-positive supervised contrastive loss.
- 03A 16-page paper reporting a 7–10% relative loss reduction on evidence regions in a 9.3M-parameter study, and analysing an encoder-geometry trade-off.
- 04The related Innovate UK AI Champions: Frontier AI Phase 1 application received an overall assessment score of 77%.
evaluation & scope
The 7–10% figure is relative loss reduction on evidence regions in a 9.3M-parameter controlled study. It is not a general capability gain or a result from a frontier-scale model.
built with
- python
- pytorch
- faiss