LFM2.5-1.2B-Instruct — LiteRT-LM

LiquidAI/LFM2.5-1.2B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (requires litert-lm ≥ 0.14 / a recent AI Edge Gallery).

Update (2026-08-04): the .litertlm files were updated in place to add the ExecutorMetadata section that litert-lm ≥ 0.15 requires to bind the hybrid conv/attention state buffers (without it, 0.15 fails at inference with missing some output TensorBuffers). Weights and graph are byte-identical to the original release, and the files continue to run on litert-lm 0.14.

LFM2.5-1.2B-Instruct is Liquid AI's flagship on-device model: a hybrid architecture mixing gated short-convolution blocks with grouped-query attention. Per the base model card it outperforms Qwen3-1.7B and Gemma 3 1B on knowledge and instruction-following benchmarks at a smaller footprint.

File	Recipe	Size	GSM8K (n=100)
LFM2.5-1.2B-Instruct_int8.litertlm	int8 dynamic (linears + convs + embedding)	1.25 GB	81% (bf16 reference: 79%)
LFM2.5-1.2B-Instruct_int4.litertlm	int4 blockwise-32 + OCTAV linears, int8 embedding, convs float	736 MB	72%
LFM2.5-1.2B-Instruct_int4_gpu.litertlm	same int4 recipe, re-exported so it runs on the GPU	736 MB	= int4 (A/B in notes)

Context (KV cache)	4096 max (see --max-num-tokens note)
Backend	CPU for every file; the _int4_gpu file also runs on the GPU with litert-lm ≥ 0.16.0 — Android OpenCL, macOS, and iOS Metal, all verified by generation. On iOS set maxNumTokens to 1024 (see the iPhone section — a mismatched value breaks engine creation)
Template	bundled — full chat template incl. tool-list support + <think> channel declaration
Base model	LiquidAI/LFM2.5-1.2B-Instruct (LFM Open License v1.0)

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 1024, n=100, same harness for all rows:

Configuration	GSM8K
PyTorch bf16 (reference)	79%
LiteRT int8 (this repo)	81%
LiteRT int4-b32 OCTAV (this repo)	72%

The int8 file is at full parity with the PyTorch reference (+2pt, within noise). Both files pass an 8-question sanity gate (7/8, zero degenerate outputs). The int4 file trades −7pt for a 41% smaller download.

Usage

litert-lm run ./LFM2.5-1.2B-Instruct_int8.litertlm --prompt "Explain what a hybrid conv-attention LLM is in two sentences."

The bundle carries the tokenizer and the full chat template (ChatML-style <|im_start|>…<|im_end|> with tool-calling support). Generation stops cleanly at <|im_end|>.

--max-num-tokens tip: decode speed drops as the token budget grows (int8 decode: 101 tok/s at 1024 → 77 tok/s at 4096 on an M4 Max) — set it to the smallest value your use case needs (1024 is a good chat default; the file allows up to 4096).

Performance

litert-lm benchmark (litert-lm 0.15.0), CPU backend, Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), warm-up run discarded, otherwise idle machine: decode depends strongly on the KV budget, so both settings are listed:

Variant	--max-num-tokens	Prefill (256)	Decode	TTFT
int8	1024	1592 tok/s	93.1 tok/s	0.17 s
int4	1024	381 tok/s	109.4 tok/s	0.68 s
int8	4096	1121 tok/s	78.9 tok/s	0.24 s
int4	4096	343 tok/s	74.9 tok/s	0.76 s

On a Pixel 8a (Tensor G3, CPU backend, measured in AI Edge Gallery): int8 decodes at ~19 tok/s and int4 at ~31 tok/s — on phone-class memory bandwidth the int4 file is ~1.7× faster as well as 41% smaller, so prefer int4 on mid-range devices unless you need the last few GSM8K points.

GPU — the _int4_gpu file

Pixel 8a (Tensor G3), litert_lm_main built from the litert-lm v0.16.0 release tag, 263-token prompt, generation capped at 256, 3 runs per backend:

Backend	Prefill (263 tok)	Decode	TTFT
GPU (OpenCL)	188–193 tok/s	21.0–21.2 tok/s	1.41–1.44 s
CPU	38–54 tok/s	15.2–24.3 tok/s	4.9–7.0 s

On phone-class hardware the GPU's win is prefill and time-to-first-token (3–6× both); decode is bandwidth-bound and roughly a wash, so long prompts gain far more than long answers. The CPU rows spread because the phone throttles across a run series; the GPU rows repeat to within 2%.

Apple M4 Max (litert-lm benchmark --cache no, litert-lm 0.16.0, -p 256 -d 256 --runs 3, otherwise idle machine; both backends were verified to generate correct text before these numbers were quoted):

Backend	Prefill (256)	Decode	TTFT
GPU	3765 tok/s	318.3 tok/s	0.07 s
CPU	337.0 tok/s	80.5 tok/s	0.78 s

On a desktop GPU the gain is across the board — roughly 11× prefill and 4× decode against the same file on CPU.

The hybrid short-conv blocks are what make a 1.2B decode at ~100 tok/s on CPU — faster than same-size pure-attention models. Long prompts prefill through the large (up to 1024-token) signatures at >1500 tok/s.

GPU — Galaxy S26 (Snapdragon 8 Elite Gen 5)

Galaxy S26 (SM-S942Q / SM8850, Android 16), litert_lm_main from litert-lm 0.16.0, the _int4_gpu file, 19-token prompt, one run per backend:

Backend	Prefill (19 tok)	Decode	TTFT	Init
GPU (OpenCL)	80.3 tok/s	24.7 tok/s	0.28 s	15.0 s
CPU (XNNPACK)	10.3 tok/s	24.1 tok/s	1.89 s	6.5 s

The prompt was 19 tokens, so read the prefill column as a floor rather than a throughput figure — at that length the fixed per-call cost dominates. The 263-token Pixel 8a rows above are the throughput measurement. Decode is a wash (24.7 against 24.1), which is what bandwidth-bound decode looks like; the GPU's win here is time-to-first-token, 6.7×. The GPU took the whole graph, 542 of 542 ops on LITERT_CL; on CPU, XNNPACK took 469 of 542.

GPU — iPhone 17 Pro (iOS Metal)

The engine-creation failure previously tracked in LiteRT-LM#3129 turned out to be a context-sizing issue on the integration side, not a runtime bug. The Metal delegate compiles its kernels against the KV cache that maxNumTokens pre-allocates; a value that does not match this file's exported plan (our test app derived ~650) makes engine creation fail with a shader-compile error. On iOS Metal, set maxNumTokens to 1024 (Swift EngineConfig(maxNumTokens:), C API litert_lm_engine_settings_set_max_num_tokens).

iPhone 17 Pro, litert-lm v0.16.0 xcframework, maxNumTokens 1024, single run, device charging: decode 67.7 tok/s (the int4 file on CPU on the same phone: 52.6), TTFT 109 ms on a short prompt, prefill 1,982 tok/s at p=128, peak ~470 MB. All Metal kernels compile, generation is coherent, and the file scores the same 8-question sanity gate as on CPU (7/8).

Only the _int4_gpu file loads on the GPU. LFM2.5-1.2B-Instruct_int4.litertlm delegates 536 of its 579 ops and then fails to create the engine, because the remaining 43 do not lower (GATHER_ND: Operation is not supported, plus several INT64 tensors). Use _int4_gpu on the GPU backend and _int4 or _int8 on CPU.

Run on Android

Install a recent Google AI Edge Gallery (1.0.16+ imports .litertlm directly from Hugging Face), import this repo (or adb push a file and use local import), select the CPU backend, and chat. GPU acceleration needs the _int4_gpu file and litert-lm ≥ 0.16.0 (litert_lm_main --backend=gpu, or the SDKs); the Gallery bundles its own runtime, which may lag that version — if its GPU toggle fails there, use CPU.

⚠ Gallery's import dialog defaults to CPU-only. Scroll it to Compatible accelerators and tick GPU at import time; a model already imported has to be deleted and re-imported. Nothing in the bundle sets this. Full recipe: Running converted models on GPU (Android).

Conversion notes

Converted with released litert-torch 0.9.1 (which ships the lfm2 hybrid model support) and packaged for litert-lm ≥ 0.14, with one exporter patch: the stock LFM2 short-conv block saves its conv state from the last columns of the prefill chunk, which are padding whenever the prompt does not exactly fill a prefill signature — corrupting the first generated token of nearly every reply. The patch derives the valid length from the attention mask in-graph and gathers the conv state from the last valid columns instead; with it, engine output is token-identical to an exact per-token reference loop at every prompt length. Multi-length prefill signatures (1–1024) are exported so the runtime can pick tight chunks.

Quantization targets fully-connected + embedding ops (plus convs in the int8 file, quantized at export time); post-hoc int8 of the convs with ai-edge-quantizer breaks generation — quantize convs only at export time.

Why there is a separate _int4_gpu file. The exporter patch above reads the chunk's valid length with index_select and a mask sum, which lower to GATHER_ND and INT64 ops, and GPU delegates reject both — the CPU-only files take 536 of 579 operations on the GPU and the runtime then refuses the partial split. litert-torch 0.9.2 fixed the underlying conv-state bug upstream using valid-token masking and a one-hot matmul state select, which emits neither op. _int4_gpu is that export: litert-torch 0.9.3 + litert-converter 0.3.1, the same int4 recipe and the same weights as the int4 file, plus the executor-metadata section litert-lm ≥ 0.15 needs. It delegates fully on Android OpenCL — 501/501 and 519/519 nodes, zero rejected ops — and runs on the macOS GPU backend. Quality is unchanged: run head to head against the published int4 file on identical prompts, decoding and scoring, both score 38/50 on GSM8K (CPU, n=50), and the GPU file passes an 8-question sanity gate on both backends. The CPU-only files are left exactly as they are rather than replaced, so nothing changes for anyone already using them.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
LFM2.5-1.2B-Instruct_int4.litertlm	54.1 (53.7–54.2)	9.3 (9.3–9.3)	4.8 s	1.4 GB
LFM2.5-1.2B-Instruct_int4_gpu.litertlm	54.6 (53.5–54.6)	9.3 (9.3–9.3)	4.8 s	1.4 GB
LFM2.5-1.2B-Instruct_int8.litertlm	75.0 (74.5–75.7)	7.0 (7.0–7.1)	3.6 s	1.9 GB

License and changes

Distributed under the LFM Open License v1.0 (see LICENSE, inherited from the base model). Note the license's commercial-use limitation for organizations above US$10M annual revenue. Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified into the .litertlm bundle; exporter conv-state fix as described in Conversion notes. This repository is a community conversion and is not affiliated with Liquid AI.