Nanbeige4.1-3B — LiteRT-LM (blockwise int4)

Nanbeige/Nanbeige4.1-3B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

Nanbeige4.1-3B is a fresh (Dec 2025) phone-size reasoning model on a plain dense Llama architecture (Apache-2.0), reported to be competitive with much larger models. It works the problem inside a <think>…</think> block before giving the final answer.

File	model.litertlm (~2.2 GB; embedding externalized so every section is <2 GiB → loads on iOS)
Quantization	int4 weights — blockwise (block 32) + OCTAV optimal-clipping, symmetric; embedding INT8
Compute	integer
Context (KV cache)	4096
Base model	Nanbeige/Nanbeige4.1-3B (Apache-2.0)

Usage

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts total?"

The .litertlm bundle carries the tokenizer and the prompt template (ChatML — <|im_start|>role\n … <|im_end|>), so no separate tokenizer files are needed. This is a reasoning model: it emits a <think>…</think> chain then the final answer (best evaluated with a generous token budget), and stops cleanly at <|im_end|>.

Run on Android

Install a recent Google AI Edge Gallery (1.0.16+ can import .litertlm directly from Hugging Face), download model.litertlm (or import this repo in-app), pick the GPU backend (CPU also works), and chat. Give it a high max-tokens — it's a reasoning model with long chains of thought.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.

Device	Backend	Prefill (256)	Decode	TTFT	Load	Peak footprint
Apple M4 Max (macOS)	CPU	125 tok/s	24.4 tok/s	2.39 s	—	—
Apple M4 Max (macOS)	GPU (Metal)	1256 tok/s	90.1 tok/s	0.22 s	—	—

Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.

No on-device timing is quoted for this build. The Galaxy S26 section below reports what was measured on Android and explains why no speed rows are quoted there.

Accuracy note

Measured on GSM8K (n=50, greedy, 0-shot chain-of-thought, max-tokens 2048 — a reasoning model needs the budget to finish; scoring it at 512 tokens falsely penalises it):

Configuration	GSM8K
This model — LiteRT int4 (block32 + OCTAV)	84.0%

84% is a strong on-device GSM8K for a 3B, non-degenerate; the model also passes the local quality gate 8/8 with a clean stop at <|im_end|>. Blockwise-32 + OCTAV optimal-clipping (data-free) preserves the accuracy versus a naive min-max int4.

Galaxy S26 — GPU backend

The published bundle runs on the Android GPU backend and generates.

file	GPU backend	delegation	peak
model.litertlm	runs	2715 / 2715 ops across 2 subgraphs on LiteRT GPU	1213 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_128; the runtime accepts that split.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion

Converted with litert-torch using a blockwise int4 recipe (INT4 weights, block size 32, symmetric, OCTAV optimal-clipping) with the embedding at INT8, KV cache 4096, and a ChatML prompt template. Nanbeige4.1 is a standard dense LlamaForCausalLM, so it rides the existing converter and runtime with no custom graph code.

externalize_embedder=True (required for iPhone). The large 166k-token vocab makes the weights a >2 GiB single TFLite section, which exceeds the ~2 GiB single-section mmap limit on iOS. Externalizing the embedding drops the main section under 2 GiB so the model loads on iPhone (Metal GPU) as well as Android/desktop. Same weights, so GSM8K is unchanged.

Added-tokens tokenizer fix. Nanbeige's 10 special tokens (<|im_start|>, <|im_end|>, <think>, </think>, <tool_call>, …) live at vocab ids 166100–166109, above the base SentencePiece vocab (166100). The base SP conversion drops them, so the reasoning model would generate <think> (id 166103) and the runtime would crash with "Token id out of range." The converted tokenizer here appends those added tokens as USER_DEFINED SentencePiece pieces at their exact ids (padded to the model vocab), so <think> and friends decode correctly.

2026-08-28 — start_token fix (weights unchanged)

The LiteRT-LM engine prepends the metadata start_token to every prompt, and this model's chat template already renders <|im_start|> itself — so the model was reading <|im_start|><|im_start|>…, a stream it was never trained on. The start token has been removed. This bundle also has its prompt-template system.prefix corrected to <|im_start|>system\n: the leading marker used to come from the start token, so removing that alone would have broken the system-prompt path.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.

2026-08-29 — default system prompt restored (weights unchanged)

The upstream chat template emits a default system turn whenever the caller sends no system message — for this model: 你是南北阁，一款由BOSS直聘自主研发并训练的专业大语言模型。. The converter's template probe renders the template with a system message already present, so that block was never seen and never reached the bundle: with no system message the model was running without the default system turn it was tuned with. The chat template in model.litertlm now emits the block exactly once when no system message is given. In model.litertlm, the block is not emitted when you pass a system message. The restored block adds 21 prefill tokens to a conversation that sends no system message, so time-to-first-token grows by that much; per-token speed is unchanged.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file per token — only the file's own sha256 differs. What changed is the input: with no system message, the prompt now renders byte-identical to the upstream chat template's output, verified on the LiteRT-LM runtime. A system message you pass yourself renders as before. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-29, re-download.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
model.litertlm	16.5 (16.4–16.5)	2.5 (2.5–2.5)	18.1 s	3.1 GB

License

Apache-2.0, inherited from the base model Nanbeige/Nanbeige4.1-3B.