LiteRT

LiteRT (formerly TensorFlow Lite) is Google’s runtime for on-device inference. The model format is .tflite and language models ship as one .litertlm file for the LiteRT-LM runtime.

Export a Transformers model with litert-torch. It lowers the torch.export graph to LiteRT directly, and not through ONNX or a TensorFlow SavedModel.

pip install litert-torch

litert-torch export_hf \
    --model="HuggingFaceTB/SmolLM2-135M-Instruct" \
    --output_dir="./smollm2_litertlm"

Transformers integration

1. from_pretrained() loads the model weights in safetensors format.
2. litert-torch runs torch.export and lowers the graph to LiteRT operators. export_hf adds the KV cache, prefill and decode signatures, and int8 quantization.
3. AutoTokenizer loads the tokenizer. export_hf packs it and the chat template into the .litertlm file.
4. At runtime, .tflite runs on LiteRT and .litertlm on LiteRT-LM, from Kotlin, Swift, C++, or Python (ai-edge-litert and litert-lm-api). The older tflite-runtime wheels stop at Python 3.11.

Transformers v4 documented optimum-cli export tflite, which converted through TensorFlow. It was removed with TensorFlow support in v5 (#40760) and is not part of Optimum 2.x.

Resources

• LiteRT and LiteRT-LM docs
• Convert PyTorch models and GenAI models guides