Siger Dipa 151M

Siger Dipa 151M is an experimental chat/instruction language model built from scratch as part of the SigerLM project.

Unlike standard Transformer-only checkpoints, Dipa is based on a custom PyTorch SSM / Mamba-like backbone with long-context research components, selective sequence modeling, and LoRA-based instruction tuning. This release is an early public milestone intended for testing, feedback, and transparent community progress.

Highlights

151M parameter class
Custom SSM/Mamba-like architecture
Merged LoRA chat milestone
Latest continued-training progress: base resume reached step 4084 / 80000
Indonesian + English instruction behavior
Long-context research foundation
Built and trained openly as a from-scratch LLM project
Transparent training log with checkpoints, PPL, and release artifacts
Lightweight enough for experimentation compared to large frontier-scale models

Project Snapshot

Siger Dipa is the first public model in the SigerLM Language Family:

SigerLM Language Family: Dipa -> Arka -> Tarka -> Kavi -> Veda
Siger Visual Family:     Siger Chitra
Siger Developer Family:  Siger Karya

The current repository contains the custom model core, tokenizer, dataset builders, training pipeline, LoRA runner, inference router, retrieval/domain tools, benchmark scaffolding, and documentation. The broader goal is to grow SigerLM into a lightweight multilingual and developer-focused AI stack while keeping Dipa as an inspectable research checkpoint.

Public Artifact Status

This is a testable research checkpoint, not a final model.

Type: merged LoRA chat milestone
Base checkpoint step: 1800
Checkpoint: checkpoints/lora/dipa_long/model_dipa_long_step_0001800_merged.pt
Tokenizer: checkpoints/tokenizer_hf_bpe
Framework: custom PyTorch SigerLM codebase
Status: early public chat/instruction release

Latest Training Progress

The model is still being trained after the public step-1800 chat milestone. The latest reported continued-training run produced a base-resume checkpoint:

Item	Value
Latest base step	`4084 / 80000`
Checkpoint	`checkpoints/dipa_long/step_0004084_20260803_095343.pt`
Checkpoint kind	base resume
Rolling avg loss	`1.3587`
PPL	`3.89`
Session time	`5.05h`
Recent throughput	about `829-840 tok/s`
Release ZIP	`siger_dipa_long_step_0004084_base_release.zip`
Merged LoRA for this step	not yet

Step 4084 is not a LoRA milestone because the current release pipeline merges LoRA on milestone/final steps. The next milestone is 4200, so the training needs 116 more base steps from 4084 before the next milestone merge window.

Why This Model Exists

SigerLM is an attempt to explore a smaller, modular LLM architecture that combines several ideas:

SSM/Mamba-like sequence modeling for efficient token mixing
selective and hybrid sequence components for long-context research
Hybrid long-context foundations for future longer memory
LoRA adapters for instruction and domain learning
evaluation and release automation for repeatable community testing
Modular future paths for coding, frontend vision, database planning, and multilingual/local-language work

The goal is not to claim state-of-the-art performance yet. The goal is to build a real, inspectable research model step by step and let the community follow the progress.

Architecture Notes

SigerLM is not a Transformer-only model. The Dipa line is built around a custom PyTorch SSM/Mamba-like backbone with modular long-context profiles, hybrid tokenization, LoRA adaptation, and planned support for additional research families such as MoE, ONNX/quantized deployment, multimodal adapters, and developer-agent tooling.

Current repository foundations include:

base training and resumable checkpointing
HF ByteLevel BPE tokenizer support with fallback tokenizer paths
LoRA instruction tuning and merge/export pipeline
long-context dipa_long profile with streaming/segmented training support
engineering evaluation harness and benchmark scaffolding
frontend assistant, database planner, task-agent, and multimodal adapter foundations for future Siger capabilities

Usage

This checkpoint currently requires the SigerLM repository code. It is not yet a standard Transformers AutoModelForCausalLM checkpoint.

Example after placing the files inside the SigerLM repository root:

python chat_cli.py \
  --checkpoint checkpoints/lora/dipa_long/model_dipa_long_step_0001800_merged.pt \
  --mode chat \
  --prompt "Jelaskan cara membuat branch git baru dengan aman." \
  --device cuda \
  --max-new-tokens 80 \
  --no-abstain

Intended Use

This model is intended for:

research and experimentation
Indonesian/English chat testing
lightweight instruction-following experiments
architecture exploration around SSM/Mamba-like language models
public build-log tracking and checkpoint comparison
early evaluation harness integration
community feedback on small custom language models

Limitations

This is an early checkpoint and has important limitations:

not benchmarked against major public suites yet
may hallucinate or produce incomplete answers
not safety-aligned for production use
requires custom SigerLM code
not yet compatible with standard Hugging Face Transformers loading
long-context capability is still under active training and evaluation

Do not use this model for medical, legal, financial, or safety-critical decisions.

Evaluation

Formal benchmark results are planned for future releases.

Planned benchmark targets:

MMLU
ARC
HellaSwag
HumanEval
MBPP

Current status: not_run for public benchmark reporting.

The repository now includes benchmark automation scaffolding so future Kaggle runs can export evaluation metadata together with model releases. Until official scores are published, this checkpoint should be treated as an experimental community-testing artifact.

Roadmap

Upcoming goals:

continue base training toward step 4200, then keep moving toward 80000
publish regular versioned releases
add public benchmark reports
improve Hugging Face compatibility
add compact inference artifacts
expand coding, frontend, database-planning, and multilingual capabilities
document architecture and training pipeline more completely

Community

This model is part of an ongoing build-in-public LLM project. Feedback, issues, tests, prompts, and suggestions are very welcome.

If you test Siger Dipa 151M, please share:

prompts that work well
prompts that fail
hallucination examples
benchmark attempts
ideas for architecture or dataset improvements

Downloads last month: -

Syarifsoden
/

siger_dipa_151M