Ciaranshu/decor-qwen3-4b-original

DecoR: Qwen3-4B Fine-tuned on Original LIMO (Baseline)

This model is Qwen3-4B fine-tuned on the original (unmodified) LIMO dataset. It serves as the baseline for comparison with DSE-cleaned variants.

Key Results

Benchmark	Base Qwen3-4B	Original LIMO SFT	DSE LIMO SFT
MATH-500	56.6%	69.6%	72.8%
AIME 2025	13.3%	40.0%	43.3%
AIME 2026	36.7%	46.7%	53.3%
GPQA Diamond	43.4%	55.6%	49.0%

Training Details

• Base model: Qwen/Qwen3-4B
• Training data: LIMO-Original (817 samples, unmodified)
• Framework: LLaMA-Factory + DeepSpeed ZeRO-2
• Hardware: 2× NVIDIA A100 80GB (CSD3 HPC)
• Hyperparameters:
  • Learning rate: 5e-6 (cosine schedule, 10% warmup)
  • Batch size: 8 (effective)
  • Epochs: 15
  • Max sequence length: 16384
  • Full fine-tuning (no LoRA)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Ciaranshu/decor-qwen3-4b-original")
tokenizer = AutoTokenizer.from_pretrained("Ciaranshu/decor-qwen3-4b-original")

Framework versions

• Transformers 4.52.4
• Pytorch 2.5.1+cu124
• Datasets 3.6.0
• Tokenizers 0.21.1