Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+checkpoint-100/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-135/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-50/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-60/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,68 @@
+---
+library_name: transformers
+model_name: aria-ft-dpo
+tags:
+- generated_from_trainer
+- dpo
+- trl
+licence: license
+---
+
+# Model Card for aria-ft-dpo
+
+This model is a fine-tuned version of [None](https://huggingface.co/None).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+
+## Quick start
+
+```python
+from transformers import pipeline
+
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+
+## Training procedure
+
+ 
+
+
+
+This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
+
+### Framework versions
+
+- TRL: 1.3.0
+- Transformers: 5.7.0
+- Pytorch: 2.11.0
+- Datasets: 4.8.5
+- Tokenizers: 0.22.2
+
+## Citations
+
+Cite DPO as:
+
+```bibtex
+@inproceedings{rafailov2023direct,
+    title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
+    author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
+    year         = 2023,
+    booktitle    = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
+    url          = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
+    editor       = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
+}
+```
+
+Cite TRL as:
+    
+```bibtex
+@software{vonwerra2020trl,
+  title   = {{TRL: Transformers Reinforcement Learning}},
+  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
+  license = {Apache-2.0},
+  url     = {https://github.com/huggingface/trl},
+  year    = {2020}
+}
+```

chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

checkpoint-100/README.md

@@ -0,0 +1,209 @@
+---
+base_model: data/models/aria-ft-sft
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:data/models/aria-ft-sft
+- dpo
+- lora
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

checkpoint-100/adapter_config.json

@@ -0,0 +1,43 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "data/models/aria-ft-sft",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-100/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0efd89cc970fc0908572e2f25348eb8a00174db6e4282feddb428b58c46c199b
+size 12931296

checkpoint-100/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

checkpoint-100/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa6278a3cd6ebf1b1208a057d4557901895e98c85343bfe03892e9a114407bae
+size 17942731

checkpoint-100/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f196323d7423b60f8e4ceb7dbf8715ee326c0d068e5ff164f13c63b279b9f1a0
+size 14645

checkpoint-100/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c4e44404b58ce3af1b46c3d4a85a59edbbc386f340c476e894715a1199e1aed
+size 1465

checkpoint-100/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726
+size 33384567

checkpoint-100/tokenizer_config.json

@@ -0,0 +1,25 @@
+{
+  "backend": "tokenizers",
+  "boi_token": "<start_of_image>",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eoi_token": "<end_of_image>",
+  "eos_token": "<eos>",
+  "image_token": "<image_soft_token>",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "boi_token": "<start_of_image>",
+    "eoi_token": "<end_of_image>",
+    "image_token": "<image_soft_token>"
+  },
+  "pad_token": "<pad>",
+  "processor_class": "Gemma3Processor",
+  "sp_model_kwargs": null,
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-100/trainer_state.json

@@ -0,0 +1,394 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 6.689655172413794,
+  "eval_steps": 500,
+  "global_step": 100,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 0.7935861229896546,
+      "epoch": 0.3448275862068966,
+      "grad_norm": 5.0625,
+      "learning_rate": 4.8e-05,
+      "logits/chosen": -4.007446998194211,
+      "logits/rejected": -4.111432883182067,
+      "logps/chosen": -471.74893646240236,
+      "logps/rejected": -173.62605743408204,
+      "loss": 0.5125677108764648,
+      "mean_token_accuracy": 0.5682478994131088,
+      "num_tokens": 11659.0,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.6584509511245414,
+      "rewards/margins": 0.46967237079516055,
+      "rewards/rejected": 0.18877857606858015,
+      "step": 5
+    },
+    {
+      "entropy": 0.8235619857907295,
+      "epoch": 0.6896551724137931,
+      "grad_norm": 0.703125,
+      "learning_rate": 4.55e-05,
+      "logits/chosen": -3.770766021141263,
+      "logits/rejected": -3.977053060466863,
+      "logps/chosen": -428.3692001342773,
+      "logps/rejected": -166.3092363357544,
+      "loss": 0.09679355025291443,
+      "mean_token_accuracy": 0.5847053378820419,
+      "num_tokens": 23174.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.5914027988910675,
+      "rewards/margins": 2.6518951922655107,
+      "rewards/rejected": 0.9395075976848603,
+      "step": 10
+    },
+    {
+      "entropy": 0.8904998931619856,
+      "epoch": 1.0,
+      "grad_norm": 0.1171875,
+      "learning_rate": 4.3e-05,
+      "logits/chosen": -3.701669782430722,
+      "logits/rejected": -4.094562449373758,
+      "logps/chosen": -413.61048210991754,
+      "logps/rejected": -158.8006969028049,
+      "loss": 0.009873698651790618,
+      "mean_token_accuracy": 0.588409294684728,
+      "num_tokens": 33972.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 7.163314024607341,
+      "rewards/margins": 4.957976659138997,
+      "rewards/rejected": 2.2053373638126583,
+      "step": 15
+    },
+    {
+      "entropy": 0.93388631939888,
+      "epoch": 1.3448275862068966,
+      "grad_norm": 0.01708984375,
+      "learning_rate": 4.05e-05,
+      "logits/chosen": -3.515498939129551,
+      "logits/rejected": -3.9057978357629892,
+      "logps/chosen": -385.5538497924805,
+      "logps/rejected": -148.66188621520996,
+      "loss": 0.0015057120472192764,
+      "mean_token_accuracy": 0.5857352793216706,
+      "num_tokens": 45734.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 9.765967106819152,
+      "rewards/margins": 6.95135418176651,
+      "rewards/rejected": 2.8146128758788107,
+      "step": 20
+    },
+    {
+      "entropy": 0.9550707444548607,
+      "epoch": 1.6896551724137931,
+      "grad_norm": 0.0234375,
+      "learning_rate": 3.8e-05,
+      "logits/chosen": -3.4727558104574365,
+      "logits/rejected": -3.84623566928028,
+      "logps/chosen": -361.4485481262207,
+      "logps/rejected": -146.26134338378907,
+      "loss": 0.000905947107821703,
+      "mean_token_accuracy": 0.591161236166954,
+      "num_tokens": 57498.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 11.036408770084382,
+      "rewards/margins": 7.591825819015503,
+      "rewards/rejected": 3.4445829778909682,
+      "step": 25
+    },
+    {
+      "entropy": 1.000456228852272,
+      "epoch": 2.0,
+      "grad_norm": 0.0081787109375,
+      "learning_rate": 3.55e-05,
+      "logits/chosen": -3.3968431508762134,
+      "logits/rejected": -3.7621962730840672,
+      "logps/chosen": -359.68355136447485,
+      "logps/rejected": -141.86985800001355,
+      "loss": 0.0007186151575297117,
+      "mean_token_accuracy": 0.5946498264869055,
+      "num_tokens": 67944.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 11.177175945705837,
+      "rewards/margins": 7.978424787521362,
+      "rewards/rejected": 3.198751082022985,
+      "step": 30
+    },
+    {
+      "entropy": 0.9751293674111366,
+      "epoch": 2.344827586206897,
+      "grad_norm": 0.00860595703125,
+      "learning_rate": 3.3e-05,
+      "logits/chosen": -3.380549045012162,
+      "logits/rejected": -3.7619534125984417,
+      "logps/chosen": -359.2783546447754,
+      "logps/rejected": -143.991379737854,
+      "loss": 0.0005051379092037677,
+      "mean_token_accuracy": 0.5927061937749386,
+      "num_tokens": 79630.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 11.851354885101319,
+      "rewards/margins": 8.312449264526368,
+      "rewards/rejected": 3.538905668258667,
+      "step": 35
+    },
+    {
+      "entropy": 0.9894796743988991,
+      "epoch": 2.689655172413793,
+      "grad_norm": 0.00836181640625,
+      "learning_rate": 3.05e-05,
+      "logits/chosen": -3.4094987648490958,
+      "logits/rejected": -3.8709906737963586,
+      "logps/chosen": -355.86348724365234,
+      "logps/rejected": -142.16433296203613,
+      "loss": 0.0003436990547925234,
+      "mean_token_accuracy": 0.5933584332466125,
+      "num_tokens": 91583.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.51960310935974,
+      "rewards/margins": 8.580410575866699,
+      "rewards/rejected": 3.939192494750023,
+      "step": 40
+    },
+    {
+      "entropy": 1.0504967404736414,
+      "epoch": 3.0,
+      "grad_norm": 0.01080322265625,
+      "learning_rate": 2.8000000000000003e-05,
+      "logits/chosen": -3.36502425275768,
+      "logits/rejected": -3.7324026401684187,
+      "logps/chosen": -346.72445593939887,
+      "logps/rejected": -136.11718813578287,
+      "loss": 0.0003958356101065874,
+      "mean_token_accuracy": 0.5880120148261389,
+      "num_tokens": 101916.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.048056178622776,
+      "rewards/margins": 8.65420545472039,
+      "rewards/rejected": 3.393850710656908,
+      "step": 45
+    },
+    {
+      "entropy": 1.01220805644989,
+      "epoch": 3.344827586206897,
+      "grad_norm": 0.006805419921875,
+      "learning_rate": 2.5500000000000003e-05,
+      "logits/chosen": -3.3868815911927768,
+      "logits/rejected": -3.742613550857216,
+      "logps/chosen": -355.06092681884763,
+      "logps/rejected": -136.23973789215088,
+      "loss": 0.000314924237318337,
+      "mean_token_accuracy": 0.5929604865610599,
+      "num_tokens": 113677.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.345191144943238,
+      "rewards/margins": 8.660565280914307,
+      "rewards/rejected": 3.6846258997917176,
+      "step": 50
+    },
+    {
+      "entropy": 0.9930158525705337,
+      "epoch": 3.689655172413793,
+      "grad_norm": 0.00958251953125,
+      "learning_rate": 2.3000000000000003e-05,
+      "logits/chosen": -3.3474578022314185,
+      "logits/rejected": -3.7326277215029195,
+      "logps/chosen": -347.77015380859376,
+      "logps/rejected": -142.32026901245118,
+      "loss": 0.00043723396956920626,
+      "mean_token_accuracy": 0.600645287334919,
+      "num_tokens": 125383.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.202914929389953,
+      "rewards/margins": 8.495901811122895,
+      "rewards/rejected": 3.707013100385666,
+      "step": 55
+    },
+    {
+      "entropy": 1.0284520834684372,
+      "epoch": 4.0,
+      "grad_norm": 0.0159912109375,
+      "learning_rate": 2.05e-05,
+      "logits/chosen": -3.3911868677866988,
+      "logits/rejected": -3.879937585420405,
+      "logps/chosen": -353.35245090060766,
+      "logps/rejected": -142.19803449842664,
+      "loss": 0.00034696566872298716,
+      "mean_token_accuracy": 0.5808572901619805,
+      "num_tokens": 135888.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.55628784497579,
+      "rewards/margins": 8.83048865530226,
+      "rewards/rejected": 3.725799196296268,
+      "step": 60
+    },
+    {
+      "entropy": 0.9992264002561569,
+      "epoch": 4.344827586206897,
+      "grad_norm": 0.004302978515625,
+      "learning_rate": 1.8e-05,
+      "logits/chosen": -3.347245438745982,
+      "logits/rejected": -3.777001193182941,
+      "logps/chosen": -347.93334274291993,
+      "logps/rejected": -139.33049812316895,
+      "loss": 0.0003306626807898283,
+      "mean_token_accuracy": 0.5913299791514873,
+      "num_tokens": 147560.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.145365619659424,
+      "rewards/margins": 8.691704177856446,
+      "rewards/rejected": 3.4536614298820494,
+      "step": 65
+    },
+    {
+      "entropy": 1.0325676158070565,
+      "epoch": 4.689655172413794,
+      "grad_norm": 0.006927490234375,
+      "learning_rate": 1.55e-05,
+      "logits/chosen": -3.3495644466137042,
+      "logits/rejected": -3.7736230068284753,
+      "logps/chosen": -355.5695266723633,
+      "logps/rejected": -140.7855079650879,
+      "loss": 0.0003685819683596492,
+      "mean_token_accuracy": 0.5904536455869674,
+      "num_tokens": 159219.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.21111993789673,
+      "rewards/margins": 8.628398954868317,
+      "rewards/rejected": 3.582720997929573,
+      "step": 70
+    },
+    {
+      "entropy": 0.998895494474305,
+      "epoch": 5.0,
+      "grad_norm": 0.0025177001953125,
+      "learning_rate": 1.3000000000000001e-05,
+      "logits/chosen": -3.4303033975940433,
+      "logits/rejected": -3.7985490847090535,
+      "logps/chosen": -352.206903245714,
+      "logps/rejected": -140.3814148373074,
+      "loss": 0.00032770405523478984,
+      "mean_token_accuracy": 0.5949484490685992,
+      "num_tokens": 169860.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.80911119778951,
+      "rewards/margins": 8.679809384875828,
+      "rewards/rejected": 4.12930182284779,
+      "step": 75
+    },
+    {
+      "entropy": 0.9812990128993988,
+      "epoch": 5.344827586206897,
+      "grad_norm": 0.003173828125,
+      "learning_rate": 1.05e-05,
+      "logits/chosen": -3.420436977397286,
+      "logits/rejected": -3.745464290129913,
+      "logps/chosen": -354.4942123413086,
+      "logps/rejected": -141.51624279022218,
+      "loss": 0.00027657533064484596,
+      "mean_token_accuracy": 0.601226168870926,
+      "num_tokens": 181770.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.855251288414001,
+      "rewards/margins": 9.028668785095215,
+      "rewards/rejected": 3.826582506299019,
+      "step": 80
+    },
+    {
+      "entropy": 1.0137892261147499,
+      "epoch": 5.689655172413794,
+      "grad_norm": 0.0093994140625,
+      "learning_rate": 8.000000000000001e-06,
+      "logits/chosen": -3.3323429011829147,
+      "logits/rejected": -3.851841452987793,
+      "logps/chosen": -347.1164161682129,
+      "logps/rejected": -139.89001064300538,
+      "loss": 0.000323146372102201,
+      "mean_token_accuracy": 0.5889961808919907,
+      "num_tokens": 193375.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.292453861236572,
+      "rewards/margins": 8.635055756568908,
+      "rewards/rejected": 3.6573980897665024,
+      "step": 85
+    },
+    {
+      "entropy": 1.0396647685103946,
+      "epoch": 6.0,
+      "grad_norm": 0.0179443359375,
+      "learning_rate": 5.500000000000001e-06,
+      "logits/chosen": -3.372452634571764,
+      "logits/rejected": -3.7438357628088963,
+      "logps/chosen": -354.4196319580078,
+      "logps/rejected": -139.0562195248074,
+      "loss": 0.0004979531280696392,
+      "mean_token_accuracy": 0.5837964490056038,
+      "num_tokens": 203832.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 11.918954292933146,
+      "rewards/margins": 8.29782505830129,
+      "rewards/rejected": 3.621129294236501,
+      "step": 90
+    },
+    {
+      "entropy": 1.0393116056919098,
+      "epoch": 6.344827586206897,
+      "grad_norm": 0.00518798828125,
+      "learning_rate": 3e-06,
+      "logits/chosen": -3.360836368420224,
+      "logits/rejected": -3.734754671897479,
+      "logps/chosen": -356.809147644043,
+      "logps/rejected": -140.77989463806153,
+      "loss": 0.00042734318412840366,
+      "mean_token_accuracy": 0.5832010932266712,
+      "num_tokens": 215382.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.140155410766601,
+      "rewards/margins": 8.665791845321655,
+      "rewards/rejected": 3.4743634849786758,
+      "step": 95
+    },
+    {
+      "entropy": 0.9969131574034691,
+      "epoch": 6.689655172413794,
+      "grad_norm": 0.004150390625,
+      "learning_rate": 5.000000000000001e-07,
+      "logits/chosen": -3.350688554334807,
+      "logits/rejected": -3.8668883818162825,
+      "logps/chosen": -354.6611267089844,
+      "logps/rejected": -138.6419864654541,
+      "loss": 0.0003045067191123962,
+      "mean_token_accuracy": 0.5958905071020126,
+      "num_tokens": 227310.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 12.683370137214661,
+      "rewards/margins": 8.687329852581025,
+      "rewards/rejected": 3.996040293574333,
+      "step": 100
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 100,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 7,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6729220104441984.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-100/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eecdd7063df6d938396a107c5c6396d070a154f2fb363182000b92d366e0a7f4
+size 5905

checkpoint-135/README.md

@@ -0,0 +1,209 @@
+---
+base_model: data/models/aria-ft-sft
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:data/models/aria-ft-sft
+- dpo
+- lora
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

checkpoint-135/adapter_config.json

@@ -0,0 +1,43 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "data/models/aria-ft-sft",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-135/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e449aece674300bf0e196eb2915ba3072b22335be8173f33af845d287aebd341
+size 12931296

checkpoint-135/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

checkpoint-135/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c38cdd4430bc66fb6bcc6a26a112705365a2af2b7c0d407d6f39215a161ecbca
+size 17942731

checkpoint-135/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4a9f217e852f439efa6bd32fde98d6867f11aa6ea13ddc021ba10af6a0b0934
+size 14645

checkpoint-135/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1cc3a4ab8801dd76de1fb4bc5a1bab0900b2f86bee30058e7c5e95f519ebfcf
+size 1465

checkpoint-135/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726
+size 33384567

checkpoint-135/tokenizer_config.json

@@ -0,0 +1,25 @@
+{
+  "backend": "tokenizers",
+  "boi_token": "<start_of_image>",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eoi_token": "<end_of_image>",
+  "eos_token": "<eos>",
+  "image_token": "<image_soft_token>",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "boi_token": "<start_of_image>",
+    "eoi_token": "<end_of_image>",
+    "image_token": "<image_soft_token>"
+  },
+  "pad_token": "<pad>",
+  "processor_class": "Gemma3Processor",
+  "sp_model_kwargs": null,
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-135/trainer_state.json

@@ -0,0 +1,520 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.0,
+  "eval_steps": 500,
+  "global_step": 135,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 0.3847092197276652,
+      "epoch": 0.1111111111111111,
+      "grad_norm": 9.125,
+      "learning_rate": 1.9407407407407407e-05,
+      "logits/chosen": -4.885123966774574,
+      "logits/rejected": -5.1270562646178774,
+      "logps/chosen": -158.49386949539183,
+      "logps/rejected": -121.08237323760986,
+      "loss": 0.6680909156799316,
+      "mean_token_accuracy": 0.7906534194946289,
+      "num_tokens": 9886.0,
+      "rewards/accuracies": 0.525,
+      "rewards/chosen": 0.005077153391175671,
+      "rewards/margins": 0.05780954551883042,
+      "rewards/rejected": -0.052732392022153365,
+      "step": 5
+    },
+    {
+      "entropy": 0.3929780837148428,
+      "epoch": 0.2222222222222222,
+      "grad_norm": 7.65625,
+      "learning_rate": 1.866666666666667e-05,
+      "logits/chosen": -4.828267853297093,
+      "logits/rejected": -5.127053542949903,
+      "logps/chosen": -156.8652853012085,
+      "logps/rejected": -134.7806537628174,
+      "loss": 0.42449655532836916,
+      "mean_token_accuracy": 0.7939460828900338,
+      "num_tokens": 19712.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.2097376121673733,
+      "rewards/margins": 0.6777558078989386,
+      "rewards/rejected": -0.468018195554032,
+      "step": 10
+    },
+    {
+      "entropy": 0.338850056566298,
+      "epoch": 0.3333333333333333,
+      "grad_norm": 5.15625,
+      "learning_rate": 1.7925925925925927e-05,
+      "logits/chosen": -5.028701342243549,
+      "logits/rejected": -5.435351208367371,
+      "logps/chosen": -131.43772993087768,
+      "logps/rejected": -127.4791706085205,
+      "loss": 0.24627490043640138,
+      "mean_token_accuracy": 0.8101926892995834,
+      "num_tokens": 29323.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.3851022788323462,
+      "rewards/margins": 1.4705152098089456,
+      "rewards/rejected": -1.0854129238054155,
+      "step": 15
+    },
+    {
+      "entropy": 0.26730762515217066,
+      "epoch": 0.4444444444444444,
+      "grad_norm": 1.625,
+      "learning_rate": 1.7185185185185185e-05,
+      "logits/chosen": -5.2203684086809305,
+      "logits/rejected": -5.50024154494532,
+      "logps/chosen": -110.35588598251343,
+      "logps/rejected": -139.20155849456788,
+      "loss": 0.11154030561447144,
+      "mean_token_accuracy": 0.8486447870731354,
+      "num_tokens": 38958.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 0.628392239799723,
+      "rewards/margins": 2.5915900975465775,
+      "rewards/rejected": -1.9631978537887336,
+      "step": 20
+    },
+    {
+      "entropy": 0.38323604678735135,
+      "epoch": 0.5555555555555556,
+      "grad_norm": 1.3671875,
+      "learning_rate": 1.6444444444444444e-05,
+      "logits/chosen": -5.028822769720685,
+      "logits/rejected": -5.223528454144411,
+      "logps/chosen": -158.7658320426941,
+      "logps/rejected": -128.570237159729,
+      "loss": 0.07294961810112,
+      "mean_token_accuracy": 0.8030859053134918,
+      "num_tokens": 49048.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.5944283031858504,
+      "rewards/margins": 3.1330706253647804,
+      "rewards/rejected": -1.538642304390669,
+      "step": 25
+    },
+    {
+      "entropy": 0.380564391054213,
+      "epoch": 0.6666666666666666,
+      "grad_norm": 0.8828125,
+      "learning_rate": 1.5703703703703705e-05,
+      "logits/chosen": -4.957261663249699,
+      "logits/rejected": -5.23616631647916,
+      "logps/chosen": -141.71369543075562,
+      "logps/rejected": -136.54375,
+      "loss": 0.05011730194091797,
+      "mean_token_accuracy": 0.8009175747632981,
+      "num_tokens": 58939.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.2282287888228893,
+      "rewards/margins": 3.826947730779648,
+      "rewards/rejected": -1.5987189412117004,
+      "step": 30
+    },
+    {
+      "entropy": 0.5169597734697163,
+      "epoch": 0.7777777777777778,
+      "grad_norm": 3.671875,
+      "learning_rate": 1.4962962962962964e-05,
+      "logits/chosen": -4.6997975165115635,
+      "logits/rejected": -5.029099791034667,
+      "logps/chosen": -181.9060929298401,
+      "logps/rejected": -138.3939012527466,
+      "loss": 0.03964149057865143,
+      "mean_token_accuracy": 0.7527312003076077,
+      "num_tokens": 69119.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.5208606626838446,
+      "rewards/margins": 4.38912188410759,
+      "rewards/rejected": -0.8682612210512162,
+      "step": 35
+    },
+    {
+      "entropy": 0.3221268252469599,
+      "epoch": 0.8888888888888888,
+      "grad_norm": 0.2255859375,
+      "learning_rate": 1.4222222222222224e-05,
+      "logits/chosen": -5.15146982525422,
+      "logits/rejected": -5.51558997221156,
+      "logps/chosen": -126.08706741333008,
+      "logps/rejected": -130.62417697906494,
+      "loss": 0.02573828101158142,
+      "mean_token_accuracy": 0.8234837621450424,
+      "num_tokens": 79017.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.930635741353035,
+      "rewards/margins": 5.160017335414887,
+      "rewards/rejected": -2.22938157916069,
+      "step": 40
+    },
+    {
+      "entropy": 0.3421414390206337,
+      "epoch": 1.0,
+      "grad_norm": 0.1083984375,
+      "learning_rate": 1.3481481481481482e-05,
+      "logits/chosen": -5.156180893981231,
+      "logits/rejected": -5.384254451849773,
+      "logps/chosen": -138.75859479904176,
+      "logps/rejected": -139.98251609802247,
+      "loss": 0.01194305494427681,
+      "mean_token_accuracy": 0.8202281922101975,
+      "num_tokens": 89078.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.1425785372965036,
+      "rewards/margins": 5.660494846105576,
+      "rewards/rejected": -2.5179163038730623,
+      "step": 45
+    },
+    {
+      "entropy": 0.30250622164458035,
+      "epoch": 1.1111111111111112,
+      "grad_norm": 0.70703125,
+      "learning_rate": 1.2740740740740742e-05,
+      "logits/chosen": -5.289744471980145,
+      "logits/rejected": -5.602541218371521,
+      "logps/chosen": -114.58103532791138,
+      "logps/rejected": -139.44555015563964,
+      "loss": 0.007524536550045013,
+      "mean_token_accuracy": 0.8363645195960998,
+      "num_tokens": 98826.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.5273715522140265,
+      "rewards/margins": 5.879914045333862,
+      "rewards/rejected": -3.35254248380661,
+      "step": 50
+    },
+    {
+      "entropy": 0.35719655752182006,
+      "epoch": 1.2222222222222223,
+      "grad_norm": 0.1328125,
+      "learning_rate": 1.2e-05,
+      "logits/chosen": -5.1927954861206524,
+      "logits/rejected": -5.204662941880118,
+      "logps/chosen": -137.08301887512206,
+      "logps/rejected": -131.85167694091797,
+      "loss": 0.009242907166481018,
+      "mean_token_accuracy": 0.8156318858265876,
+      "num_tokens": 108732.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.8932553085498514,
+      "rewards/margins": 5.934962052106857,
+      "rewards/rejected": -2.041706770658493,
+      "step": 55
+    },
+    {
+      "entropy": 0.37963976208120587,
+      "epoch": 1.3333333333333333,
+      "grad_norm": 0.11181640625,
+      "learning_rate": 1.125925925925926e-05,
+      "logits/chosen": -5.041228718573622,
+      "logits/rejected": -5.33423051260696,
+      "logps/chosen": -141.2979410171509,
+      "logps/rejected": -142.37700386047362,
+      "loss": 0.0046984534710645676,
+      "mean_token_accuracy": 0.8114737182855606,
+      "num_tokens": 118755.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.4130684825591744,
+      "rewards/margins": 6.152735310792923,
+      "rewards/rejected": -2.73966683447361,
+      "step": 60
+    },
+    {
+      "entropy": 0.46611395105719566,
+      "epoch": 1.4444444444444444,
+      "grad_norm": 0.34375,
+      "learning_rate": 1.0518518518518519e-05,
+      "logits/chosen": -4.849528463953343,
+      "logits/rejected": -5.256960167783497,
+      "logps/chosen": -147.61933441162108,
+      "logps/rejected": -140.91876430511473,
+      "loss": 0.006713083386421204,
+      "mean_token_accuracy": 0.7804476261138916,
+      "num_tokens": 128631.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.9182209906168284,
+      "rewards/margins": 5.857921981811524,
+      "rewards/rejected": -1.9397010073065757,
+      "step": 65
+    },
+    {
+      "entropy": 0.2871916053816676,
+      "epoch": 1.5555555555555556,
+      "grad_norm": 0.059326171875,
+      "learning_rate": 9.777777777777779e-06,
+      "logits/chosen": -5.2217503776900935,
+      "logits/rejected": -5.593685073313231,
+      "logps/chosen": -102.35045728683471,
+      "logps/rejected": -145.3573398590088,
+      "loss": 0.003894594311714172,
+      "mean_token_accuracy": 0.8443350821733475,
+      "num_tokens": 138274.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.543359859660268,
+      "rewards/margins": 6.292724764347076,
+      "rewards/rejected": -3.7493649244308473,
+      "step": 70
+    },
+    {
+      "entropy": 0.4787766240537167,
+      "epoch": 1.6666666666666665,
+      "grad_norm": 0.06640625,
+      "learning_rate": 9.037037037037037e-06,
+      "logits/chosen": -4.776332851292631,
+      "logits/rejected": -5.186666890664824,
+      "logps/chosen": -157.0829038619995,
+      "logps/rejected": -139.26881351470948,
+      "loss": 0.005699554830789566,
+      "mean_token_accuracy": 0.782428502291441,
+      "num_tokens": 148499.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.594912226591259,
+      "rewards/margins": 6.162280440330505,
+      "rewards/rejected": -1.5673681855201722,
+      "step": 75
+    },
+    {
+      "entropy": 0.44437767583876847,
+      "epoch": 1.7777777777777777,
+      "grad_norm": 0.11181640625,
+      "learning_rate": 8.296296296296297e-06,
+      "logits/chosen": -4.855933510461718,
+      "logits/rejected": -5.266161387721932,
+      "logps/chosen": -159.03973083496095,
+      "logps/rejected": -134.62970504760742,
+      "loss": 0.005957455560564995,
+      "mean_token_accuracy": 0.7810968108475208,
+      "num_tokens": 158605.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.2852000228129326,
+      "rewards/margins": 6.133232879638672,
+      "rewards/rejected": -1.8480328619480133,
+      "step": 80
+    },
+    {
+      "entropy": 0.37488885726779697,
+      "epoch": 1.8888888888888888,
+      "grad_norm": 0.05712890625,
+      "learning_rate": 7.555555555555556e-06,
+      "logits/chosen": -5.033377405184649,
+      "logits/rejected": -5.268708712912721,
+      "logps/chosen": -130.94030866622924,
+      "logps/rejected": -136.1019260406494,
+      "loss": 0.003259475901722908,
+      "mean_token_accuracy": 0.8084902346134186,
+      "num_tokens": 168481.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.99751735040918,
+      "rewards/margins": 6.223078119754791,
+      "rewards/rejected": -2.2255607724189757,
+      "step": 85
+    },
+    {
+      "entropy": 0.3812096565961838,
+      "epoch": 2.0,
+      "grad_norm": 0.1748046875,
+      "learning_rate": 6.814814814814815e-06,
+      "logits/chosen": -5.115266960797761,
+      "logits/rejected": -5.4317961796590755,
+      "logps/chosen": -123.69483127593995,
+      "logps/rejected": -134.99574127197266,
+      "loss": 0.005838125199079514,
+      "mean_token_accuracy": 0.8094184212386608,
+      "num_tokens": 178156.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.6110342731699347,
+      "rewards/margins": 6.127008867263794,
+      "rewards/rejected": -2.515974608063698,
+      "step": 90
+    },
+    {
+      "entropy": 0.48049051789566877,
+      "epoch": 2.111111111111111,
+      "grad_norm": 0.06298828125,
+      "learning_rate": 6.0740740740740745e-06,
+      "logits/chosen": -4.754851989808321,
+      "logits/rejected": -5.108402855816343,
+      "logps/chosen": -150.79896955490113,
+      "logps/rejected": -137.2414821624756,
+      "loss": 0.0049744863063097,
+      "mean_token_accuracy": 0.7772506453096867,
+      "num_tokens": 188087.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.416545544657856,
+      "rewards/margins": 6.014387732744217,
+      "rewards/rejected": -1.5978421807289123,
+      "step": 95
+    },
+    {
+      "entropy": 0.3937450809404254,
+      "epoch": 2.2222222222222223,
+      "grad_norm": 0.04931640625,
+      "learning_rate": 5.333333333333334e-06,
+      "logits/chosen": -4.987015212180383,
+      "logits/rejected": -5.31882650873857,
+      "logps/chosen": -147.39776992797852,
+      "logps/rejected": -140.32675018310547,
+      "loss": 0.0032551392912864687,
+      "mean_token_accuracy": 0.8052853919565678,
+      "num_tokens": 198181.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.012734616734088,
+      "rewards/margins": 6.379687869548798,
+      "rewards/rejected": -2.3669532597064973,
+      "step": 100
+    },
+    {
+      "entropy": 0.47445099698379634,
+      "epoch": 2.3333333333333335,
+      "grad_norm": 0.2197265625,
+      "learning_rate": 4.592592592592593e-06,
+      "logits/chosen": -4.872802370742442,
+      "logits/rejected": -5.133891665969566,
+      "logps/chosen": -148.45297689437865,
+      "logps/rejected": -136.64771842956543,
+      "loss": 0.004790737852454185,
+      "mean_token_accuracy": 0.7806533187627792,
+      "num_tokens": 208026.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.906725855730474,
+      "rewards/margins": 6.092754209041596,
+      "rewards/rejected": -1.1860283434391021,
+      "step": 105
+    },
+    {
+      "entropy": 0.2486900213174522,
+      "epoch": 2.4444444444444446,
+      "grad_norm": 0.054443359375,
+      "learning_rate": 3.851851851851852e-06,
+      "logits/chosen": -5.419191984928265,
+      "logits/rejected": -5.653281491846501,
+      "logps/chosen": -103.91493577957154,
+      "logps/rejected": -145.38850498199463,
+      "loss": 0.0024279681965708733,
+      "mean_token_accuracy": 0.8548816680908203,
+      "num_tokens": 217698.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 2.2315581436501817,
+      "rewards/margins": 6.524952787160873,
+      "rewards/rejected": -4.293394649028778,
+      "step": 110
+    },
+    {
+      "entropy": 0.4003423499874771,
+      "epoch": 2.5555555555555554,
+      "grad_norm": 0.1787109375,
+      "learning_rate": 3.1111111111111116e-06,
+      "logits/chosen": -5.113867099840315,
+      "logits/rejected": -5.288849612980478,
+      "logps/chosen": -137.69256296157837,
+      "logps/rejected": -131.14210166931153,
+      "loss": 0.004194004833698273,
+      "mean_token_accuracy": 0.7987362958490849,
+      "num_tokens": 227587.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.299069392681122,
+      "rewards/margins": 6.199445736408234,
+      "rewards/rejected": -1.9003763258457185,
+      "step": 115
+    },
+    {
+      "entropy": 0.3460330101661384,
+      "epoch": 2.6666666666666665,
+      "grad_norm": 0.1083984375,
+      "learning_rate": 2.3703703703703707e-06,
+      "logits/chosen": -5.148864203073933,
+      "logits/rejected": -5.420440423279305,
+      "logps/chosen": -123.47473087310792,
+      "logps/rejected": -140.37193145751954,
+      "loss": 0.003929775208234787,
+      "mean_token_accuracy": 0.821786867082119,
+      "num_tokens": 237387.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.0951911724638195,
+      "rewards/margins": 6.3194074630737305,
+      "rewards/rejected": -3.224216267466545,
+      "step": 120
+    },
+    {
+      "entropy": 0.4868515723384917,
+      "epoch": 2.7777777777777777,
+      "grad_norm": 0.2421875,
+      "learning_rate": 1.62962962962963e-06,
+      "logits/chosen": -4.745206078239351,
+      "logits/rejected": -5.18349850424564,
+      "logps/chosen": -150.0078001976013,
+      "logps/rejected": -140.4278434753418,
+      "loss": 0.006099024042487145,
+      "mean_token_accuracy": 0.7751828819513321,
+      "num_tokens": 247274.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.6764592288061975,
+      "rewards/margins": 5.9473675191402435,
+      "rewards/rejected": -2.2709083050489425,
+      "step": 125
+    },
+    {
+      "entropy": 0.36328242216259243,
+      "epoch": 2.888888888888889,
+      "grad_norm": 0.08642578125,
+      "learning_rate": 8.88888888888889e-07,
+      "logits/chosen": -5.08842414738806,
+      "logits/rejected": -5.358179073880636,
+      "logps/chosen": -131.02277193069457,
+      "logps/rejected": -134.9316036224365,
+      "loss": 0.004048585146665573,
+      "mean_token_accuracy": 0.8177572898566723,
+      "num_tokens": 257217.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.8732570334337653,
+      "rewards/margins": 6.225375068187714,
+      "rewards/rejected": -2.352118057012558,
+      "step": 130
+    },
+    {
+      "entropy": 0.29574637319892644,
+      "epoch": 3.0,
+      "grad_norm": 0.050048828125,
+      "learning_rate": 1.4814814814814817e-07,
+      "logits/chosen": -5.15853156240503,
+      "logits/rejected": -5.602498101789573,
+      "logps/chosen": -116.41512088775634,
+      "logps/rejected": -141.08580684661865,
+      "loss": 0.0035273078829050064,
+      "mean_token_accuracy": 0.842218890786171,
+      "num_tokens": 267234.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.174783520924393,
+      "rewards/margins": 6.486308968067169,
+      "rewards/rejected": -3.3115254521369932,
+      "step": 135
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 135,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 135,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6697852104887424.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-135/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47285f48295cccd09ff2fa37ed4bd4b356be28d14c303fb4348526056285cccd
+size 5905

checkpoint-50/README.md

@@ -0,0 +1,209 @@
+---
+base_model: data/models/aria-ft-sft
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:data/models/aria-ft-sft
+- dpo
+- lora
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

checkpoint-50/adapter_config.json

@@ -0,0 +1,43 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "data/models/aria-ft-sft",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-50/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a1c734567eeb6976435520f0d11c9dba0e97242a9f7147c70e90a9f15cea484
+size 12931296

checkpoint-50/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

checkpoint-50/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d293fde1260fed314f39d13b4d1eab8fd821faf1d47c213b48d5ba6dd45e546
+size 17942731

checkpoint-50/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8e2011629d8bed3ef560fa11175cac55684c4e12a72634bb24abf767b6c7399
+size 14645

checkpoint-50/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:623ca1baebe375ebca31f69d12d0e047e25a5e6b4b922c4848649c2d9ca4d6b6
+size 1465

checkpoint-50/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726
+size 33384567

checkpoint-50/tokenizer_config.json

@@ -0,0 +1,25 @@
+{
+  "backend": "tokenizers",
+  "boi_token": "<start_of_image>",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eoi_token": "<end_of_image>",
+  "eos_token": "<eos>",
+  "image_token": "<image_soft_token>",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "boi_token": "<start_of_image>",
+    "eoi_token": "<end_of_image>",
+    "image_token": "<image_soft_token>"
+  },
+  "pad_token": "<pad>",
+  "processor_class": "Gemma3Processor",
+  "sp_model_kwargs": null,
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-50/trainer_state.json

@@ -0,0 +1,214 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.344827586206897,
+  "eval_steps": 500,
+  "global_step": 50,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 0.7854079902172089,
+      "epoch": 0.3448275862068966,
+      "grad_norm": 17.5,
+      "learning_rate": 1.8400000000000003e-05,
+      "logits/chosen": -4.038567820069238,
+      "logits/rejected": -4.130903336062028,
+      "logps/chosen": -476.98968200683595,
+      "logps/rejected": -175.084077835083,
+      "loss": 0.6134438991546631,
+      "mean_token_accuracy": 0.5670168288052082,
+      "num_tokens": 11659.0,
+      "rewards/accuracies": 0.65,
+      "rewards/chosen": 0.2687527514062822,
+      "rewards/margins": 0.18279968351125717,
+      "rewards/rejected": 0.08595306589268148,
+      "step": 5
+    },
+    {
+      "entropy": 0.7794478192925454,
+      "epoch": 0.6896551724137931,
+      "grad_norm": 6.53125,
+      "learning_rate": 1.64e-05,
+      "logits/chosen": -3.932353105615328,
+      "logits/rejected": -4.069504129448527,
+      "logps/chosen": -456.5636215209961,
+      "logps/rejected": -173.76006660461425,
+      "loss": 0.2986054182052612,
+      "mean_token_accuracy": 0.5806230813264847,
+      "num_tokens": 23174.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.5439212292432785,
+      "rewards/margins": 1.1550721026957036,
+      "rewards/rejected": 0.38884911630302665,
+      "step": 10
+    },
+    {
+      "entropy": 0.804901111457083,
+      "epoch": 1.0,
+      "grad_norm": 3.109375,
+      "learning_rate": 1.4400000000000001e-05,
+      "logits/chosen": -4.006123838245866,
+      "logits/rejected": -4.28639834349455,
+      "logps/chosen": -468.32677798801,
+      "logps/rejected": -175.75607766045465,
+      "loss": 0.10222437381744384,
+      "mean_token_accuracy": 0.5784292072057724,
+      "num_tokens": 33972.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.383368664317661,
+      "rewards/margins": 2.3637701272964478,
+      "rewards/rejected": 1.0195985275010269,
+      "step": 15
+    },
+    {
+      "entropy": 0.8101444244384766,
+      "epoch": 1.3448275862068966,
+      "grad_norm": 0.53125,
+      "learning_rate": 1.2400000000000002e-05,
+      "logits/chosen": -3.9039807304454768,
+      "logits/rejected": -4.13752561066084,
+      "logps/chosen": -456.7418830871582,
+      "logps/rejected": -169.44639072418212,
+      "loss": 0.028239738941192628,
+      "mean_token_accuracy": 0.5729322396218777,
+      "num_tokens": 45734.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 5.294327348470688,
+      "rewards/margins": 3.822002524137497,
+      "rewards/rejected": 1.4723248168826104,
+      "step": 20
+    },
+    {
+      "entropy": 0.8125152215361595,
+      "epoch": 1.6896551724137931,
+      "grad_norm": 0.423828125,
+      "learning_rate": 1.04e-05,
+      "logits/chosen": -3.875419017637917,
+      "logits/rejected": -4.087594791894558,
+      "logps/chosen": -437.49902725219727,
+      "logps/rejected": -170.6690534591675,
+      "loss": 0.011206220090389251,
+      "mean_token_accuracy": 0.5829927772283554,
+      "num_tokens": 57498.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 6.862721538543701,
+      "rewards/margins": 4.855097621679306,
+      "rewards/rejected": 2.0076238617300985,
+      "step": 25
+    },
+    {
+      "entropy": 0.8494062589274513,
+      "epoch": 2.0,
+      "grad_norm": 0.181640625,
+      "learning_rate": 8.400000000000001e-06,
+      "logits/chosen": -3.7918020422260117,
+      "logits/rejected": -3.9911775349085037,
+      "logps/chosen": -434.0316925048828,
+      "logps/rejected": -163.91968239678278,
+      "loss": 0.006641139835119247,
+      "mean_token_accuracy": 0.5838068723678589,
+      "num_tokens": 67944.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 7.484723581208123,
+      "rewards/margins": 5.497186402479808,
+      "rewards/rejected": 1.987537168794208,
+      "step": 30
+    },
+    {
+      "entropy": 0.8175499409437179,
+      "epoch": 2.344827586206897,
+      "grad_norm": 0.150390625,
+      "learning_rate": 6.4000000000000006e-06,
+      "logits/chosen": -3.7909739277903443,
+      "logits/rejected": -3.994593238139848,
+      "logps/chosen": -436.38923797607424,
+      "logps/rejected": -167.69779472351075,
+      "loss": 0.004208391904830933,
+      "mean_token_accuracy": 0.5816447250545025,
+      "num_tokens": 79630.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 8.280532884597779,
+      "rewards/margins": 5.944004595279694,
+      "rewards/rejected": 2.3365282833576204,
+      "step": 35
+    },
+    {
+      "entropy": 0.8197228327393532,
+      "epoch": 2.689655172413793,
+      "grad_norm": 0.1171875,
+      "learning_rate": 4.4e-06,
+      "logits/chosen": -3.830608885995658,
+      "logits/rejected": -4.153223704155633,
+      "logps/chosen": -436.25521926879884,
+      "logps/rejected": -168.31839504241944,
+      "loss": 0.0030239127576351167,
+      "mean_token_accuracy": 0.5829450696706772,
+      "num_tokens": 91583.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 8.960859537124634,
+      "rewards/margins": 6.313286983966828,
+      "rewards/rejected": 2.647572535276413,
+      "step": 40
+    },
+    {
+      "entropy": 0.8821523437897364,
+      "epoch": 3.0,
+      "grad_norm": 0.244140625,
+      "learning_rate": 2.4000000000000003e-06,
+      "logits/chosen": -3.7844727586007707,
+      "logits/rejected": -3.9654474595462568,
+      "logps/chosen": -423.0563507080078,
+      "logps/rejected": -158.2893812391493,
+      "loss": 0.003181111440062523,
+      "mean_token_accuracy": 0.5783860153622098,
+      "num_tokens": 101916.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 8.829733226034376,
+      "rewards/margins": 6.476470510164897,
+      "rewards/rejected": 2.3532627125581107,
+      "step": 45
+    },
+    {
+      "entropy": 0.8417329847812652,
+      "epoch": 3.344827586206897,
+      "grad_norm": 0.1025390625,
+      "learning_rate": 4.0000000000000003e-07,
+      "logits/chosen": -3.809344702093694,
+      "logits/rejected": -3.9889677666123213,
+      "logps/chosen": -433.3476905822754,
+      "logps/rejected": -160.33308601379395,
+      "loss": 0.0024677075445652006,
+      "mean_token_accuracy": 0.5838818199932575,
+      "num_tokens": 113677.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 9.033029174804687,
+      "rewards/margins": 6.4824470520019535,
+      "rewards/rejected": 2.5505820989608763,
+      "step": 50
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 50,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 50,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3365612085540096.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-50/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c4d3ba8f073034f7ae52cc3d73c7001e6b026581b8aba58472337868ff96814
+size 5905

checkpoint-60/README.md

@@ -0,0 +1,209 @@
+---
+base_model: data/models/aria-ft-sft
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:data/models/aria-ft-sft
+- dpo
+- lora
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

checkpoint-60/adapter_config.json

@@ -0,0 +1,43 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "data/models/aria-ft-sft",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-60/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff0f0d8c2343b74298d096bb79ab7a914aa7ecfa0ef7680487881e4e1fbf8b77
+size 12931296

checkpoint-60/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

checkpoint-60/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4dc5145afe4da63b2c11960d304acd8207d3bdfd3549a5a84e95ff49270adb1a
+size 17942731

checkpoint-60/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4a9f217e852f439efa6bd32fde98d6867f11aa6ea13ddc021ba10af6a0b0934
+size 14645

checkpoint-60/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2dd468a12037943abc731ea6bb19f58d78121bbf9a8793bca485cff41de78943
+size 1465

checkpoint-60/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726
+size 33384567

checkpoint-60/tokenizer_config.json

@@ -0,0 +1,25 @@
+{
+  "backend": "tokenizers",
+  "boi_token": "<start_of_image>",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eoi_token": "<end_of_image>",
+  "eos_token": "<eos>",
+  "image_token": "<image_soft_token>",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "boi_token": "<start_of_image>",
+    "eoi_token": "<end_of_image>",
+    "image_token": "<image_soft_token>"
+  },
+  "pad_token": "<pad>",
+  "processor_class": "Gemma3Processor",
+  "sp_model_kwargs": null,
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-60/trainer_state.json

@@ -0,0 +1,250 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 2.888888888888889,
+  "eval_steps": 500,
+  "global_step": 60,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 0.4776241024956107,
+      "epoch": 0.24691358024691357,
+      "grad_norm": 16.0,
+      "learning_rate": 1.866666666666667e-05,
+      "logits/chosen": -4.089478311277062,
+      "logits/rejected": -4.493732379473381,
+      "logps/chosen": -268.6658588409424,
+      "logps/rejected": -150.31285419464112,
+      "loss": 0.6501077651977539,
+      "mean_token_accuracy": 0.7079996474087238,
+      "num_tokens": 10253.0,
+      "rewards/accuracies": 0.55,
+      "rewards/chosen": 0.1445553233847022,
+      "rewards/margins": 0.09593124534003436,
+      "rewards/rejected": 0.04862407729960978,
+      "step": 5
+    },
+    {
+      "entropy": 0.48156379014253614,
+      "epoch": 0.49382716049382713,
+      "grad_norm": 6.3125,
+      "learning_rate": 1.7e-05,
+      "logits/chosen": -4.0623599267958195,
+      "logits/rejected": -4.4711698133175855,
+      "logps/chosen": -271.1767599105835,
+      "logps/rejected": -154.23251075744628,
+      "loss": 0.4772006034851074,
+      "mean_token_accuracy": 0.7104790471494198,
+      "num_tokens": 20666.0,
+      "rewards/accuracies": 0.975,
+      "rewards/chosen": 0.7345219888840802,
+      "rewards/margins": 0.5422453716397285,
+      "rewards/rejected": 0.19227661472395993,
+      "step": 10
+    },
+    {
+      "entropy": 0.5128806505352259,
+      "epoch": 0.7407407407407407,
+      "grad_norm": 4.5625,
+      "learning_rate": 1.5333333333333334e-05,
+      "logits/chosen": -3.846199988066794,
+      "logits/rejected": -4.348598556794308,
+      "logps/chosen": -241.72016925811766,
+      "logps/rejected": -152.45027236938478,
+      "loss": 0.3331197023391724,
+      "mean_token_accuracy": 0.701729304343462,
+      "num_tokens": 30640.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.3711295807734132,
+      "rewards/margins": 1.0514055285602808,
+      "rewards/rejected": 0.319724050629884,
+      "step": 15
+    },
+    {
+      "entropy": 0.4515763284638524,
+      "epoch": 0.9876543209876543,
+      "grad_norm": 2.78125,
+      "learning_rate": 1.3666666666666667e-05,
+      "logits/chosen": -3.9408105831493843,
+      "logits/rejected": -4.458051202170762,
+      "logps/chosen": -216.35807037353516,
+      "logps/rejected": -137.70420036315917,
+      "loss": 0.2393646001815796,
+      "mean_token_accuracy": 0.7314693063497544,
+      "num_tokens": 40658.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 1.9342670194804668,
+      "rewards/margins": 1.5497536167502404,
+      "rewards/rejected": 0.3845134010305628,
+      "step": 20
+    },
+    {
+      "entropy": 0.533805785152842,
+      "epoch": 1.1975308641975309,
+      "grad_norm": 1.578125,
+      "learning_rate": 1.2e-05,
+      "logits/chosen": -3.8604098323676954,
+      "logits/rejected": -4.454983181046675,
+      "logps/chosen": -256.26211334677305,
+      "logps/rejected": -149.4621139975155,
+      "loss": 0.11001660823822021,
+      "mean_token_accuracy": 0.6833898074486676,
+      "num_tokens": 49152.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.127581123043509,
+      "rewards/margins": 2.5184768157846786,
+      "rewards/rejected": 0.6091043011230581,
+      "step": 25
+    },
+    {
+      "entropy": 0.4583894399926066,
+      "epoch": 1.4444444444444444,
+      "grad_norm": 1.5859375,
+      "learning_rate": 1.0333333333333335e-05,
+      "logits/chosen": -3.880995288681619,
+      "logits/rejected": -4.38205130490537,
+      "logps/chosen": -222.737411403656,
+      "logps/rejected": -146.73333396911622,
+      "loss": 0.11508327722549438,
+      "mean_token_accuracy": 0.7408040933310985,
+      "num_tokens": 59335.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.002569316327572,
+      "rewards/margins": 2.6181753650307655,
+      "rewards/rejected": 0.3843939293175936,
+      "step": 30
+    },
+    {
+      "entropy": 0.5070892035961151,
+      "epoch": 1.691358024691358,
+      "grad_norm": 0.9453125,
+      "learning_rate": 8.666666666666668e-06,
+      "logits/chosen": -3.916580181348936,
+      "logits/rejected": -4.275795278703906,
+      "logps/chosen": -251.3110230445862,
+      "logps/rejected": -150.10150470733643,
+      "loss": 0.06905438899993896,
+      "mean_token_accuracy": 0.7120013549923897,
+      "num_tokens": 69532.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.733099599182606,
+      "rewards/margins": 3.172137904167175,
+      "rewards/rejected": 0.5609616965055466,
+      "step": 35
+    },
+    {
+      "entropy": 0.5159024139866233,
+      "epoch": 1.9382716049382716,
+      "grad_norm": 0.86328125,
+      "learning_rate": 7e-06,
+      "logits/chosen": -3.875394947759234,
+      "logits/rejected": -4.348108818548195,
+      "logps/chosen": -240.79218463897706,
+      "logps/rejected": -143.1299747467041,
+      "loss": 0.0537603497505188,
+      "mean_token_accuracy": 0.7110521957278252,
+      "num_tokens": 79802.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.002222263813019,
+      "rewards/margins": 3.4601644307374952,
+      "rewards/rejected": 0.5420578181743622,
+      "step": 40
+    },
+    {
+      "entropy": 0.42990265786647797,
+      "epoch": 2.148148148148148,
+      "grad_norm": 1.203125,
+      "learning_rate": 5.333333333333334e-06,
+      "logits/chosen": -4.000936708220123,
+      "logits/rejected": -4.378904186986099,
+      "logps/chosen": -205.56845799614402,
+      "logps/rejected": -148.97788418040557,
+      "loss": 0.05299478769302368,
+      "mean_token_accuracy": 0.7551224529743195,
+      "num_tokens": 88253.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.6734568196184494,
+      "rewards/margins": 3.4625525684917675,
+      "rewards/rejected": 0.21090425112668207,
+      "step": 45
+    },
+    {
+      "entropy": 0.5233618581667543,
+      "epoch": 2.3950617283950617,
+      "grad_norm": 0.45703125,
+      "learning_rate": 3.6666666666666666e-06,
+      "logits/chosen": -3.935984155012784,
+      "logits/rejected": -4.419382672710851,
+      "logps/chosen": -265.77395582199097,
+      "logps/rejected": -150.95017681121826,
+      "loss": 0.02822282612323761,
+      "mean_token_accuracy": 0.691740982979536,
+      "num_tokens": 98748.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.833640116453171,
+      "rewards/margins": 4.224490064382553,
+      "rewards/rejected": 0.6091500520706177,
+      "step": 50
+    },
+    {
+      "entropy": 0.4936262046918273,
+      "epoch": 2.6419753086419755,
+      "grad_norm": 0.625,
+      "learning_rate": 2.0000000000000003e-06,
+      "logits/chosen": -3.8994168509171785,
+      "logits/rejected": -4.323522702714263,
+      "logps/chosen": -232.54643173217772,
+      "logps/rejected": -147.94255752563475,
+      "loss": 0.0374829888343811,
+      "mean_token_accuracy": 0.7230880469083786,
+      "num_tokens": 108955.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 4.045481747388839,
+      "rewards/margins": 3.670798897743225,
+      "rewards/rejected": 0.37468285113573074,
+      "step": 55
+    },
+    {
+      "entropy": 0.514755728840828,
+      "epoch": 2.888888888888889,
+      "grad_norm": 0.5859375,
+      "learning_rate": 3.3333333333333335e-07,
+      "logits/chosen": -3.7785729406360007,
+      "logits/rejected": -4.339876007693261,
+      "logps/chosen": -217.19937067031861,
+      "logps/rejected": -144.4653621673584,
+      "loss": 0.035057461261749266,
+      "mean_token_accuracy": 0.7147957846522331,
+      "num_tokens": 118789.0,
+      "rewards/accuracies": 1.0,
+      "rewards/chosen": 3.8464197725057603,
+      "rewards/margins": 3.7014809250831604,
+      "rewards/rejected": 0.1449388176202774,
+      "step": 60
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 60,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 60,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3084171422870016.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-60/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3c9c2929c024c0798b7087a9dfc9f46739974e884cc396662d26917da204af9
+size 5905

config.json

@@ -0,0 +1,109 @@
+{
+  "architectures": [
+    "Gemma3ForConditionalGeneration"
+  ],
+  "boi_token_index": 255999,
+  "dtype": "bfloat16",
+  "eoi_token_index": 256000,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "image_token_index": 262144,
+  "initializer_range": 0.02,
+  "mm_tokens_per_image": 256,
+  "model_type": "gemma3",
+  "text_config": {
+    "_sliding_window_pattern": 6,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": null,
+    "bos_token_id": 2,
+    "dtype": "bfloat16",
+    "eos_token_id": 1,
+    "final_logit_softcapping": null,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 2560,
+    "initializer_range": 0.02,
+    "intermediate_size": 10240,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention"
+    ],
+    "max_position_embeddings": 131072,
+    "model_type": "gemma3_text",
+    "num_attention_heads": 8,
+    "num_hidden_layers": 34,
+    "num_key_value_heads": 4,
+    "pad_token_id": 0,
+    "query_pre_attn_scalar": 256,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "full_attention": {
+        "factor": 8.0,
+        "rope_theta": 1000000,
+        "rope_type": "linear"
+      },
+      "sliding_attention": {
+        "rope_theta": 10000,
+        "rope_type": "default"
+      }
+    },
+    "sliding_window": 1024,
+    "tie_word_embeddings": true,
+    "use_bidirectional_attention": false,
+    "use_cache": true,
+    "vocab_size": 262208
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.7.0",
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "image_size": 896,
+    "intermediate_size": 4304,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "patch_size": 14,
+    "vision_use_head": false
+  },
+  "torch_dtype": "bfloat16"
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.7.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6127d07a766c61dafc93360dda4537d62fe83745fe985a6ec59ab0f098a11149
+size 8600278008