Instructions to use ali1627/test_experiment_small_model with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ali1627/test_experiment_small_model with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ali1627/test_experiment_small_model")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ali1627/test_experiment_small_model")
model = AutoModelForCausalLM.from_pretrained("ali1627/test_experiment_small_model", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ali1627/test_experiment_small_model with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ali1627/test_experiment_small_model"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ali1627/test_experiment_small_model",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ali1627/test_experiment_small_model

SGLang

How to use ali1627/test_experiment_small_model with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ali1627/test_experiment_small_model" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ali1627/test_experiment_small_model",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ali1627/test_experiment_small_model" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ali1627/test_experiment_small_model",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ali1627/test_experiment_small_model with Docker Model Runner:
```
docker model run hf.co/ali1627/test_experiment_small_model
```

test_experiment_small_model / cfg.yaml

ali1627

Upload cfg.yaml

4070519 over 3 years ago

Raw

History Blame Contribute Delete

2.23 kB

	architecture:
	backbone_dtype: float16
	force_embedding_gradients: false
	gradient_checkpointing: false
	intermediate_dropout: 0.0
	pretrained: true
	pretrained_weights: ''
	augmentation:
	random_parent_probability: 0.0
	skip_parent_probability: 0.0
	token_mask_probability: 0.0
	dataset:
	add_eos_token_to_answer: true
	add_eos_token_to_prompt: true
	answer_column: output
	data_sample: 1.0
	data_sample_choice:
	- Train
	- Validation
	mask_prompt_labels: true
	parent_id_column: None
	prompt_column:
	- instruction
	text_answer_separator: <\|answer\|>
	text_prompt_start: <\|prompt\|>
	train_dataframe: data/user/oasst/train_full.pq
	validation_dataframe: None
	validation_size: 0.01
	validation_strategy: automatic
	environment:
	compile_model: false
	find_unused_parameters: false
	gpus:
	- '0'
	mixed_precision: true
	number_of_workers: 8
	seed: -1
	trust_remote_code: false
	use_fsdp: false
	experiment_name: test_experiment_small_model
	llm_backbone: EleutherAI/pythia-2.8b-deduped
	logging:
	logger: None
	neptune_project: ''
	number_of_texts: 10
	output_directory: output/user/test_experiment_small_model/
	prediction:
	batch_size_inference: 0
	do_sample: false
	max_length_inference: 256
	metric: BLEU
	min_length_inference: 2
	num_beams: 2
	repetition_penalty: 1.2
	stop_tokens: ''
	temperature: 0.3
	problem_type: text_causal_language_modeling
	tokenizer:
	add_prefix_space: false
	add_prompt_answer_tokens: false
	max_length: 512
	max_length_answer: 256
	max_length_prompt: 256
	padding_quantile: 1.0
	training:
	batch_size: 3
	differential_learning_rate: 1.0e-05
	differential_learning_rate_layers: []
	drop_last_batch: true
	epochs: 1
	evaluate_before_training: true
	evaluation_epochs: 1.0
	grad_accumulation: 1
	gradient_clip: 0.0
	learning_rate: 0.0001
	lora: true
	lora_alpha: 16
	lora_dropout: 0.05
	lora_r: 4
	lora_target_modules: ''
	loss_function: CrossEntropy
	optimizer: AdamW
	save_best_checkpoint: false
	schedule: Cosine
	train_validation_data: false
	warmup_epochs: 0.0
	weight_decay: 0.0