Instructions to use teknium/OpenHermes-2.5-Mistral-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use teknium/OpenHermes-2.5-Mistral-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="teknium/OpenHermes-2.5-Mistral-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("teknium/OpenHermes-2.5-Mistral-7B")
model = AutoModelForCausalLM.from_pretrained("teknium/OpenHermes-2.5-Mistral-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use teknium/OpenHermes-2.5-Mistral-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "teknium/OpenHermes-2.5-Mistral-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "teknium/OpenHermes-2.5-Mistral-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/teknium/OpenHermes-2.5-Mistral-7B

SGLang

How to use teknium/OpenHermes-2.5-Mistral-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "teknium/OpenHermes-2.5-Mistral-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "teknium/OpenHermes-2.5-Mistral-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "teknium/OpenHermes-2.5-Mistral-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "teknium/OpenHermes-2.5-Mistral-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use teknium/OpenHermes-2.5-Mistral-7B with Docker Model Runner:
```
docker model run hf.co/teknium/OpenHermes-2.5-Mistral-7B
```

Issue with HF transformer tokenizer apply chat template.

#12

by dyoung - opened Dec 6, 2023

Discussion

dyoung

Dec 6, 2023

•

edited Dec 6, 2023

Hello,

I'm not getting a proper prompt string from huggingfaces transformers AutoTokenizer apply_chat_template for this model when using add_generation_prompt=True as a parameter to apply_chat_template.

Here is the code that causes what I'm seeing:

from transformers import AutoTokenizer

MODEL =  "teknium/OpenHermes-2.5-Mistral-7B"

tokenizer = AutoTokenizer.from_pretrained(MODEL ,  skip_special_tokens=True, use_fast=True)

test_msg = [
        {"role": "system", "content": "You are Alex, a friendly AI assistant."},
        {"role": "user", "content": "Hello, how are you?"},
        {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
        {"role": "user", "content": "Tell me a good joke! Not a short one either."},
    ]

prompt_str = tokenizer.apply_chat_template(test_msg, tokenize=False, add_generation_prompt=True)

print(f'"""{prompt_str}"""')

Which outputs:

"""<|im_start|>system
You are Alex, a friendly AI assistant.<|im_end|>
<|im_start|>user
Hello, how are you?<|im_end|>
<|im_start|>assistant
I'm doing great. How can I help you today?<|im_end|>
<|im_start|>user
Tell me a good joke! Not a short one either.<|im_end|>
"""

Notice that its missing "<|im_start|>assistant\n" at the end. Which should be included as noted in the HF documentation: https://huggingface.co/docs/transformers/main/chat_templating#what-are-generation-prompts

I can do a dirty fix by just appending what I need to the end of the prompt string after calling apply_chat_template. But I feel that it would be best that 'add_generation_prompt' worked as intended for this model.

I'm not sure if this is a tokenizer config template that is malformed and accompanying this models repo files, or if it's something with HF's transformer tokenizer library.

I feel that I'm doing things properly and the issue is likely not on my side. If not, I'm happy to hear some feed back.

If I'm doing things properly, I hope that me pointing out this unexpected behavior helps in some way.

Thanks for taking time to read this.

dyoung

Dec 6, 2023

I think I have an idea of what's up. I'm running transformers 4.43.0. 4.45.x is the current stable I think. Likely updating my transformers library will fix this. I'll keep you all posted on how that goes.

dyoung

Dec 6, 2023

Yep, that was the solution to the situation. So I'm good now.
Found this which helped: https://github.com/huggingface/transformers/issues/27370
Would of been nice if I got a warning or raised exception from HF's AutoTokenizer.apply_chat_template. I might say something to them about it.

dyoung changed discussion status to closed Dec 6, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment