Instructions to use nvidia/Llama3-ChatQA-1.5-70B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nvidia/Llama3-ChatQA-1.5-70B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nvidia/Llama3-ChatQA-1.5-70B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nvidia/Llama3-ChatQA-1.5-70B")
model = AutoModelForCausalLM.from_pretrained("nvidia/Llama3-ChatQA-1.5-70B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use nvidia/Llama3-ChatQA-1.5-70B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nvidia/Llama3-ChatQA-1.5-70B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Llama3-ChatQA-1.5-70B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nvidia/Llama3-ChatQA-1.5-70B

SGLang

How to use nvidia/Llama3-ChatQA-1.5-70B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nvidia/Llama3-ChatQA-1.5-70B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Llama3-ChatQA-1.5-70B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nvidia/Llama3-ChatQA-1.5-70B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Llama3-ChatQA-1.5-70B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nvidia/Llama3-ChatQA-1.5-70B with Docker Model Runner:
```
docker model run hf.co/nvidia/Llama3-ChatQA-1.5-70B
```

The Model Stop Engaging in conversation

#14

by Albihany - opened May 16, 2024

Discussion

Albihany

May 16, 2024

Hi,
I'm having issue with the model where it will just stop engaging with the conversation after a little while specially when i use it through obsidian app, i'm also attaching a screenshot where the model just stopped responding through while i was using it through Ollama, i really like the model any suggestions?

zihanliu

NVIDIA org May 16, 2024

Hi,
It looks like you are using our model in the chat scenario where the context is not involved. Did you follow the prompt format for this case in our model card?
It should look like this

System: {System}

User: {Question}

Assistant: {Response}

User: {Question}

Assistant:

Albihany

May 17, 2024

Hi
Honestly, after spending a decent time today chatting with the model through Ollama i had no issue so far and i gotta say i'm impressed with how fast, smart and chatty is the model, However, it still doesn't work well with obsidian neither with Textgen, nor with Copilot plugging's , i have tested many other models with those plugins (including llama3, llava-llama3, llava, llava-llama3) Yet your model is the only model that stop after little while of chatting, or not work at all is yours, i really need this model to work with obsidian, it's the best close to Chat GPT-4 and i can chat with it locally with no limitation, i really need this. i hope you give it a little test with obsidian plugins and tell us what the issue is?

Cheers
ALBI

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment