Instructions to use vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16
# Run inference directly in the terminal:
./llama-cli -hf vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16

Use Docker

docker model run hf.co/vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16

LM Studio
Jan

vLLM

How to use vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16

Ollama
How to use vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF with Ollama:
```
ollama run hf.co/vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16
```
Unsloth Desktop

How to use vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF with Docker Model Runner:
```
docker model run hf.co/vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16
```

Lemonade

How to use vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16

Run and chat with the model

lemonade run user.Muse-Glimmer-30B-ROCmFPX-GGUF-BF16

List all available models

lemonade list

Hermes Agent

How to use vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF:BF16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

vmlinux commited on 30 days ago

Commit

5b7c2bf

verified ·

1 Parent(s): 2bd83a8

Show t/s for every benchmark line

Browse files

Add prompt-processing and output-generation t/s to every artifact and aggregate benchmark row, plus all 15 raw per-prompt throughput measurements.

Files changed (1) hide show

README.md +29 -18

README.md CHANGED Viewed

@@ -44,17 +44,18 @@ files; read the compatibility section before downloading.
 ## Files
-| File | ROCmFPX preset | Size | BPW | iMatrix | Measured decode |
-| --- | --- | ---: | ---: | --- | ---: |
-| `Muse-Glimmer-30B-ROCmFP4.gguf` | `Q4_0_ROCMFP4_STRIX` | 14.17 GiB | 4.36 | Yes | 14.9 tok/s |
-| `Muse-Glimmer-30B-ROCmFP4-Q6-QUALITY.gguf` | `Q4_0_ROCMFP4_COHERENT` | 14.94 GiB | 4.60 | Yes | 14.0 tok/s |
-| `Muse-Glimmer-30B-ROCmFP8.gguf` | `Q8_0_ROCMFPX` | 26.77 GiB | 8.25 | No | 7.8 tok/s |
-| `mmproj-Muse-Glimmer-30B-BF16.gguf` | BF16 vision projector | 3.59 GiB | — | — | — |
-| `Muse-Glimmer-30B-DFlash-ROCmFP4.gguf` | `Q4_0_ROCMFP4_STRIX` drafter | 1.39 GiB | 4.63 | No | 28.3 tok/s¹ |
-| `Muse-Glimmer-30B-DFlash-ROCmFP8.gguf` | `Q8_0_ROCMFPX` drafter | 2.47 GiB | 8.25 | No | 27.2 tok/s¹ |
-¹ End-to-end target decode with `Muse-Glimmer-30B-ROCmFP4.gguf`, DFlash enabled,
-and a six-token draft window; see the benchmark section below.
 Suggested choices:
@@ -200,13 +201,23 @@ greedy decoding, three text prompts, 256 generated tokens per prompt, a
 2,048-token context, flash attention, full `ROCm0` offload, and Q4_0 target and
 draft KV caches. Values are arithmetic means of the three runs.
-| Mode | Draft window | Decode | Speedup | Per-prompt range | Weighted draft acceptance |
-| --- | ---: | ---: | ---: | ---: | ---: |
-| No speculation | — | 13.7 tok/s | 1.00× | 13.7–13.7 | — |
-| DFlash ROCmFP4 | 6 | **28.3 tok/s** | **2.07×** | 24.3–31.8 | 35.8% (519/1,448) |
-| DFlash ROCmFP8 | 6 | 27.2 tok/s | 1.99× | 24.1–33.0 | 35.7% (519/1,454) |
-| DFlash ROCmFP4 | 15 | 24.6 tok/s | 1.80× | 18.5–30.7 | 17.1% (543/3,171) |
-| DFlash ROCmFP8 | 15 | 26.3 tok/s | 1.92× | 19.2–34.6 | 19.5% (563/2,891) |
 This is a small local throughput benchmark, not a universal performance claim.
 Acceptance depends strongly on prompt and generation content. Six draft tokens

 ## Files
+| File | ROCmFPX preset | Size | BPW | iMatrix | Prompt t/s | Output t/s |
+| --- | --- | ---: | ---: | --- | ---: | ---: |
+| `Muse-Glimmer-30B-ROCmFP4.gguf` | `Q4_0_ROCMFP4_STRIX` | 14.17 GiB | 4.36 | Yes | 113.7 | 14.9 |
+| `Muse-Glimmer-30B-ROCmFP4-Q6-QUALITY.gguf` | `Q4_0_ROCMFP4_COHERENT` | 14.94 GiB | 4.60 | Yes | 39.0 | 14.0 |
+| `Muse-Glimmer-30B-ROCmFP8.gguf` | `Q8_0_ROCMFPX` | 26.77 GiB | 8.25 | No | 96.7 | 7.8 |
+| `mmproj-Muse-Glimmer-30B-BF16.gguf` | BF16 vision projector | 3.59 GiB | — | — | 81.7 | 14.9 |
+| `Muse-Glimmer-30B-DFlash-ROCmFP4.gguf` | `Q4_0_ROCMFP4_STRIX` drafter | 1.39 GiB | 4.63 | No | 65.5¹ | **28.3¹** |
+| `Muse-Glimmer-30B-DFlash-ROCmFP8.gguf` | `Q8_0_ROCMFPX` drafter | 2.47 GiB | 8.25 | No | 65.1¹ | 27.2¹ |
+¹ End-to-end target measurement with `Muse-Glimmer-30B-ROCmFP4.gguf`, DFlash
+enabled, and a six-token draft window. These are three-run means; the main-model
+and projector rows are the earlier short smoke tests described under Validation.
 Suggested choices:
 2,048-token context, flash attention, full `ROCm0` offload, and Q4_0 target and
 draft KV caches. Values are arithmetic means of the three runs.
+| Mode | Draft window | Prompt t/s | Output t/s | Speedup | Output range | Weighted draft acceptance |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| No speculation | — | 73.7 | 13.7 | 1.00× | 13.7–13.7 | — |
+| DFlash ROCmFP4 | 6 | 65.5 | **28.3** | **2.07×** | 24.3–31.8 | 35.8% (519/1,448) |
+| DFlash ROCmFP8 | 6 | 65.1 | 27.2 | 1.99× | 24.1–33.0 | 35.7% (519/1,454) |
+| DFlash ROCmFP4 | 15 | 65.6 | 24.6 | 1.80× | 18.5–30.7 | 17.1% (543/3,171) |
+| DFlash ROCmFP8 | 15 | 65.3 | 26.3 | 1.92× | 19.2–34.6 | 19.5% (563/2,891) |
+Raw llama.cpp throughput lines for all 15 benchmark runs:
+| Mode | Window | Prompt 1 `[Prompt \| Output]` | Prompt 2 `[Prompt \| Output]` | Prompt 3 `[Prompt \| Output]` |
+| --- | ---: | ---: | ---: | ---: |
+| No speculation | — | 55.3 t/s \| 13.7 t/s | 83.5 t/s \| 13.7 t/s | 82.4 t/s \| 13.7 t/s |
+| DFlash ROCmFP4 | 6 | 50.3 t/s \| 28.8 t/s | 73.7 t/s \| 24.3 t/s | 72.5 t/s \| 31.8 t/s |
+| DFlash ROCmFP8 | 6 | 50.2 t/s \| 24.6 t/s | 73.4 t/s \| 24.1 t/s | 71.8 t/s \| 33.0 t/s |
+| DFlash ROCmFP4 | 15 | 50.2 t/s \| 24.6 t/s | 74.2 t/s \| 18.5 t/s | 72.5 t/s \| 30.7 t/s |
+| DFlash ROCmFP8 | 15 | 50.4 t/s \| 25.2 t/s | 73.5 t/s \| 19.2 t/s | 72.1 t/s \| 34.6 t/s |
 This is a small local throughput benchmark, not a universal performance claim.
 Acceptance depends strongly on prompt and generation content. Six draft tokens