Upload RPC-Bench leaderboard Space

README.md

@@ -1,13 +1,44 @@
 ---
-title: Test
-emoji: 📊
-colorFrom: gray
-colorTo: yellow
+title: RPC-Bench Leaderboard
+emoji: 📄
+colorFrom: blue
+colorTo: gray
 sdk: gradio
-sdk_version: 6.19.0
-python_version: '3.13'
+sdk_version: 5.0.0
+python_version: 3.11
 app_file: app.py
 pinned: false
+license: apache-2.0
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# RPC-Bench Leaderboard Space
+
+This Space displays RPC-Bench leaderboard results and documents the public submission format.
+
+The design follows the same high-level pattern as the Terminal-Bench 2.0
+leaderboard repository, but the current RPC-Bench flow is maintainer-run rather
+than webhook-driven:
+
+1. Submitters open a Pull Request to a dataset repository.
+2. Maintainers validate the submitted `metadata.yaml` and `predictions.jsonl`.
+3. The official evaluator runs locally in the maintainers' controlled environment.
+4. Accepted aggregate results are imported into `data/leaderboard_seed.csv`.
+5. This Space displays the updated leaderboard.
+
+RPC-Bench submissions contain prediction files and lightweight metadata, not executable user code. The public Space does not run evaluation.
+
+## Expected Dataset Repository Layout
+
+```text
+submissions/
+  rpc-bench/
+    <organization>__<model>__<input_config>/
+        metadata.yaml
+        predictions.jsonl
+        generation_config.json      # optional
+        artifacts/                  # optional logs, prompts, run notes
+```
+
+## Leaderboard Data Source
+
+The app reads `data/leaderboard_seed.csv` first and falls back to `data/leaderboard.json`. Maintainer-only scripts for fetching submissions, running evaluation, updating the CSV/JSON, and uploading the Space live outside this Space directory in `leaderboard_ops/`.

app.py

@@ -0,0 +1,498 @@
+import html
+import json
+import re
+from pathlib import Path
+
+import gradio as gr
+import pandas as pd
+
+
+ROOT = Path(__file__).parent
+DATA_DIR = ROOT / "data"
+LEADERBOARD_JSON_PATH = DATA_DIR / "leaderboard.json"
+LEADERBOARD_CSV_PATH = DATA_DIR / "leaderboard_seed.csv"
+
+MODEL_LINK_RE = re.compile(r"^\[(?P<name>.*)\]\((?P<url>.*)\)$")
+DISPLAY_COLUMNS = [
+    "Rank",
+    "Model",
+    "Organization",
+    "Input Config",
+    "Conciseness",
+    "Correctness",
+    "Completeness",
+    "F1-like",
+    "Informativeness",
+    "Date",
+]
+NUMERIC_COLUMNS = ["Conciseness", "Correctness", "Completeness", "F1-like", "Informativeness"]
+
+CUSTOM_CSS = """
+footer { display: none !important; }
+.api-docs, .show-api, .built-with, [data-testid="api-info"] { display: none !important; }
+.gradio-container { max-width: 100% !important; padding: 18px 24px 16px !important; }
+#component-0 { max-width: 100% !important; }
+.rpc-title h1 { margin-bottom: 4px !important; }
+.rpc-title p { margin-top: 0 !important; color: #555; }
+.rpc-links {
+  display: flex;
+  justify-content: center;
+  align-items: center;
+  gap: 8px;
+  flex-wrap: wrap;
+  margin: 4px 0 12px;
+  color: #4b5563;
+}
+.rpc-links a { color: #2563eb; text-decoration: none; }
+.rpc-links a:hover { text-decoration: underline; }
+.leaderboard-toolbar {
+  display: flex;
+  align-items: center;
+  justify-content: space-between;
+  gap: 12px;
+  margin: 8px 0 12px;
+  flex-wrap: wrap;
+}
+.config-filter {
+  display: inline-flex;
+  align-items: center;
+  gap: 4px;
+  padding: 3px;
+  border: 1px solid #e5e7eb;
+  border-radius: 8px;
+  background: #f8fafc;
+}
+.config-filter button {
+  border: 0;
+  background: transparent;
+  color: #4b5563;
+  cursor: pointer;
+  font-size: 13px;
+  font-weight: 600;
+  padding: 6px 10px;
+  border-radius: 6px;
+}
+.config-filter button.active {
+  background: #111827;
+  color: #ffffff;
+}
+.table-count {
+  color: #6b7280;
+  font-size: 13px;
+}
+.leaderboard-shell {
+  height: calc(100vh - 245px);
+  min-height: 460px;
+  max-height: 780px;
+  overflow: auto;
+  border: 1px solid #e5e7eb;
+  border-radius: 8px;
+  background: white;
+}
+.rpc-table {
+  width: 100%;
+  border-collapse: separate;
+  border-spacing: 0;
+  font-size: 14px;
+}
+.rpc-table th {
+  position: sticky;
+  top: 0;
+  z-index: 2;
+  background: #f8fafc;
+  color: #111827;
+  font-weight: 650;
+  text-align: left;
+  border-bottom: 1px solid #d1d5db;
+  padding: 9px 11px;
+  white-space: nowrap;
+  cursor: pointer;
+  user-select: none;
+}
+.rpc-table th::after {
+  content: "↕";
+  color: #9ca3af;
+  font-size: 11px;
+  margin-left: 6px;
+}
+.rpc-table th.sort-asc::after { content: "↑"; color: #111827; }
+.rpc-table th.sort-desc::after { content: "↓"; color: #111827; }
+.rpc-table td {
+  border-bottom: 1px solid #eef2f7;
+  padding: 9px 11px;
+  vertical-align: middle;
+  white-space: nowrap;
+}
+.rpc-table tbody tr:hover { background: #f9fafb; }
+.rpc-table .num { text-align: right; font-variant-numeric: tabular-nums; }
+.rpc-table .rank { width: 64px; text-align: right; color: #4b5563; }
+.rpc-table .model { min-width: 210px; font-weight: 600; }
+.rpc-table .org { min-width: 180px; }
+.config-badge {
+  display: inline-flex;
+  align-items: center;
+  justify-content: center;
+  min-width: 58px;
+  padding: 3px 8px;
+  border-radius: 999px;
+  font-size: 12px;
+  font-weight: 700;
+  letter-spacing: 0.02em;
+}
+.config-text {
+  background: #e0f2fe;
+  color: #075985;
+}
+.config-visual {
+  background: #fef3c7;
+  color: #92400e;
+}
+.rpc-table a { color: #2563eb; text-decoration: none; }
+.rpc-table a:hover { text-decoration: underline; }
+.submit-panel { max-width: 980px; }
+.submit-panel pre { border-radius: 8px; }
+"""
+
+
+TABLE_SCRIPT = """
+<script>
+(function () {
+  const table = document.getElementById("rpc-leaderboard-table");
+  if (!table) return;
+
+  const tbody = table.querySelector("tbody");
+  const headers = table.querySelectorAll("th[data-sort]");
+  const filterButtons = document.querySelectorAll(".config-filter button");
+  const countEl = document.getElementById("table-count");
+  let activeConfig = "ALL";
+  let sortColumn = "Informativeness";
+  let sortDirection = "desc";
+
+  function parseValue(row, column) {
+    const cell = row.querySelector(`[data-col="${column}"]`);
+    if (!cell) return "";
+    const raw = cell.getAttribute("data-value") || cell.textContent || "";
+    if (["Rank", "Conciseness", "Correctness", "Completeness", "F1-like", "Informativeness"].includes(column)) {
+      const num = Number.parseFloat(raw);
+      return Number.isNaN(num) ? -Infinity : num;
+    }
+    if (column === "Date") {
+      const time = Date.parse(raw);
+      return Number.isNaN(time) ? 0 : time;
+    }
+    return raw.toLowerCase();
+  }
+
+
+  function apply() {
+    const rows = Array.from(tbody.querySelectorAll("tr"));
+    const sorted = rows.slice().sort((a, b) => {
+      const av = parseValue(a, sortColumn);
+      const bv = parseValue(b, sortColumn);
+      if (av < bv) return sortDirection === "asc" ? -1 : 1;
+      if (av > bv) return sortDirection === "asc" ? 1 : -1;
+      return 0;
+    });
+
+    sorted.forEach(row => tbody.appendChild(row));
+
+    let shown = 0;
+    Array.from(tbody.querySelectorAll("tr")).forEach(row => {
+      const visible = activeConfig === "ALL" || row.dataset.config === activeConfig;
+      row.style.display = visible ? "" : "none";
+      if (visible) {
+        shown += 1;
+        const rankCell = row.querySelector('[data-col="Rank"]');
+        if (rankCell) {
+          rankCell.textContent = shown;
+          rankCell.setAttribute("data-value", String(shown));
+        }
+      }
+    });
+
+    if (countEl) countEl.textContent = `${shown} entries`;
+    headers.forEach(header => {
+      header.classList.remove("sort-asc", "sort-desc");
+      if (header.dataset.sort === sortColumn) {
+        header.classList.add(sortDirection === "asc" ? "sort-asc" : "sort-desc");
+      }
+    });
+  }
+
+  headers.forEach(header => {
+    header.addEventListener("click", () => {
+      const column = header.dataset.sort;
+      if (sortColumn === column) {
+        sortDirection = sortDirection === "asc" ? "desc" : "asc";
+      } else {
+        sortColumn = column;
+        sortDirection = ["Model", "Organization", "Input Config", "Date"].includes(column) ? "asc" : "desc";
+      }
+      apply();
+    });
+  });
+
+  filterButtons.forEach(button => {
+    button.addEventListener("click", () => {
+      activeConfig = button.dataset.config;
+      filterButtons.forEach(item => item.classList.toggle("active", item === button));
+      apply();
+    });
+  });
+
+  apply();
+})();
+</script>
+"""
+
+SUBMISSION_GUIDE = """
+<div class="submit-panel">
+
+### How to Submit
+
+1. Fork <a href="https://huggingface.co/datasets/zai-org/RPC-Bench" target="_blank" rel="noopener noreferrer">this repository</a>.
+2. Create a new branch for your submission.
+3. Add your submission folder under
+   `submissions/rpc-bench/<organization>__<model>__<input_config>/`.
+4. Open a Pull Request with the new submission folder.
+
+### Submission Directory Requirements
+
+Each submission directory must contain the metadata and predictions for one
+model/input configuration pair:
+
+```text
+<organization>__<model>__<input_config>/
+  metadata.yaml
+  predictions.jsonl
+  generation_config.json      # optional, recommended
+  artifacts/                  # optional logs or prompt notes
+```
+
+Use URL-safe directory names. Replace spaces, slashes, and special characters
+with hyphens; keep `input_config` as `TEXT` or `VISUAL`.
+
+### `metadata.yaml`
+
+```yaml
+model_name: "My Model"
+organization: "My Org"
+model_url: https://...        # optional work link: paper, GitHub, model card, etc.
+date: "2026-06-17"           # model release date, not submission date
+split: test
+input_config: TEXT            # TEXT or VISUAL
+```
+
+### `predictions.jsonl`
+
+Each line must be one JSON object:
+
+```json
+{
+  "id": "paper-id",
+  "part_idx": 1,
+  "question": "question text",
+  "category": "category",
+  "gen_answer": "model answer"
+}
+```
+
+`part_idx` is the question index in the current paper's `qa_pairs` list (`1` for the first item). `category` must match the corresponding item in `test.json`.
+
+### Validation Rules
+
+Your submission will be validated before evaluation. To pass:
+
+- `metadata.yaml` must include `model_name`, `organization`, `date`, `split`,
+  and `input_config`.
+- `model_url` is optional.
+- `date` is the model release date, not the submission date.
+- `split` must be `test`.
+- `input_config` must be `TEXT` or `VISUAL`.
+- `predictions.jsonl` must contain exactly one line for every QA item in
+  `test.json`.
+- `part_idx` is the question index in the current paper's `qa_pairs` list
+  (`1` for the first item).
+- `id`, `part_idx`, `question`, and `category` must exactly match the benchmark
+  item.
+- `gen_answer` must be a string.
+- For `Claim_Verification`, `gen_answer` must be exactly `True` or `False`.
+
+### Submission Process
+
+1. Open PR: add your folder under
+   `submissions/rpc-bench/<organization>__<model>__<input_config>/`.
+2. Fix issues: if validation fails, update the PR with corrected files.
+3. Review: once validation passes, a maintainer reviews the submission.
+4. Evaluate: maintainers run the official evaluator in a controlled local
+   environment.
+5. Import: accepted aggregate results are imported to the leaderboard.
+
+</div>
+"""
+
+
+def _parse_markdown_link(value):
+    text = str(value).strip()
+    match = MODEL_LINK_RE.match(text)
+    if match:
+        return match.group("name"), match.group("url")
+    return text, ""
+
+
+def _read_csv_leaderboard():
+    df = pd.read_csv(LEADERBOARD_CSV_PATH)
+    if "Info" in df.columns and "Informativeness" not in df.columns:
+        df = df.rename(columns={"Info": "Informativeness"})
+    names = []
+    urls = []
+    for value in df.get("Model", []):
+        name, url = _parse_markdown_link(value)
+        names.append(name)
+        urls.append(url)
+    if "Model" in df.columns:
+        df["Model"] = names
+        df["url"] = urls
+    for col in NUMERIC_COLUMNS:
+        if col in df.columns:
+            df[col] = pd.to_numeric(df[col], errors="coerce")
+    df = df.sort_values("Informativeness", ascending=False, na_position="last").reset_index(drop=True)
+    df.insert(0, "Rank", range(1, len(df) + 1))
+    return df
+
+
+def _read_json_leaderboard():
+    with LEADERBOARD_JSON_PATH.open("r", encoding="utf-8") as f:
+        data = json.load(f)
+    rows = []
+    for season in data.get("seasons", {}).values():
+        for row in season.get("models", []):
+            rows.append({
+                "Model": row.get("name", ""),
+                "url": row.get("url", ""),
+                "Organization": row.get("org", ""),
+                "Input Config": str(row.get("modality", "")).upper(),
+                "Conciseness": row.get("conciseness", 0),
+                "Correctness": row.get("correctness", 0),
+                "Completeness": row.get("completeness", 0),
+                "F1-like": row.get("f1_like", row.get("informativeness", 0)),
+                "Informativeness": row.get("informativeness", row.get("info", row.get("overall", 0))),
+                "Date": row.get("date", ""),
+            })
+    df = pd.DataFrame(rows)
+    if df.empty:
+        return pd.DataFrame(columns=DISPLAY_COLUMNS + ["url"])
+    df = df.sort_values("Informativeness", ascending=False, na_position="last").reset_index(drop=True)
+    df.insert(0, "Rank", range(1, len(df) + 1))
+    return df
+
+
+def load_leaderboard_table():
+    if LEADERBOARD_CSV_PATH.exists():
+        try:
+            return _read_csv_leaderboard()
+        except Exception:
+            pass
+    return _read_json_leaderboard()
+
+
+def _format_cell(value, column):
+    if pd.isna(value):
+        return ""
+    if column in NUMERIC_COLUMNS:
+        return f"{float(value):.2f}"
+    return html.escape(str(value))
+
+
+def _render_input_config(value):
+    config = str(value).upper()
+    if config == "TEXT":
+        return '<span class="config-badge config-text">TEXT</span>'
+    if config == "VISUAL":
+        return '<span class="config-badge config-visual">VISUAL</span>'
+    return html.escape(config)
+
+
+def render_leaderboard_html():
+    df = load_leaderboard_table()
+    columns = [col for col in DISPLAY_COLUMNS if col in df.columns]
+
+    thead = "".join(
+        f'<th data-sort="{html.escape(col, quote=True)}">{html.escape(col)}</th>'
+        for col in columns
+    )
+    body_rows = []
+    for _, row in df.iterrows():
+        cells = []
+        config_value = str(row.get("Input Config", "")).upper()
+        for col in columns:
+            classes = []
+            if col == "Rank":
+                classes.append("rank")
+            if col == "Model":
+                classes.append("model")
+            if col == "Organization":
+                classes.append("org")
+            if col in NUMERIC_COLUMNS or col == "Rank":
+                classes.append("num")
+            class_attr = f' class="{" ".join(classes)}"' if classes else ""
+            data_value = html.escape(str(row[col]), quote=True)
+            data_col = html.escape(col, quote=True)
+            if col == "Model" and row.get("url"):
+                text = html.escape(str(row[col]))
+                url = html.escape(str(row["url"]), quote=True)
+                value = f'<a href="{url}" target="_blank" rel="noopener noreferrer">{text}</a>'
+            elif col == "Input Config":
+                value = _render_input_config(row[col])
+            else:
+                value = _format_cell(row[col], col)
+            cells.append(f'<td{class_attr} data-col="{data_col}" data-value="{data_value}">{value}</td>')
+        body_rows.append(f'<tr data-config="{html.escape(config_value, quote=True)}">' + "".join(cells) + "</tr>")
+
+    return f"""
+<div class="leaderboard-toolbar">
+  <div class="config-filter" aria-label="Input Config filter">
+    <button type="button" class="active" data-config="ALL">All</button>
+    <button type="button" data-config="TEXT">TEXT</button>
+    <button type="button" data-config="VISUAL">VISUAL</button>
+  </div>
+  <div id="table-count" class="table-count">{len(df)} entries</div>
+</div>
+<div class="leaderboard-shell">
+  <table id="rpc-leaderboard-table" class="rpc-table">
+    <thead><tr>{thead}</tr></thead>
+    <tbody>{''.join(body_rows)}</tbody>
+  </table>
+</div>
+{TABLE_SCRIPT}
+"""
+
+
+
+with gr.Blocks(title="RPC-Bench Leaderboard", analytics_enabled=False, css=CUSTOM_CSS) as demo:
+    gr.Markdown(
+        """
+# RPC-Bench Leaderboard
+
+<div class="rpc-links">
+  <span>🌐 <a href="https://rpc-bench.github.io/" target="_blank" rel="noopener noreferrer">Project Page</a></span>
+  <span>•</span>
+  <span>📖 <a href="https://arxiv.org/abs/2601.14289" target="_blank" rel="noopener noreferrer">Paper</a></span>
+  <span>•</span>
+  <span>🤗 <a href="https://huggingface.co/datasets/zai-org/RPC-Bench" target="_blank" rel="noopener noreferrer">Hugging Face</a></span>
+  <span>•</span>
+  <span>🧭 <a href="https://modelscope.cn/datasets/ZhipuAI/RPC-Bench" target="_blank" rel="noopener noreferrer">ModelScope</a></span>
+</div>
+""",
+        elem_classes=["rpc-title"],
+    )
+
+    with gr.Tab("Leaderboard"):
+        gr.HTML(render_leaderboard_html())
+
+    with gr.Tab("Submit"):
+        gr.Markdown(SUBMISSION_GUIDE)
+
+
+if __name__ == "__main__":
+    demo.launch(show_api=False)

data/leaderboard.json

@@ -0,0 +1,442 @@
+{
+  "benchmark": "rpc-bench",
+  "schema_version": "1.0",
+  "last_updated": "2026-06-17",
+  "source": "data/leaderboard_seed.csv",
+  "seasons": {
+    "default": {
+      "period": "rolling",
+      "dataset_version": "rpc-bench-test-v1",
+      "eval_version": "eval.py@rpc-leaderboard-v1",
+      "judges": {
+        "open_qa": [
+          "gpt-5-2025-08-07",
+          "gemini-2.5-pro"
+        ],
+        "claim_verification": "exact-match"
+      },
+      "models": [
+        {
+          "rank": 1,
+          "name": "GPT-5",
+          "url": "https://openai.com/index/introducing-gpt-5/",
+          "org": "OpenAI",
+          "modality": "text",
+          "date": "2025-8-7",
+          "status": "published",
+          "conciseness": 54.93,
+          "correctness": 69.1,
+          "completeness": 67.33,
+          "f1_like": 68.2,
+          "info": 37.46,
+          "overall": 37.46
+        },
+        {
+          "rank": 2,
+          "name": "GPT-5.2",
+          "url": "https://openai.com/index/introducing-gpt-5-2/",
+          "org": "OpenAI",
+          "modality": "text",
+          "date": "2025-12-11",
+          "status": "published",
+          "conciseness": 53.81,
+          "correctness": 66.84,
+          "completeness": 64.03,
+          "f1_like": 65.4,
+          "info": 35.19,
+          "overall": 35.19
+        },
+        {
+          "rank": 3,
+          "name": "GPT-5",
+          "url": "https://openai.com/index/introducing-gpt-5/",
+          "org": "OpenAI",
+          "modality": "visual",
+          "date": "2025-8-7",
+          "status": "published",
+          "conciseness": 61.47,
+          "correctness": 58.9,
+          "completeness": 55.34,
+          "f1_like": 57.07,
+          "info": 35.08,
+          "overall": 35.08
+        },
+        {
+          "rank": 4,
+          "name": "Gemini-2.5-Pro",
+          "url": "https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/",
+          "org": "Google",
+          "modality": "text",
+          "date": "2025-3-25",
+          "status": "published",
+          "conciseness": 54.87,
+          "correctness": 62.65,
+          "completeness": 59.03,
+          "f1_like": 60.79,
+          "info": 33.35,
+          "overall": 33.35
+        },
+        {
+          "rank": 5,
+          "name": "Gemini-3-Pro",
+          "url": "https://blog.google/products-and-platforms/products/gemini/gemini-3/",
+          "org": "Google",
+          "modality": "text",
+          "date": "2025-11-18",
+          "status": "published",
+          "conciseness": 52.81,
+          "correctness": 62.69,
+          "completeness": 60.28,
+          "f1_like": 61.46,
+          "info": 32.46,
+          "overall": 32.46
+        },
+        {
+          "rank": 6,
+          "name": "DeepSeek-V3.2",
+          "url": "https://api-docs.deepseek.com/news/news251201",
+          "org": "DeepSeek-AI",
+          "modality": "text",
+          "date": "2025-12-1",
+          "status": "published",
+          "conciseness": 56.31,
+          "correctness": 58.73,
+          "completeness": 55.19,
+          "f1_like": 56.91,
+          "info": 32.04,
+          "overall": 32.04
+        },
+        {
+          "rank": 7,
+          "name": "GPT-5.2",
+          "url": "https://openai.com/index/introducing-gpt-5-2/",
+          "org": "OpenAI",
+          "modality": "visual",
+          "date": "2025-12-11",
+          "status": "published",
+          "conciseness": 56.43,
+          "correctness": 56.75,
+          "completeness": 52.82,
+          "f1_like": 54.72,
+          "info": 30.88,
+          "overall": 30.88
+        },
+        {
+          "rank": 8,
+          "name": "DeepSeek-V3.1",
+          "url": "https://api-docs.deepseek.com/news/news250821",
+          "org": "DeepSeek-AI",
+          "modality": "text",
+          "date": "2025-8-21",
+          "status": "published",
+          "conciseness": 54.76,
+          "correctness": 57.85,
+          "completeness": 54.85,
+          "f1_like": 56.31,
+          "info": 30.84,
+          "overall": 30.84
+        },
+        {
+          "rank": 9,
+          "name": "GLM-4.6V",
+          "url": "https://github.com/zai-org/GLM-V",
+          "org": "Z.ai",
+          "modality": "visual",
+          "date": "2025-12-8",
+          "status": "published",
+          "conciseness": 64.55,
+          "correctness": 47.32,
+          "completeness": 43.43,
+          "f1_like": 45.29,
+          "info": 29.23,
+          "overall": 29.23
+        },
+        {
+          "rank": 10,
+          "name": "GLM-4.7",
+          "url": "https://z.ai/blog/glm-4.7",
+          "org": "Z.ai",
+          "modality": "text",
+          "date": "2025-12-22",
+          "status": "published",
+          "conciseness": 54.34,
+          "correctness": 54.36,
+          "completeness": 51.75,
+          "f1_like": 53.02,
+          "info": 28.81,
+          "overall": 28.81
+        },
+        {
+          "rank": 11,
+          "name": "GLM-4.5V",
+          "url": "https://github.com/zai-org/GLM-V",
+          "org": "Z.ai",
+          "modality": "visual",
+          "date": "2025-8-11",
+          "status": "published",
+          "conciseness": 59.44,
+          "correctness": 48.79,
+          "completeness": 43.62,
+          "f1_like": 46.06,
+          "info": 27.38,
+          "overall": 27.38
+        },
+        {
+          "rank": 12,
+          "name": "gemini-3-pro",
+          "url": "https://blog.google/products-and-platforms/products/gemini/gemini-3/",
+          "org": "Google",
+          "modality": "visual",
+          "date": "2025-11-18",
+          "status": "published",
+          "conciseness": 50.22,
+          "correctness": 56.06,
+          "completeness": 52.69,
+          "f1_like": 54.32,
+          "info": 27.28,
+          "overall": 27.28
+        },
+        {
+          "rank": 13,
+          "name": "GLM-4.5",
+          "url": "https://z.ai/blog/glm-4.5",
+          "org": "Z.ai",
+          "modality": "text",
+          "date": "2025-7-28",
+          "status": "published",
+          "conciseness": 43.41,
+          "correctness": 58.95,
+          "completeness": 59.54,
+          "f1_like": 59.24,
+          "info": 25.72,
+          "overall": 25.72
+        },
+        {
+          "rank": 14,
+          "name": "gemini-2.5-pro",
+          "url": "https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/",
+          "org": "Google",
+          "modality": "visual",
+          "date": "2025-3-25",
+          "status": "published",
+          "conciseness": 51.71,
+          "correctness": 48.39,
+          "completeness": 45.59,
+          "f1_like": 46.95,
+          "info": 24.28,
+          "overall": 24.28
+        },
+        {
+          "rank": 15,
+          "name": "Claude-Sonnet-4",
+          "url": "https://www.anthropic.com/news/claude-4",
+          "org": "Anthropic",
+          "modality": "text",
+          "date": "2025-5-23",
+          "status": "published",
+          "conciseness": 41.37,
+          "correctness": 58.53,
+          "completeness": 58.44,
+          "f1_like": 58.48,
+          "info": 24.19,
+          "overall": 24.19
+        },
+        {
+          "rank": 16,
+          "name": "Qwen3",
+          "url": "https://github.com/QwenLM/Qwen3",
+          "org": "Alibaba",
+          "modality": "text",
+          "date": "2025-7-21",
+          "status": "published",
+          "conciseness": 41.44,
+          "correctness": 55.88,
+          "completeness": 56.64,
+          "f1_like": 56.26,
+          "info": 23.31,
+          "overall": 23.31
+        },
+        {
+          "rank": 17,
+          "name": "Claude-Sonnet-4.5",
+          "url": "https://www.anthropic.com/news/claude-sonnet-4-5",
+          "org": "Anthropic",
+          "modality": "text",
+          "date": "2025-9-30",
+          "status": "published",
+          "conciseness": 31.02,
+          "correctness": 64.31,
+          "completeness": 64.97,
+          "f1_like": 64.64,
+          "info": 20.05,
+          "overall": 20.05
+        },
+        {
+          "rank": 18,
+          "name": "Claude-Sonnet-4.5",
+          "url": "https://www.anthropic.com/news/claude-sonnet-4-5",
+          "org": "Anthropic",
+          "modality": "visual",
+          "date": "2025-9-30",
+          "status": "published",
+          "conciseness": 31.95,
+          "correctness": 55.35,
+          "completeness": 54.45,
+          "f1_like": 54.89,
+          "info": 17.54,
+          "overall": 17.54
+        },
+        {
+          "rank": 19,
+          "name": "Claude-Sonnet-4",
+          "url": "https://www.anthropic.com/news/claude-4",
+          "org": "Anthropic",
+          "modality": "visual",
+          "date": "2025-5-23",
+          "status": "published",
+          "conciseness": 31.63,
+          "correctness": 54.16,
+          "completeness": 53.32,
+          "f1_like": 53.74,
+          "info": 16.99,
+          "overall": 16.99
+        },
+        {
+          "rank": 20,
+          "name": "HippoRAG2",
+          "url": "https://github.com/ianliuwd/HippoRAG2",
+          "org": "The Ohio State University",
+          "modality": "text",
+          "date": "2025-6-19",
+          "status": "published",
+          "conciseness": 45.77,
+          "correctness": 33.13,
+          "completeness": 27.88,
+          "f1_like": 30.28,
+          "info": 13.86,
+          "overall": 13.86
+        },
+        {
+          "rank": 21,
+          "name": "MemoRAG",
+          "url": "https://github.com/qhjqhj00/MemoRAG",
+          "org": "Peking University & Hong Kong Polytechnic University",
+          "modality": "text",
+          "date": "2025-4-9",
+          "status": "published",
+          "conciseness": 51.31,
+          "correctness": 24.19,
+          "completeness": 19.1,
+          "f1_like": 21.35,
+          "info": 10.96,
+          "overall": 10.96
+        },
+        {
+          "rank": 22,
+          "name": "VdocRAG",
+          "url": "https://vdocrag.github.io/",
+          "org": "NTT Corporation & Tohoku University",
+          "modality": "visual",
+          "date": "2025-4-14",
+          "status": "published",
+          "conciseness": 61.54,
+          "correctness": 21.17,
+          "completeness": 13.88,
+          "f1_like": 16.77,
+          "info": 10.32,
+          "overall": 10.32
+        },
+        {
+          "rank": 23,
+          "name": "VisRAG",
+          "url": "https://github.com/OpenBMB/VisRAG",
+          "org": "Tsinghua University & ModelBest Inc.",
+          "modality": "visual",
+          "date": "2025-3-2",
+          "status": "published",
+          "conciseness": 39.9,
+          "correctness": 26.24,
+          "completeness": 23.63,
+          "f1_like": 24.87,
+          "info": 9.92,
+          "overall": 9.92
+        },
+        {
+          "rank": 24,
+          "name": "Raptor",
+          "url": "https://github.com/parthsarthi03/raptor",
+          "org": "Stanford University",
+          "modality": "text",
+          "date": "2024-1-31",
+          "status": "published",
+          "conciseness": 36.47,
+          "correctness": 25.28,
+          "completeness": 20.82,
+          "f1_like": 22.84,
+          "info": 8.33,
+          "overall": 8.33
+        },
+        {
+          "rank": 25,
+          "name": "Monkey",
+          "url": "https://github.com/Yuliang-Liu/Monkey",
+          "org": "Huazhong University of Science and Technology",
+          "modality": "visual",
+          "date": "2024-8-26",
+          "status": "published",
+          "conciseness": 54.61,
+          "correctness": 17.08,
+          "completeness": 11.27,
+          "f1_like": 13.58,
+          "info": 7.41,
+          "overall": 7.41
+        },
+        {
+          "rank": 26,
+          "name": "Docopilot",
+          "url": "https://github.com/OpenGVLab/Docopilot",
+          "org": "Shanghai AI Laboratory",
+          "modality": "visual",
+          "date": "2025-7-19",
+          "status": "published",
+          "conciseness": 39.31,
+          "correctness": 18.31,
+          "completeness": 17.12,
+          "f1_like": 17.69,
+          "info": 6.96,
+          "overall": 6.96
+        },
+        {
+          "rank": 27,
+          "name": "Qwen3",
+          "url": "https://github.com/QwenLM/Qwen3",
+          "org": "Alibaba",
+          "modality": "visual",
+          "date": "2025-7-21",
+          "status": "published",
+          "conciseness": 22.64,
+          "correctness": 20.17,
+          "completeness": 20.14,
+          "f1_like": 20.16,
+          "info": 4.56,
+          "overall": 4.56
+        },
+        {
+          "rank": 28,
+          "name": "DocOwl2",
+          "url": "https://github.com/X-PLUG/mPLUG-DocOwl",
+          "org": "Alibaba",
+          "modality": "visual",
+          "date": "2024-9-9",
+          "status": "published",
+          "conciseness": 50.19,
+          "correctness": 11.75,
+          "completeness": 6.66,
+          "f1_like": 8.5,
+          "info": 4.27,
+          "overall": 4.27
+        }
+      ]
+    }
+  }
+}

data/leaderboard_seed.csv

@@ -0,0 +1,29 @@
+Model,Organization,Input Config,Date,Conciseness,Correctness,Completeness,F1-like,Informativeness
+[GPT-5](https://openai.com/index/introducing-gpt-5/),OpenAI,TEXT,2025-8-7,54.93,69.10,67.33,68.20,37.46
+[GPT-5.2](https://openai.com/index/introducing-gpt-5-2/),OpenAI,TEXT,2025-12-11,53.81,66.84,64.03,65.40,35.19
+[GPT-5](https://openai.com/index/introducing-gpt-5/),OpenAI,VISUAL,2025-8-7,61.47,58.90,55.34,57.07,35.08
+[Gemini-2.5-Pro](https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/),Google,TEXT,2025-3-25,54.87,62.65,59.03,60.79,33.35
+[Gemini-3-Pro](https://blog.google/products-and-platforms/products/gemini/gemini-3/),Google,TEXT,2025-11-18,52.81,62.69,60.28,61.46,32.46
+[DeepSeek-V3.2](https://api-docs.deepseek.com/news/news251201),DeepSeek-AI,TEXT,2025-12-1,56.31,58.73,55.19,56.91,32.04
+[GPT-5.2](https://openai.com/index/introducing-gpt-5-2/),OpenAI,VISUAL,2025-12-11,56.43,56.75,52.82,54.72,30.88
+[DeepSeek-V3.1](https://api-docs.deepseek.com/news/news250821),DeepSeek-AI,TEXT,2025-8-21,54.76,57.85,54.85,56.31,30.84
+[GLM-4.6V](https://github.com/zai-org/GLM-V),Z.ai,VISUAL,2025-12-8,64.55,47.32,43.43,45.29,29.23
+[GLM-4.7](https://z.ai/blog/glm-4.7),Z.ai,TEXT,2025-12-22,54.34,54.36,51.75,53.02,28.81
+[GLM-4.5V](https://github.com/zai-org/GLM-V),Z.ai,VISUAL,2025-8-11,59.44,48.79,43.62,46.06,27.38
+[gemini-3-pro](https://blog.google/products-and-platforms/products/gemini/gemini-3/),Google,VISUAL,2025-11-18,50.22,56.06,52.69,54.32,27.28
+[GLM-4.5](https://z.ai/blog/glm-4.5),Z.ai,TEXT,2025-7-28,43.41,58.95,59.54,59.24,25.72
+[gemini-2.5-pro](https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/),Google,VISUAL,2025-3-25,51.71,48.39,45.59,46.95,24.28
+[Claude-Sonnet-4](https://www.anthropic.com/news/claude-4),Anthropic,TEXT,2025-5-23,41.37,58.53,58.44,58.48,24.19
+[Qwen3](https://github.com/QwenLM/Qwen3),Alibaba,TEXT,2025-7-21,41.44,55.88,56.64,56.26,23.31
+[Claude-Sonnet-4.5](https://www.anthropic.com/news/claude-sonnet-4-5),Anthropic,TEXT,2025-9-30,31.02,64.31,64.97,64.64,20.05
+[Claude-Sonnet-4.5](https://www.anthropic.com/news/claude-sonnet-4-5),Anthropic,VISUAL,2025-9-30,31.95,55.35,54.45,54.89,17.54
+[Claude-Sonnet-4](https://www.anthropic.com/news/claude-4),Anthropic,VISUAL,2025-5-23,31.63,54.16,53.32,53.74,16.99
+[HippoRAG2](https://github.com/ianliuwd/HippoRAG2),The Ohio State University,TEXT,2025-6-19,45.77,33.13,27.88,30.28,13.86
+[MemoRAG](https://github.com/qhjqhj00/MemoRAG),Peking University & Hong Kong Polytechnic University,TEXT,2025-4-9,51.31,24.19,19.10,21.35,10.96
+[VdocRAG](https://vdocrag.github.io/),NTT Corporation & Tohoku University,VISUAL,2025-4-14,61.54,21.17,13.88,16.77,10.32
+[VisRAG](https://github.com/OpenBMB/VisRAG),Tsinghua University & ModelBest Inc.,VISUAL,2025-3-2,39.90,26.24,23.63,24.87,9.92
+[Raptor](https://github.com/parthsarthi03/raptor),Stanford University,TEXT,2024-1-31,36.47,25.28,20.82,22.84,8.33
+[Monkey](https://github.com/Yuliang-Liu/Monkey),Huazhong University of Science and Technology,VISUAL,2024-8-26,54.61,17.08,11.27,13.58,7.41
+[Docopilot](https://github.com/OpenGVLab/Docopilot),Shanghai AI Laboratory,VISUAL,2025-7-19,39.31,18.31,17.12,17.69,6.96
+[Qwen3](https://github.com/QwenLM/Qwen3),Alibaba,VISUAL,2025-7-21,22.64,20.17,20.14,20.16,4.56
+[DocOwl2](https://github.com/X-PLUG/mPLUG-DocOwl),Alibaba,VISUAL,2024-9-9,50.19,11.75,6.66,8.50,4.27

requirements.txt

@@ -0,0 +1,5 @@
+gradio>=5.0.0,<6.0.0
+pandas>=2.0.0
+huggingface_hub>=0.24.0,<1.0.0
+pyyaml>=6.0.0
+pyaudioop>=0.2.0; python_version >= "3.13"