| --- |
| tags: |
| - ctranslate2 |
| - int8 |
| - float16 |
| - mteb |
| - Sentence Transformers |
| - sentence-similarity |
| - sentence-transformers |
| model-index: |
| - name: e5-large |
| results: |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/amazon_counterfactual |
| name: MTEB AmazonCounterfactualClassification (en) |
| config: en |
| split: test |
| revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
| metrics: |
| - type: accuracy |
| value: 77.68656716417911 |
| - type: ap |
| value: 41.336896075573584 |
| - type: f1 |
| value: 71.788561468075 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/amazon_polarity |
| name: MTEB AmazonPolarityClassification |
| config: default |
| split: test |
| revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
| metrics: |
| - type: accuracy |
| value: 90.04965 |
| - type: ap |
| value: 86.24637009569418 |
| - type: f1 |
| value: 90.03896671762645 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/amazon_reviews_multi |
| name: MTEB AmazonReviewsClassification (en) |
| config: en |
| split: test |
| revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
| metrics: |
| - type: accuracy |
| value: 43.016000000000005 |
| - type: f1 |
| value: 42.1942431880186 |
| - task: |
| type: Retrieval |
| dataset: |
| type: arguana |
| name: MTEB ArguAna |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 25.107000000000003 |
| - type: map_at_10 |
| value: 40.464 |
| - type: map_at_100 |
| value: 41.577999999999996 |
| - type: map_at_1000 |
| value: 41.588 |
| - type: map_at_3 |
| value: 35.301 |
| - type: map_at_5 |
| value: 38.263000000000005 |
| - type: mrr_at_1 |
| value: 25.605 |
| - type: mrr_at_10 |
| value: 40.64 |
| - type: mrr_at_100 |
| value: 41.760000000000005 |
| - type: mrr_at_1000 |
| value: 41.77 |
| - type: mrr_at_3 |
| value: 35.443000000000005 |
| - type: mrr_at_5 |
| value: 38.448 |
| - type: ndcg_at_1 |
| value: 25.107000000000003 |
| - type: ndcg_at_10 |
| value: 49.352000000000004 |
| - type: ndcg_at_100 |
| value: 53.98500000000001 |
| - type: ndcg_at_1000 |
| value: 54.208 |
| - type: ndcg_at_3 |
| value: 38.671 |
| - type: ndcg_at_5 |
| value: 43.991 |
| - type: precision_at_1 |
| value: 25.107000000000003 |
| - type: precision_at_10 |
| value: 7.795000000000001 |
| - type: precision_at_100 |
| value: 0.979 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_3 |
| value: 16.145 |
| - type: precision_at_5 |
| value: 12.262 |
| - type: recall_at_1 |
| value: 25.107000000000003 |
| - type: recall_at_10 |
| value: 77.952 |
| - type: recall_at_100 |
| value: 97.866 |
| - type: recall_at_1000 |
| value: 99.57300000000001 |
| - type: recall_at_3 |
| value: 48.435 |
| - type: recall_at_5 |
| value: 61.309000000000005 |
| - task: |
| type: Clustering |
| dataset: |
| type: mteb/arxiv-clustering-p2p |
| name: MTEB ArxivClusteringP2P |
| config: default |
| split: test |
| revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
| metrics: |
| - type: v_measure |
| value: 46.19278045044154 |
| - task: |
| type: Clustering |
| dataset: |
| type: mteb/arxiv-clustering-s2s |
| name: MTEB ArxivClusteringS2S |
| config: default |
| split: test |
| revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 |
| metrics: |
| - type: v_measure |
| value: 41.37976387757665 |
| - task: |
| type: Reranking |
| dataset: |
| type: mteb/askubuntudupquestions-reranking |
| name: MTEB AskUbuntuDupQuestions |
| config: default |
| split: test |
| revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
| metrics: |
| - type: map |
| value: 60.07433334608074 |
| - type: mrr |
| value: 73.44347711383723 |
| - task: |
| type: STS |
| dataset: |
| type: mteb/biosses-sts |
| name: MTEB BIOSSES |
| config: default |
| split: test |
| revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
| metrics: |
| - type: cos_sim_pearson |
| value: 86.4298072183543 |
| - type: cos_sim_spearman |
| value: 84.73144873582848 |
| - type: euclidean_pearson |
| value: 85.15885058870728 |
| - type: euclidean_spearman |
| value: 85.42062106559356 |
| - type: manhattan_pearson |
| value: 84.89409921792054 |
| - type: manhattan_spearman |
| value: 85.31941394024344 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/banking77 |
| name: MTEB Banking77Classification |
| config: default |
| split: test |
| revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
| metrics: |
| - type: accuracy |
| value: 84.14285714285714 |
| - type: f1 |
| value: 84.11674412565644 |
| - task: |
| type: Clustering |
| dataset: |
| type: mteb/biorxiv-clustering-p2p |
| name: MTEB BiorxivClusteringP2P |
| config: default |
| split: test |
| revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 |
| metrics: |
| - type: v_measure |
| value: 37.600076342340785 |
| - task: |
| type: Clustering |
| dataset: |
| type: mteb/biorxiv-clustering-s2s |
| name: MTEB BiorxivClusteringS2S |
| config: default |
| split: test |
| revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 |
| metrics: |
| - type: v_measure |
| value: 35.08861812135148 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackAndroidRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 32.684000000000005 |
| - type: map_at_10 |
| value: 41.675000000000004 |
| - type: map_at_100 |
| value: 42.963 |
| - type: map_at_1000 |
| value: 43.078 |
| - type: map_at_3 |
| value: 38.708999999999996 |
| - type: map_at_5 |
| value: 40.316 |
| - type: mrr_at_1 |
| value: 39.485 |
| - type: mrr_at_10 |
| value: 47.152 |
| - type: mrr_at_100 |
| value: 47.96 |
| - type: mrr_at_1000 |
| value: 48.010000000000005 |
| - type: mrr_at_3 |
| value: 44.754 |
| - type: mrr_at_5 |
| value: 46.285 |
| - type: ndcg_at_1 |
| value: 39.485 |
| - type: ndcg_at_10 |
| value: 46.849000000000004 |
| - type: ndcg_at_100 |
| value: 52.059 |
| - type: ndcg_at_1000 |
| value: 54.358 |
| - type: ndcg_at_3 |
| value: 42.705 |
| - type: ndcg_at_5 |
| value: 44.663000000000004 |
| - type: precision_at_1 |
| value: 39.485 |
| - type: precision_at_10 |
| value: 8.455 |
| - type: precision_at_100 |
| value: 1.3379999999999999 |
| - type: precision_at_1000 |
| value: 0.178 |
| - type: precision_at_3 |
| value: 19.695 |
| - type: precision_at_5 |
| value: 13.905999999999999 |
| - type: recall_at_1 |
| value: 32.684000000000005 |
| - type: recall_at_10 |
| value: 56.227000000000004 |
| - type: recall_at_100 |
| value: 78.499 |
| - type: recall_at_1000 |
| value: 94.021 |
| - type: recall_at_3 |
| value: 44.157999999999994 |
| - type: recall_at_5 |
| value: 49.694 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackEnglishRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 31.875999999999998 |
| - type: map_at_10 |
| value: 41.603 |
| - type: map_at_100 |
| value: 42.825 |
| - type: map_at_1000 |
| value: 42.961 |
| - type: map_at_3 |
| value: 38.655 |
| - type: map_at_5 |
| value: 40.294999999999995 |
| - type: mrr_at_1 |
| value: 40.127 |
| - type: mrr_at_10 |
| value: 47.959 |
| - type: mrr_at_100 |
| value: 48.59 |
| - type: mrr_at_1000 |
| value: 48.634 |
| - type: mrr_at_3 |
| value: 45.786 |
| - type: mrr_at_5 |
| value: 46.964 |
| - type: ndcg_at_1 |
| value: 40.127 |
| - type: ndcg_at_10 |
| value: 47.176 |
| - type: ndcg_at_100 |
| value: 51.346000000000004 |
| - type: ndcg_at_1000 |
| value: 53.502 |
| - type: ndcg_at_3 |
| value: 43.139 |
| - type: ndcg_at_5 |
| value: 44.883 |
| - type: precision_at_1 |
| value: 40.127 |
| - type: precision_at_10 |
| value: 8.72 |
| - type: precision_at_100 |
| value: 1.387 |
| - type: precision_at_1000 |
| value: 0.188 |
| - type: precision_at_3 |
| value: 20.637 |
| - type: precision_at_5 |
| value: 14.446 |
| - type: recall_at_1 |
| value: 31.875999999999998 |
| - type: recall_at_10 |
| value: 56.54900000000001 |
| - type: recall_at_100 |
| value: 73.939 |
| - type: recall_at_1000 |
| value: 87.732 |
| - type: recall_at_3 |
| value: 44.326 |
| - type: recall_at_5 |
| value: 49.445 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackGamingRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 41.677 |
| - type: map_at_10 |
| value: 52.222 |
| - type: map_at_100 |
| value: 53.229000000000006 |
| - type: map_at_1000 |
| value: 53.288000000000004 |
| - type: map_at_3 |
| value: 49.201 |
| - type: map_at_5 |
| value: 51.00599999999999 |
| - type: mrr_at_1 |
| value: 47.524 |
| - type: mrr_at_10 |
| value: 55.745999999999995 |
| - type: mrr_at_100 |
| value: 56.433 |
| - type: mrr_at_1000 |
| value: 56.464999999999996 |
| - type: mrr_at_3 |
| value: 53.37499999999999 |
| - type: mrr_at_5 |
| value: 54.858 |
| - type: ndcg_at_1 |
| value: 47.524 |
| - type: ndcg_at_10 |
| value: 57.406 |
| - type: ndcg_at_100 |
| value: 61.403 |
| - type: ndcg_at_1000 |
| value: 62.7 |
| - type: ndcg_at_3 |
| value: 52.298 |
| - type: ndcg_at_5 |
| value: 55.02 |
| - type: precision_at_1 |
| value: 47.524 |
| - type: precision_at_10 |
| value: 8.865 |
| - type: precision_at_100 |
| value: 1.179 |
| - type: precision_at_1000 |
| value: 0.134 |
| - type: precision_at_3 |
| value: 22.612 |
| - type: precision_at_5 |
| value: 15.461 |
| - type: recall_at_1 |
| value: 41.677 |
| - type: recall_at_10 |
| value: 69.346 |
| - type: recall_at_100 |
| value: 86.344 |
| - type: recall_at_1000 |
| value: 95.703 |
| - type: recall_at_3 |
| value: 55.789 |
| - type: recall_at_5 |
| value: 62.488 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackGisRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 25.991999999999997 |
| - type: map_at_10 |
| value: 32.804 |
| - type: map_at_100 |
| value: 33.812999999999995 |
| - type: map_at_1000 |
| value: 33.897 |
| - type: map_at_3 |
| value: 30.567 |
| - type: map_at_5 |
| value: 31.599 |
| - type: mrr_at_1 |
| value: 27.797 |
| - type: mrr_at_10 |
| value: 34.768 |
| - type: mrr_at_100 |
| value: 35.702 |
| - type: mrr_at_1000 |
| value: 35.766 |
| - type: mrr_at_3 |
| value: 32.637 |
| - type: mrr_at_5 |
| value: 33.614 |
| - type: ndcg_at_1 |
| value: 27.797 |
| - type: ndcg_at_10 |
| value: 36.966 |
| - type: ndcg_at_100 |
| value: 41.972 |
| - type: ndcg_at_1000 |
| value: 44.139 |
| - type: ndcg_at_3 |
| value: 32.547 |
| - type: ndcg_at_5 |
| value: 34.258 |
| - type: precision_at_1 |
| value: 27.797 |
| - type: precision_at_10 |
| value: 5.514 |
| - type: precision_at_100 |
| value: 0.8340000000000001 |
| - type: precision_at_1000 |
| value: 0.106 |
| - type: precision_at_3 |
| value: 13.333 |
| - type: precision_at_5 |
| value: 9.04 |
| - type: recall_at_1 |
| value: 25.991999999999997 |
| - type: recall_at_10 |
| value: 47.941 |
| - type: recall_at_100 |
| value: 71.039 |
| - type: recall_at_1000 |
| value: 87.32799999999999 |
| - type: recall_at_3 |
| value: 36.01 |
| - type: recall_at_5 |
| value: 40.056000000000004 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackMathematicaRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 17.533 |
| - type: map_at_10 |
| value: 24.336 |
| - type: map_at_100 |
| value: 25.445 |
| - type: map_at_1000 |
| value: 25.561 |
| - type: map_at_3 |
| value: 22.116 |
| - type: map_at_5 |
| value: 23.347 |
| - type: mrr_at_1 |
| value: 21.642 |
| - type: mrr_at_10 |
| value: 28.910999999999998 |
| - type: mrr_at_100 |
| value: 29.836000000000002 |
| - type: mrr_at_1000 |
| value: 29.907 |
| - type: mrr_at_3 |
| value: 26.638 |
| - type: mrr_at_5 |
| value: 27.857 |
| - type: ndcg_at_1 |
| value: 21.642 |
| - type: ndcg_at_10 |
| value: 28.949 |
| - type: ndcg_at_100 |
| value: 34.211000000000006 |
| - type: ndcg_at_1000 |
| value: 37.031 |
| - type: ndcg_at_3 |
| value: 24.788 |
| - type: ndcg_at_5 |
| value: 26.685 |
| - type: precision_at_1 |
| value: 21.642 |
| - type: precision_at_10 |
| value: 5.137 |
| - type: precision_at_100 |
| value: 0.893 |
| - type: precision_at_1000 |
| value: 0.127 |
| - type: precision_at_3 |
| value: 11.733 |
| - type: precision_at_5 |
| value: 8.383000000000001 |
| - type: recall_at_1 |
| value: 17.533 |
| - type: recall_at_10 |
| value: 38.839 |
| - type: recall_at_100 |
| value: 61.458999999999996 |
| - type: recall_at_1000 |
| value: 81.58 |
| - type: recall_at_3 |
| value: 27.328999999999997 |
| - type: recall_at_5 |
| value: 32.168 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackPhysicsRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 28.126 |
| - type: map_at_10 |
| value: 37.872 |
| - type: map_at_100 |
| value: 39.229 |
| - type: map_at_1000 |
| value: 39.353 |
| - type: map_at_3 |
| value: 34.93 |
| - type: map_at_5 |
| value: 36.59 |
| - type: mrr_at_1 |
| value: 34.071 |
| - type: mrr_at_10 |
| value: 43.056 |
| - type: mrr_at_100 |
| value: 43.944 |
| - type: mrr_at_1000 |
| value: 43.999 |
| - type: mrr_at_3 |
| value: 40.536 |
| - type: mrr_at_5 |
| value: 42.065999999999995 |
| - type: ndcg_at_1 |
| value: 34.071 |
| - type: ndcg_at_10 |
| value: 43.503 |
| - type: ndcg_at_100 |
| value: 49.120000000000005 |
| - type: ndcg_at_1000 |
| value: 51.410999999999994 |
| - type: ndcg_at_3 |
| value: 38.767 |
| - type: ndcg_at_5 |
| value: 41.075 |
| - type: precision_at_1 |
| value: 34.071 |
| - type: precision_at_10 |
| value: 7.843999999999999 |
| - type: precision_at_100 |
| value: 1.2489999999999999 |
| - type: precision_at_1000 |
| value: 0.163 |
| - type: precision_at_3 |
| value: 18.223 |
| - type: precision_at_5 |
| value: 13.050999999999998 |
| - type: recall_at_1 |
| value: 28.126 |
| - type: recall_at_10 |
| value: 54.952 |
| - type: recall_at_100 |
| value: 78.375 |
| - type: recall_at_1000 |
| value: 93.29899999999999 |
| - type: recall_at_3 |
| value: 41.714 |
| - type: recall_at_5 |
| value: 47.635 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackProgrammersRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 25.957 |
| - type: map_at_10 |
| value: 34.749 |
| - type: map_at_100 |
| value: 35.929 |
| - type: map_at_1000 |
| value: 36.043 |
| - type: map_at_3 |
| value: 31.947 |
| - type: map_at_5 |
| value: 33.575 |
| - type: mrr_at_1 |
| value: 32.078 |
| - type: mrr_at_10 |
| value: 39.844 |
| - type: mrr_at_100 |
| value: 40.71 |
| - type: mrr_at_1000 |
| value: 40.77 |
| - type: mrr_at_3 |
| value: 37.386 |
| - type: mrr_at_5 |
| value: 38.83 |
| - type: ndcg_at_1 |
| value: 32.078 |
| - type: ndcg_at_10 |
| value: 39.97 |
| - type: ndcg_at_100 |
| value: 45.254 |
| - type: ndcg_at_1000 |
| value: 47.818 |
| - type: ndcg_at_3 |
| value: 35.453 |
| - type: ndcg_at_5 |
| value: 37.631 |
| - type: precision_at_1 |
| value: 32.078 |
| - type: precision_at_10 |
| value: 7.158 |
| - type: precision_at_100 |
| value: 1.126 |
| - type: precision_at_1000 |
| value: 0.153 |
| - type: precision_at_3 |
| value: 16.743 |
| - type: precision_at_5 |
| value: 11.872 |
| - type: recall_at_1 |
| value: 25.957 |
| - type: recall_at_10 |
| value: 50.583 |
| - type: recall_at_100 |
| value: 73.593 |
| - type: recall_at_1000 |
| value: 91.23599999999999 |
| - type: recall_at_3 |
| value: 37.651 |
| - type: recall_at_5 |
| value: 43.626 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 27.1505 |
| - type: map_at_10 |
| value: 34.844833333333334 |
| - type: map_at_100 |
| value: 35.95216666666667 |
| - type: map_at_1000 |
| value: 36.06675 |
| - type: map_at_3 |
| value: 32.41975 |
| - type: map_at_5 |
| value: 33.74233333333333 |
| - type: mrr_at_1 |
| value: 31.923666666666662 |
| - type: mrr_at_10 |
| value: 38.87983333333334 |
| - type: mrr_at_100 |
| value: 39.706250000000004 |
| - type: mrr_at_1000 |
| value: 39.76708333333333 |
| - type: mrr_at_3 |
| value: 36.72008333333333 |
| - type: mrr_at_5 |
| value: 37.96933333333334 |
| - type: ndcg_at_1 |
| value: 31.923666666666662 |
| - type: ndcg_at_10 |
| value: 39.44258333333334 |
| - type: ndcg_at_100 |
| value: 44.31475 |
| - type: ndcg_at_1000 |
| value: 46.75 |
| - type: ndcg_at_3 |
| value: 35.36299999999999 |
| - type: ndcg_at_5 |
| value: 37.242333333333335 |
| - type: precision_at_1 |
| value: 31.923666666666662 |
| - type: precision_at_10 |
| value: 6.643333333333333 |
| - type: precision_at_100 |
| value: 1.0612499999999998 |
| - type: precision_at_1000 |
| value: 0.14575 |
| - type: precision_at_3 |
| value: 15.875250000000001 |
| - type: precision_at_5 |
| value: 11.088916666666664 |
| - type: recall_at_1 |
| value: 27.1505 |
| - type: recall_at_10 |
| value: 49.06349999999999 |
| - type: recall_at_100 |
| value: 70.60841666666666 |
| - type: recall_at_1000 |
| value: 87.72049999999999 |
| - type: recall_at_3 |
| value: 37.60575000000001 |
| - type: recall_at_5 |
| value: 42.511166666666675 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackStatsRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 25.101000000000003 |
| - type: map_at_10 |
| value: 30.147000000000002 |
| - type: map_at_100 |
| value: 30.98 |
| - type: map_at_1000 |
| value: 31.080000000000002 |
| - type: map_at_3 |
| value: 28.571 |
| - type: map_at_5 |
| value: 29.319 |
| - type: mrr_at_1 |
| value: 27.761000000000003 |
| - type: mrr_at_10 |
| value: 32.716 |
| - type: mrr_at_100 |
| value: 33.504 |
| - type: mrr_at_1000 |
| value: 33.574 |
| - type: mrr_at_3 |
| value: 31.135 |
| - type: mrr_at_5 |
| value: 32.032 |
| - type: ndcg_at_1 |
| value: 27.761000000000003 |
| - type: ndcg_at_10 |
| value: 33.358 |
| - type: ndcg_at_100 |
| value: 37.569 |
| - type: ndcg_at_1000 |
| value: 40.189 |
| - type: ndcg_at_3 |
| value: 30.291 |
| - type: ndcg_at_5 |
| value: 31.558000000000003 |
| - type: precision_at_1 |
| value: 27.761000000000003 |
| - type: precision_at_10 |
| value: 4.939 |
| - type: precision_at_100 |
| value: 0.759 |
| - type: precision_at_1000 |
| value: 0.106 |
| - type: precision_at_3 |
| value: 12.577 |
| - type: precision_at_5 |
| value: 8.497 |
| - type: recall_at_1 |
| value: 25.101000000000003 |
| - type: recall_at_10 |
| value: 40.739 |
| - type: recall_at_100 |
| value: 60.089999999999996 |
| - type: recall_at_1000 |
| value: 79.768 |
| - type: recall_at_3 |
| value: 32.16 |
| - type: recall_at_5 |
| value: 35.131 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackTexRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 20.112 |
| - type: map_at_10 |
| value: 26.119999999999997 |
| - type: map_at_100 |
| value: 27.031 |
| - type: map_at_1000 |
| value: 27.150000000000002 |
| - type: map_at_3 |
| value: 24.230999999999998 |
| - type: map_at_5 |
| value: 25.15 |
| - type: mrr_at_1 |
| value: 24.535 |
| - type: mrr_at_10 |
| value: 30.198000000000004 |
| - type: mrr_at_100 |
| value: 30.975 |
| - type: mrr_at_1000 |
| value: 31.051000000000002 |
| - type: mrr_at_3 |
| value: 28.338 |
| - type: mrr_at_5 |
| value: 29.269000000000002 |
| - type: ndcg_at_1 |
| value: 24.535 |
| - type: ndcg_at_10 |
| value: 30.147000000000002 |
| - type: ndcg_at_100 |
| value: 34.544000000000004 |
| - type: ndcg_at_1000 |
| value: 37.512 |
| - type: ndcg_at_3 |
| value: 26.726 |
| - type: ndcg_at_5 |
| value: 28.046 |
| - type: precision_at_1 |
| value: 24.535 |
| - type: precision_at_10 |
| value: 5.179 |
| - type: precision_at_100 |
| value: 0.859 |
| - type: precision_at_1000 |
| value: 0.128 |
| - type: precision_at_3 |
| value: 12.159 |
| - type: precision_at_5 |
| value: 8.424 |
| - type: recall_at_1 |
| value: 20.112 |
| - type: recall_at_10 |
| value: 38.312000000000005 |
| - type: recall_at_100 |
| value: 58.406000000000006 |
| - type: recall_at_1000 |
| value: 79.863 |
| - type: recall_at_3 |
| value: 28.358 |
| - type: recall_at_5 |
| value: 31.973000000000003 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackUnixRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 27.111 |
| - type: map_at_10 |
| value: 34.096 |
| - type: map_at_100 |
| value: 35.181000000000004 |
| - type: map_at_1000 |
| value: 35.276 |
| - type: map_at_3 |
| value: 31.745 |
| - type: map_at_5 |
| value: 33.045 |
| - type: mrr_at_1 |
| value: 31.343 |
| - type: mrr_at_10 |
| value: 37.994 |
| - type: mrr_at_100 |
| value: 38.873000000000005 |
| - type: mrr_at_1000 |
| value: 38.934999999999995 |
| - type: mrr_at_3 |
| value: 35.743 |
| - type: mrr_at_5 |
| value: 37.077 |
| - type: ndcg_at_1 |
| value: 31.343 |
| - type: ndcg_at_10 |
| value: 38.572 |
| - type: ndcg_at_100 |
| value: 43.854 |
| - type: ndcg_at_1000 |
| value: 46.190999999999995 |
| - type: ndcg_at_3 |
| value: 34.247 |
| - type: ndcg_at_5 |
| value: 36.28 |
| - type: precision_at_1 |
| value: 31.343 |
| - type: precision_at_10 |
| value: 6.166 |
| - type: precision_at_100 |
| value: 1 |
| - type: precision_at_1000 |
| value: 0.13 |
| - type: precision_at_3 |
| value: 15.081 |
| - type: precision_at_5 |
| value: 10.428999999999998 |
| - type: recall_at_1 |
| value: 27.111 |
| - type: recall_at_10 |
| value: 48.422 |
| - type: recall_at_100 |
| value: 71.846 |
| - type: recall_at_1000 |
| value: 88.57000000000001 |
| - type: recall_at_3 |
| value: 36.435 |
| - type: recall_at_5 |
| value: 41.765 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackWebmastersRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 26.264 |
| - type: map_at_10 |
| value: 33.522 |
| - type: map_at_100 |
| value: 34.963 |
| - type: map_at_1000 |
| value: 35.175 |
| - type: map_at_3 |
| value: 31.366 |
| - type: map_at_5 |
| value: 32.621 |
| - type: mrr_at_1 |
| value: 31.028 |
| - type: mrr_at_10 |
| value: 37.230000000000004 |
| - type: mrr_at_100 |
| value: 38.149 |
| - type: mrr_at_1000 |
| value: 38.218 |
| - type: mrr_at_3 |
| value: 35.046 |
| - type: mrr_at_5 |
| value: 36.617 |
| - type: ndcg_at_1 |
| value: 31.028 |
| - type: ndcg_at_10 |
| value: 37.964999999999996 |
| - type: ndcg_at_100 |
| value: 43.342000000000006 |
| - type: ndcg_at_1000 |
| value: 46.471000000000004 |
| - type: ndcg_at_3 |
| value: 34.67 |
| - type: ndcg_at_5 |
| value: 36.458 |
| - type: precision_at_1 |
| value: 31.028 |
| - type: precision_at_10 |
| value: 6.937 |
| - type: precision_at_100 |
| value: 1.346 |
| - type: precision_at_1000 |
| value: 0.22799999999999998 |
| - type: precision_at_3 |
| value: 15.942 |
| - type: precision_at_5 |
| value: 11.462 |
| - type: recall_at_1 |
| value: 26.264 |
| - type: recall_at_10 |
| value: 45.571 |
| - type: recall_at_100 |
| value: 70.246 |
| - type: recall_at_1000 |
| value: 90.971 |
| - type: recall_at_3 |
| value: 36.276 |
| - type: recall_at_5 |
| value: 41.162 |
| - task: |
| type: Retrieval |
| dataset: |
| type: BeIR/cqadupstack |
| name: MTEB CQADupstackWordpressRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 23.372999999999998 |
| - type: map_at_10 |
| value: 28.992 |
| - type: map_at_100 |
| value: 29.837999999999997 |
| - type: map_at_1000 |
| value: 29.939 |
| - type: map_at_3 |
| value: 26.999000000000002 |
| - type: map_at_5 |
| value: 28.044999999999998 |
| - type: mrr_at_1 |
| value: 25.692999999999998 |
| - type: mrr_at_10 |
| value: 30.984 |
| - type: mrr_at_100 |
| value: 31.799 |
| - type: mrr_at_1000 |
| value: 31.875999999999998 |
| - type: mrr_at_3 |
| value: 29.267 |
| - type: mrr_at_5 |
| value: 30.163 |
| - type: ndcg_at_1 |
| value: 25.692999999999998 |
| - type: ndcg_at_10 |
| value: 32.45 |
| - type: ndcg_at_100 |
| value: 37.103 |
| - type: ndcg_at_1000 |
| value: 39.678000000000004 |
| - type: ndcg_at_3 |
| value: 28.725 |
| - type: ndcg_at_5 |
| value: 30.351 |
| - type: precision_at_1 |
| value: 25.692999999999998 |
| - type: precision_at_10 |
| value: 4.806 |
| - type: precision_at_100 |
| value: 0.765 |
| - type: precision_at_1000 |
| value: 0.108 |
| - type: precision_at_3 |
| value: 11.768 |
| - type: precision_at_5 |
| value: 8.096 |
| - type: recall_at_1 |
| value: 23.372999999999998 |
| - type: recall_at_10 |
| value: 41.281 |
| - type: recall_at_100 |
| value: 63.465 |
| - type: recall_at_1000 |
| value: 82.575 |
| - type: recall_at_3 |
| value: 31.063000000000002 |
| - type: recall_at_5 |
| value: 34.991 |
| - task: |
| type: Retrieval |
| dataset: |
| type: climate-fever |
| name: MTEB ClimateFEVER |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 8.821 |
| - type: map_at_10 |
| value: 15.383 |
| - type: map_at_100 |
| value: 17.244999999999997 |
| - type: map_at_1000 |
| value: 17.445 |
| - type: map_at_3 |
| value: 12.64 |
| - type: map_at_5 |
| value: 13.941999999999998 |
| - type: mrr_at_1 |
| value: 19.544 |
| - type: mrr_at_10 |
| value: 29.738999999999997 |
| - type: mrr_at_100 |
| value: 30.923000000000002 |
| - type: mrr_at_1000 |
| value: 30.969 |
| - type: mrr_at_3 |
| value: 26.384 |
| - type: mrr_at_5 |
| value: 28.199 |
| - type: ndcg_at_1 |
| value: 19.544 |
| - type: ndcg_at_10 |
| value: 22.398 |
| - type: ndcg_at_100 |
| value: 30.253999999999998 |
| - type: ndcg_at_1000 |
| value: 33.876 |
| - type: ndcg_at_3 |
| value: 17.473 |
| - type: ndcg_at_5 |
| value: 19.154 |
| - type: precision_at_1 |
| value: 19.544 |
| - type: precision_at_10 |
| value: 7.217999999999999 |
| - type: precision_at_100 |
| value: 1.564 |
| - type: precision_at_1000 |
| value: 0.22300000000000003 |
| - type: precision_at_3 |
| value: 13.225000000000001 |
| - type: precision_at_5 |
| value: 10.319 |
| - type: recall_at_1 |
| value: 8.821 |
| - type: recall_at_10 |
| value: 28.110000000000003 |
| - type: recall_at_100 |
| value: 55.64 |
| - type: recall_at_1000 |
| value: 75.964 |
| - type: recall_at_3 |
| value: 16.195 |
| - type: recall_at_5 |
| value: 20.678 |
| - task: |
| type: Retrieval |
| dataset: |
| type: dbpedia-entity |
| name: MTEB DBPedia |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 9.344 |
| - type: map_at_10 |
| value: 20.301 |
| - type: map_at_100 |
| value: 28.709 |
| - type: map_at_1000 |
| value: 30.470999999999997 |
| - type: map_at_3 |
| value: 14.584 |
| - type: map_at_5 |
| value: 16.930999999999997 |
| - type: mrr_at_1 |
| value: 67.25 |
| - type: mrr_at_10 |
| value: 75.393 |
| - type: mrr_at_100 |
| value: 75.742 |
| - type: mrr_at_1000 |
| value: 75.75 |
| - type: mrr_at_3 |
| value: 73.958 |
| - type: mrr_at_5 |
| value: 74.883 |
| - type: ndcg_at_1 |
| value: 56.00000000000001 |
| - type: ndcg_at_10 |
| value: 42.394 |
| - type: ndcg_at_100 |
| value: 47.091 |
| - type: ndcg_at_1000 |
| value: 54.215 |
| - type: ndcg_at_3 |
| value: 46.995 |
| - type: ndcg_at_5 |
| value: 44.214999999999996 |
| - type: precision_at_1 |
| value: 67.25 |
| - type: precision_at_10 |
| value: 33.525 |
| - type: precision_at_100 |
| value: 10.67 |
| - type: precision_at_1000 |
| value: 2.221 |
| - type: precision_at_3 |
| value: 49.417 |
| - type: precision_at_5 |
| value: 42.15 |
| - type: recall_at_1 |
| value: 9.344 |
| - type: recall_at_10 |
| value: 25.209 |
| - type: recall_at_100 |
| value: 52.329 |
| - type: recall_at_1000 |
| value: 74.2 |
| - type: recall_at_3 |
| value: 15.699 |
| - type: recall_at_5 |
| value: 19.24 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/emotion |
| name: MTEB EmotionClassification |
| config: default |
| split: test |
| revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
| metrics: |
| - type: accuracy |
| value: 48.05 |
| - type: f1 |
| value: 43.06718139212933 |
| - task: |
| type: Retrieval |
| dataset: |
| type: fever |
| name: MTEB FEVER |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 46.452 |
| - type: map_at_10 |
| value: 58.825 |
| - type: map_at_100 |
| value: 59.372 |
| - type: map_at_1000 |
| value: 59.399 |
| - type: map_at_3 |
| value: 56.264 |
| - type: map_at_5 |
| value: 57.879999999999995 |
| - type: mrr_at_1 |
| value: 49.82 |
| - type: mrr_at_10 |
| value: 62.178999999999995 |
| - type: mrr_at_100 |
| value: 62.641999999999996 |
| - type: mrr_at_1000 |
| value: 62.658 |
| - type: mrr_at_3 |
| value: 59.706 |
| - type: mrr_at_5 |
| value: 61.283 |
| - type: ndcg_at_1 |
| value: 49.82 |
| - type: ndcg_at_10 |
| value: 65.031 |
| - type: ndcg_at_100 |
| value: 67.413 |
| - type: ndcg_at_1000 |
| value: 68.014 |
| - type: ndcg_at_3 |
| value: 60.084 |
| - type: ndcg_at_5 |
| value: 62.858000000000004 |
| - type: precision_at_1 |
| value: 49.82 |
| - type: precision_at_10 |
| value: 8.876000000000001 |
| - type: precision_at_100 |
| value: 1.018 |
| - type: precision_at_1000 |
| value: 0.109 |
| - type: precision_at_3 |
| value: 24.477 |
| - type: precision_at_5 |
| value: 16.208 |
| - type: recall_at_1 |
| value: 46.452 |
| - type: recall_at_10 |
| value: 80.808 |
| - type: recall_at_100 |
| value: 91.215 |
| - type: recall_at_1000 |
| value: 95.52000000000001 |
| - type: recall_at_3 |
| value: 67.62899999999999 |
| - type: recall_at_5 |
| value: 74.32900000000001 |
| - task: |
| type: Retrieval |
| dataset: |
| type: fiqa |
| name: MTEB FiQA2018 |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 18.351 |
| - type: map_at_10 |
| value: 30.796 |
| - type: map_at_100 |
| value: 32.621 |
| - type: map_at_1000 |
| value: 32.799 |
| - type: map_at_3 |
| value: 26.491 |
| - type: map_at_5 |
| value: 28.933999999999997 |
| - type: mrr_at_1 |
| value: 36.265 |
| - type: mrr_at_10 |
| value: 45.556999999999995 |
| - type: mrr_at_100 |
| value: 46.323 |
| - type: mrr_at_1000 |
| value: 46.359 |
| - type: mrr_at_3 |
| value: 42.695 |
| - type: mrr_at_5 |
| value: 44.324000000000005 |
| - type: ndcg_at_1 |
| value: 36.265 |
| - type: ndcg_at_10 |
| value: 38.558 |
| - type: ndcg_at_100 |
| value: 45.18 |
| - type: ndcg_at_1000 |
| value: 48.292 |
| - type: ndcg_at_3 |
| value: 34.204 |
| - type: ndcg_at_5 |
| value: 35.735 |
| - type: precision_at_1 |
| value: 36.265 |
| - type: precision_at_10 |
| value: 10.879999999999999 |
| - type: precision_at_100 |
| value: 1.77 |
| - type: precision_at_1000 |
| value: 0.234 |
| - type: precision_at_3 |
| value: 23.044999999999998 |
| - type: precision_at_5 |
| value: 17.253 |
| - type: recall_at_1 |
| value: 18.351 |
| - type: recall_at_10 |
| value: 46.116 |
| - type: recall_at_100 |
| value: 70.786 |
| - type: recall_at_1000 |
| value: 89.46300000000001 |
| - type: recall_at_3 |
| value: 31.404 |
| - type: recall_at_5 |
| value: 37.678 |
| - task: |
| type: Retrieval |
| dataset: |
| type: hotpotqa |
| name: MTEB HotpotQA |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 36.847 |
| - type: map_at_10 |
| value: 54.269999999999996 |
| - type: map_at_100 |
| value: 55.152 |
| - type: map_at_1000 |
| value: 55.223 |
| - type: map_at_3 |
| value: 51.166 |
| - type: map_at_5 |
| value: 53.055 |
| - type: mrr_at_1 |
| value: 73.693 |
| - type: mrr_at_10 |
| value: 79.975 |
| - type: mrr_at_100 |
| value: 80.202 |
| - type: mrr_at_1000 |
| value: 80.214 |
| - type: mrr_at_3 |
| value: 78.938 |
| - type: mrr_at_5 |
| value: 79.595 |
| - type: ndcg_at_1 |
| value: 73.693 |
| - type: ndcg_at_10 |
| value: 63.334999999999994 |
| - type: ndcg_at_100 |
| value: 66.452 |
| - type: ndcg_at_1000 |
| value: 67.869 |
| - type: ndcg_at_3 |
| value: 58.829 |
| - type: ndcg_at_5 |
| value: 61.266 |
| - type: precision_at_1 |
| value: 73.693 |
| - type: precision_at_10 |
| value: 13.122 |
| - type: precision_at_100 |
| value: 1.5559999999999998 |
| - type: precision_at_1000 |
| value: 0.174 |
| - type: precision_at_3 |
| value: 37.083 |
| - type: precision_at_5 |
| value: 24.169999999999998 |
| - type: recall_at_1 |
| value: 36.847 |
| - type: recall_at_10 |
| value: 65.61099999999999 |
| - type: recall_at_100 |
| value: 77.792 |
| - type: recall_at_1000 |
| value: 87.17099999999999 |
| - type: recall_at_3 |
| value: 55.625 |
| - type: recall_at_5 |
| value: 60.425 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/imdb |
| name: MTEB ImdbClassification |
| config: default |
| split: test |
| revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
| metrics: |
| - type: accuracy |
| value: 82.1096 |
| - type: ap |
| value: 76.67089212843918 |
| - type: f1 |
| value: 82.03535056754939 |
| - task: |
| type: Retrieval |
| dataset: |
| type: msmarco |
| name: MTEB MSMARCO |
| config: default |
| split: dev |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 24.465 |
| - type: map_at_10 |
| value: 37.072 |
| - type: map_at_100 |
| value: 38.188 |
| - type: map_at_1000 |
| value: 38.232 |
| - type: map_at_3 |
| value: 33.134 |
| - type: map_at_5 |
| value: 35.453 |
| - type: mrr_at_1 |
| value: 25.142999999999997 |
| - type: mrr_at_10 |
| value: 37.669999999999995 |
| - type: mrr_at_100 |
| value: 38.725 |
| - type: mrr_at_1000 |
| value: 38.765 |
| - type: mrr_at_3 |
| value: 33.82 |
| - type: mrr_at_5 |
| value: 36.111 |
| - type: ndcg_at_1 |
| value: 25.142999999999997 |
| - type: ndcg_at_10 |
| value: 44.054 |
| - type: ndcg_at_100 |
| value: 49.364000000000004 |
| - type: ndcg_at_1000 |
| value: 50.456 |
| - type: ndcg_at_3 |
| value: 36.095 |
| - type: ndcg_at_5 |
| value: 40.23 |
| - type: precision_at_1 |
| value: 25.142999999999997 |
| - type: precision_at_10 |
| value: 6.845 |
| - type: precision_at_100 |
| value: 0.95 |
| - type: precision_at_1000 |
| value: 0.104 |
| - type: precision_at_3 |
| value: 15.204999999999998 |
| - type: precision_at_5 |
| value: 11.221 |
| - type: recall_at_1 |
| value: 24.465 |
| - type: recall_at_10 |
| value: 65.495 |
| - type: recall_at_100 |
| value: 89.888 |
| - type: recall_at_1000 |
| value: 98.165 |
| - type: recall_at_3 |
| value: 43.964 |
| - type: recall_at_5 |
| value: 53.891 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/mtop_domain |
| name: MTEB MTOPDomainClassification (en) |
| config: en |
| split: test |
| revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
| metrics: |
| - type: accuracy |
| value: 93.86228910168718 |
| - type: f1 |
| value: 93.69177113259104 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/mtop_intent |
| name: MTEB MTOPIntentClassification (en) |
| config: en |
| split: test |
| revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
| metrics: |
| - type: accuracy |
| value: 76.3999088007296 |
| - type: f1 |
| value: 58.96668664333438 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/amazon_massive_intent |
| name: MTEB MassiveIntentClassification (en) |
| config: en |
| split: test |
| revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| metrics: |
| - type: accuracy |
| value: 73.21788836583727 |
| - type: f1 |
| value: 71.4545936552952 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/amazon_massive_scenario |
| name: MTEB MassiveScenarioClassification (en) |
| config: en |
| split: test |
| revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| metrics: |
| - type: accuracy |
| value: 77.39071956960323 |
| - type: f1 |
| value: 77.12398952847603 |
| - task: |
| type: Clustering |
| dataset: |
| type: mteb/medrxiv-clustering-p2p |
| name: MTEB MedrxivClusteringP2P |
| config: default |
| split: test |
| revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
| metrics: |
| - type: v_measure |
| value: 32.255379528166955 |
| - task: |
| type: Clustering |
| dataset: |
| type: mteb/medrxiv-clustering-s2s |
| name: MTEB MedrxivClusteringS2S |
| config: default |
| split: test |
| revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
| metrics: |
| - type: v_measure |
| value: 29.66423362872814 |
| - task: |
| type: Reranking |
| dataset: |
| type: mteb/mind_small |
| name: MTEB MindSmallReranking |
| config: default |
| split: test |
| revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
| metrics: |
| - type: map |
| value: 30.782211620375964 |
| - type: mrr |
| value: 31.773479703044956 |
| - task: |
| type: Retrieval |
| dataset: |
| type: nfcorpus |
| name: MTEB NFCorpus |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 5.863 |
| - type: map_at_10 |
| value: 13.831 |
| - type: map_at_100 |
| value: 17.534 |
| - type: map_at_1000 |
| value: 19.012 |
| - type: map_at_3 |
| value: 10.143 |
| - type: map_at_5 |
| value: 12.034 |
| - type: mrr_at_1 |
| value: 46.749 |
| - type: mrr_at_10 |
| value: 55.376999999999995 |
| - type: mrr_at_100 |
| value: 56.009 |
| - type: mrr_at_1000 |
| value: 56.042 |
| - type: mrr_at_3 |
| value: 53.30200000000001 |
| - type: mrr_at_5 |
| value: 54.85 |
| - type: ndcg_at_1 |
| value: 44.582 |
| - type: ndcg_at_10 |
| value: 36.07 |
| - type: ndcg_at_100 |
| value: 33.39 |
| - type: ndcg_at_1000 |
| value: 41.884 |
| - type: ndcg_at_3 |
| value: 41.441 |
| - type: ndcg_at_5 |
| value: 39.861000000000004 |
| - type: precision_at_1 |
| value: 46.129999999999995 |
| - type: precision_at_10 |
| value: 26.594 |
| - type: precision_at_100 |
| value: 8.365 |
| - type: precision_at_1000 |
| value: 2.1260000000000003 |
| - type: precision_at_3 |
| value: 39.009 |
| - type: precision_at_5 |
| value: 34.861 |
| - type: recall_at_1 |
| value: 5.863 |
| - type: recall_at_10 |
| value: 17.961 |
| - type: recall_at_100 |
| value: 34.026 |
| - type: recall_at_1000 |
| value: 64.46499999999999 |
| - type: recall_at_3 |
| value: 11.242 |
| - type: recall_at_5 |
| value: 14.493 |
| - task: |
| type: Retrieval |
| dataset: |
| type: nq |
| name: MTEB NQ |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 38.601 |
| - type: map_at_10 |
| value: 55.293000000000006 |
| - type: map_at_100 |
| value: 56.092 |
| - type: map_at_1000 |
| value: 56.111999999999995 |
| - type: map_at_3 |
| value: 51.269 |
| - type: map_at_5 |
| value: 53.787 |
| - type: mrr_at_1 |
| value: 43.221 |
| - type: mrr_at_10 |
| value: 57.882999999999996 |
| - type: mrr_at_100 |
| value: 58.408 |
| - type: mrr_at_1000 |
| value: 58.421 |
| - type: mrr_at_3 |
| value: 54.765 |
| - type: mrr_at_5 |
| value: 56.809 |
| - type: ndcg_at_1 |
| value: 43.221 |
| - type: ndcg_at_10 |
| value: 62.858999999999995 |
| - type: ndcg_at_100 |
| value: 65.987 |
| - type: ndcg_at_1000 |
| value: 66.404 |
| - type: ndcg_at_3 |
| value: 55.605000000000004 |
| - type: ndcg_at_5 |
| value: 59.723000000000006 |
| - type: precision_at_1 |
| value: 43.221 |
| - type: precision_at_10 |
| value: 9.907 |
| - type: precision_at_100 |
| value: 1.169 |
| - type: precision_at_1000 |
| value: 0.121 |
| - type: precision_at_3 |
| value: 25.019000000000002 |
| - type: precision_at_5 |
| value: 17.474 |
| - type: recall_at_1 |
| value: 38.601 |
| - type: recall_at_10 |
| value: 82.966 |
| - type: recall_at_100 |
| value: 96.154 |
| - type: recall_at_1000 |
| value: 99.223 |
| - type: recall_at_3 |
| value: 64.603 |
| - type: recall_at_5 |
| value: 73.97200000000001 |
| - task: |
| type: Retrieval |
| dataset: |
| type: quora |
| name: MTEB QuoraRetrieval |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 70.77 |
| - type: map_at_10 |
| value: 84.429 |
| - type: map_at_100 |
| value: 85.04599999999999 |
| - type: map_at_1000 |
| value: 85.065 |
| - type: map_at_3 |
| value: 81.461 |
| - type: map_at_5 |
| value: 83.316 |
| - type: mrr_at_1 |
| value: 81.51 |
| - type: mrr_at_10 |
| value: 87.52799999999999 |
| - type: mrr_at_100 |
| value: 87.631 |
| - type: mrr_at_1000 |
| value: 87.632 |
| - type: mrr_at_3 |
| value: 86.533 |
| - type: mrr_at_5 |
| value: 87.214 |
| - type: ndcg_at_1 |
| value: 81.47999999999999 |
| - type: ndcg_at_10 |
| value: 88.181 |
| - type: ndcg_at_100 |
| value: 89.39200000000001 |
| - type: ndcg_at_1000 |
| value: 89.52 |
| - type: ndcg_at_3 |
| value: 85.29299999999999 |
| - type: ndcg_at_5 |
| value: 86.88 |
| - type: precision_at_1 |
| value: 81.47999999999999 |
| - type: precision_at_10 |
| value: 13.367 |
| - type: precision_at_100 |
| value: 1.5230000000000001 |
| - type: precision_at_1000 |
| value: 0.157 |
| - type: precision_at_3 |
| value: 37.227 |
| - type: precision_at_5 |
| value: 24.494 |
| - type: recall_at_1 |
| value: 70.77 |
| - type: recall_at_10 |
| value: 95.199 |
| - type: recall_at_100 |
| value: 99.37700000000001 |
| - type: recall_at_1000 |
| value: 99.973 |
| - type: recall_at_3 |
| value: 86.895 |
| - type: recall_at_5 |
| value: 91.396 |
| - task: |
| type: Clustering |
| dataset: |
| type: mteb/reddit-clustering |
| name: MTEB RedditClustering |
| config: default |
| split: test |
| revision: 24640382cdbf8abc73003fb0fa6d111a705499eb |
| metrics: |
| - type: v_measure |
| value: 50.686353396858344 |
| - task: |
| type: Clustering |
| dataset: |
| type: mteb/reddit-clustering-p2p |
| name: MTEB RedditClusteringP2P |
| config: default |
| split: test |
| revision: 282350215ef01743dc01b456c7f5241fa8937f16 |
| metrics: |
| - type: v_measure |
| value: 61.3664675312921 |
| - task: |
| type: Retrieval |
| dataset: |
| type: scidocs |
| name: MTEB SCIDOCS |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 4.7379999999999995 |
| - type: map_at_10 |
| value: 12.01 |
| - type: map_at_100 |
| value: 14.02 |
| - type: map_at_1000 |
| value: 14.310999999999998 |
| - type: map_at_3 |
| value: 8.459 |
| - type: map_at_5 |
| value: 10.281 |
| - type: mrr_at_1 |
| value: 23.3 |
| - type: mrr_at_10 |
| value: 34.108 |
| - type: mrr_at_100 |
| value: 35.217 |
| - type: mrr_at_1000 |
| value: 35.272 |
| - type: mrr_at_3 |
| value: 30.833 |
| - type: mrr_at_5 |
| value: 32.768 |
| - type: ndcg_at_1 |
| value: 23.3 |
| - type: ndcg_at_10 |
| value: 20.116999999999997 |
| - type: ndcg_at_100 |
| value: 27.961000000000002 |
| - type: ndcg_at_1000 |
| value: 33.149 |
| - type: ndcg_at_3 |
| value: 18.902 |
| - type: ndcg_at_5 |
| value: 16.742 |
| - type: precision_at_1 |
| value: 23.3 |
| - type: precision_at_10 |
| value: 10.47 |
| - type: precision_at_100 |
| value: 2.177 |
| - type: precision_at_1000 |
| value: 0.34299999999999997 |
| - type: precision_at_3 |
| value: 17.567 |
| - type: precision_at_5 |
| value: 14.78 |
| - type: recall_at_1 |
| value: 4.7379999999999995 |
| - type: recall_at_10 |
| value: 21.221999999999998 |
| - type: recall_at_100 |
| value: 44.242 |
| - type: recall_at_1000 |
| value: 69.652 |
| - type: recall_at_3 |
| value: 10.688 |
| - type: recall_at_5 |
| value: 14.982999999999999 |
| - task: |
| type: STS |
| dataset: |
| type: mteb/sickr-sts |
| name: MTEB SICK-R |
| config: default |
| split: test |
| revision: a6ea5a8cab320b040a23452cc28066d9beae2cee |
| metrics: |
| - type: cos_sim_pearson |
| value: 84.84572946827069 |
| - type: cos_sim_spearman |
| value: 80.48508130408966 |
| - type: euclidean_pearson |
| value: 82.0481530027767 |
| - type: euclidean_spearman |
| value: 80.45902876782752 |
| - type: manhattan_pearson |
| value: 82.03728222483326 |
| - type: manhattan_spearman |
| value: 80.45684282911755 |
| - task: |
| type: STS |
| dataset: |
| type: mteb/sts12-sts |
| name: MTEB STS12 |
| config: default |
| split: test |
| revision: a0d554a64d88156834ff5ae9920b964011b16384 |
| metrics: |
| - type: cos_sim_pearson |
| value: 84.33476464677516 |
| - type: cos_sim_spearman |
| value: 75.93057758003266 |
| - type: euclidean_pearson |
| value: 80.89685744015691 |
| - type: euclidean_spearman |
| value: 76.29929953441706 |
| - type: manhattan_pearson |
| value: 80.91391345459995 |
| - type: manhattan_spearman |
| value: 76.31985463110914 |
| - task: |
| type: STS |
| dataset: |
| type: mteb/sts13-sts |
| name: MTEB STS13 |
| config: default |
| split: test |
| revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
| metrics: |
| - type: cos_sim_pearson |
| value: 84.63686106359005 |
| - type: cos_sim_spearman |
| value: 85.22240034668202 |
| - type: euclidean_pearson |
| value: 84.6074814189106 |
| - type: euclidean_spearman |
| value: 85.17169644755828 |
| - type: manhattan_pearson |
| value: 84.48329306239368 |
| - type: manhattan_spearman |
| value: 85.0086508544768 |
| - task: |
| type: STS |
| dataset: |
| type: mteb/sts14-sts |
| name: MTEB STS14 |
| config: default |
| split: test |
| revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
| metrics: |
| - type: cos_sim_pearson |
| value: 82.95455774064745 |
| - type: cos_sim_spearman |
| value: 80.54074646118492 |
| - type: euclidean_pearson |
| value: 81.79598955554704 |
| - type: euclidean_spearman |
| value: 80.55837617606814 |
| - type: manhattan_pearson |
| value: 81.78213797905386 |
| - type: manhattan_spearman |
| value: 80.5666746878273 |
| - task: |
| type: STS |
| dataset: |
| type: mteb/sts15-sts |
| name: MTEB STS15 |
| config: default |
| split: test |
| revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
| metrics: |
| - type: cos_sim_pearson |
| value: 87.92813309124739 |
| - type: cos_sim_spearman |
| value: 88.81459873052108 |
| - type: euclidean_pearson |
| value: 88.21193118930564 |
| - type: euclidean_spearman |
| value: 88.87072745043731 |
| - type: manhattan_pearson |
| value: 88.22576929706727 |
| - type: manhattan_spearman |
| value: 88.8867671095791 |
| - task: |
| type: STS |
| dataset: |
| type: mteb/sts16-sts |
| name: MTEB STS16 |
| config: default |
| split: test |
| revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 |
| metrics: |
| - type: cos_sim_pearson |
| value: 83.6881529671839 |
| - type: cos_sim_spearman |
| value: 85.2807092969554 |
| - type: euclidean_pearson |
| value: 84.62334178652704 |
| - type: euclidean_spearman |
| value: 85.2116373296784 |
| - type: manhattan_pearson |
| value: 84.54948211541777 |
| - type: manhattan_spearman |
| value: 85.10737722637882 |
| - task: |
| type: STS |
| dataset: |
| type: mteb/sts17-crosslingual-sts |
| name: MTEB STS17 (en-en) |
| config: en-en |
| split: test |
| revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| metrics: |
| - type: cos_sim_pearson |
| value: 88.55963694458408 |
| - type: cos_sim_spearman |
| value: 89.36731628848683 |
| - type: euclidean_pearson |
| value: 89.64975952985465 |
| - type: euclidean_spearman |
| value: 89.29689484033007 |
| - type: manhattan_pearson |
| value: 89.61234491713135 |
| - type: manhattan_spearman |
| value: 89.20302520255782 |
| - task: |
| type: STS |
| dataset: |
| type: mteb/sts22-crosslingual-sts |
| name: MTEB STS22 (en) |
| config: en |
| split: test |
| revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| metrics: |
| - type: cos_sim_pearson |
| value: 62.411800961903886 |
| - type: cos_sim_spearman |
| value: 62.99105515749963 |
| - type: euclidean_pearson |
| value: 65.29826669549443 |
| - type: euclidean_spearman |
| value: 63.29880964105775 |
| - type: manhattan_pearson |
| value: 65.00126190601183 |
| - type: manhattan_spearman |
| value: 63.32011025899179 |
| - task: |
| type: STS |
| dataset: |
| type: mteb/stsbenchmark-sts |
| name: MTEB STSBenchmark |
| config: default |
| split: test |
| revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
| metrics: |
| - type: cos_sim_pearson |
| value: 85.83498531837608 |
| - type: cos_sim_spearman |
| value: 87.21366640615442 |
| - type: euclidean_pearson |
| value: 86.74764288798261 |
| - type: euclidean_spearman |
| value: 87.06060470780834 |
| - type: manhattan_pearson |
| value: 86.65971223951476 |
| - type: manhattan_spearman |
| value: 86.99814399831457 |
| - task: |
| type: Reranking |
| dataset: |
| type: mteb/scidocs-reranking |
| name: MTEB SciDocsRR |
| config: default |
| split: test |
| revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
| metrics: |
| - type: map |
| value: 83.94448463485881 |
| - type: mrr |
| value: 95.36291867174221 |
| - task: |
| type: Retrieval |
| dataset: |
| type: scifact |
| name: MTEB SciFact |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 59.928000000000004 |
| - type: map_at_10 |
| value: 68.577 |
| - type: map_at_100 |
| value: 69.35900000000001 |
| - type: map_at_1000 |
| value: 69.37299999999999 |
| - type: map_at_3 |
| value: 66.217 |
| - type: map_at_5 |
| value: 67.581 |
| - type: mrr_at_1 |
| value: 63 |
| - type: mrr_at_10 |
| value: 69.994 |
| - type: mrr_at_100 |
| value: 70.553 |
| - type: mrr_at_1000 |
| value: 70.56700000000001 |
| - type: mrr_at_3 |
| value: 68.167 |
| - type: mrr_at_5 |
| value: 69.11699999999999 |
| - type: ndcg_at_1 |
| value: 63 |
| - type: ndcg_at_10 |
| value: 72.58 |
| - type: ndcg_at_100 |
| value: 75.529 |
| - type: ndcg_at_1000 |
| value: 76.009 |
| - type: ndcg_at_3 |
| value: 68.523 |
| - type: ndcg_at_5 |
| value: 70.301 |
| - type: precision_at_1 |
| value: 63 |
| - type: precision_at_10 |
| value: 9.333 |
| - type: precision_at_100 |
| value: 1.09 |
| - type: precision_at_1000 |
| value: 0.11299999999999999 |
| - type: precision_at_3 |
| value: 26.444000000000003 |
| - type: precision_at_5 |
| value: 17.067 |
| - type: recall_at_1 |
| value: 59.928000000000004 |
| - type: recall_at_10 |
| value: 83.544 |
| - type: recall_at_100 |
| value: 96 |
| - type: recall_at_1000 |
| value: 100 |
| - type: recall_at_3 |
| value: 72.072 |
| - type: recall_at_5 |
| value: 76.683 |
| - task: |
| type: PairClassification |
| dataset: |
| type: mteb/sprintduplicatequestions-pairclassification |
| name: MTEB SprintDuplicateQuestions |
| config: default |
| split: test |
| revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
| metrics: |
| - type: cos_sim_accuracy |
| value: 99.82178217821782 |
| - type: cos_sim_ap |
| value: 95.41507679819003 |
| - type: cos_sim_f1 |
| value: 90.9456740442656 |
| - type: cos_sim_precision |
| value: 91.49797570850203 |
| - type: cos_sim_recall |
| value: 90.4 |
| - type: dot_accuracy |
| value: 99.77227722772277 |
| - type: dot_ap |
| value: 92.50123869445967 |
| - type: dot_f1 |
| value: 88.18414322250638 |
| - type: dot_precision |
| value: 90.26178010471205 |
| - type: dot_recall |
| value: 86.2 |
| - type: euclidean_accuracy |
| value: 99.81782178217821 |
| - type: euclidean_ap |
| value: 95.3935066749006 |
| - type: euclidean_f1 |
| value: 90.66128218071681 |
| - type: euclidean_precision |
| value: 91.53924566768603 |
| - type: euclidean_recall |
| value: 89.8 |
| - type: manhattan_accuracy |
| value: 99.81881188118813 |
| - type: manhattan_ap |
| value: 95.39767454613512 |
| - type: manhattan_f1 |
| value: 90.62019477191186 |
| - type: manhattan_precision |
| value: 92.95478443743428 |
| - type: manhattan_recall |
| value: 88.4 |
| - type: max_accuracy |
| value: 99.82178217821782 |
| - type: max_ap |
| value: 95.41507679819003 |
| - type: max_f1 |
| value: 90.9456740442656 |
| - task: |
| type: Clustering |
| dataset: |
| type: mteb/stackexchange-clustering |
| name: MTEB StackExchangeClustering |
| config: default |
| split: test |
| revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
| metrics: |
| - type: v_measure |
| value: 64.96313921233748 |
| - task: |
| type: Clustering |
| dataset: |
| type: mteb/stackexchange-clustering-p2p |
| name: MTEB StackExchangeClusteringP2P |
| config: default |
| split: test |
| revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
| metrics: |
| - type: v_measure |
| value: 33.602625720956745 |
| - task: |
| type: Reranking |
| dataset: |
| type: mteb/stackoverflowdupquestions-reranking |
| name: MTEB StackOverflowDupQuestions |
| config: default |
| split: test |
| revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
| metrics: |
| - type: map |
| value: 51.32659230651731 |
| - type: mrr |
| value: 52.33861726508785 |
| - task: |
| type: Summarization |
| dataset: |
| type: mteb/summeval |
| name: MTEB SummEval |
| config: default |
| split: test |
| revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c |
| metrics: |
| - type: cos_sim_pearson |
| value: 31.01587644214203 |
| - type: cos_sim_spearman |
| value: 30.974306908731013 |
| - type: dot_pearson |
| value: 29.83339853838187 |
| - type: dot_spearman |
| value: 30.07761671934048 |
| - task: |
| type: Retrieval |
| dataset: |
| type: trec-covid |
| name: MTEB TRECCOVID |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 0.22 |
| - type: map_at_10 |
| value: 1.9539999999999997 |
| - type: map_at_100 |
| value: 11.437 |
| - type: map_at_1000 |
| value: 27.861000000000004 |
| - type: map_at_3 |
| value: 0.6479999999999999 |
| - type: map_at_5 |
| value: 1.0410000000000001 |
| - type: mrr_at_1 |
| value: 84 |
| - type: mrr_at_10 |
| value: 90.333 |
| - type: mrr_at_100 |
| value: 90.333 |
| - type: mrr_at_1000 |
| value: 90.333 |
| - type: mrr_at_3 |
| value: 90.333 |
| - type: mrr_at_5 |
| value: 90.333 |
| - type: ndcg_at_1 |
| value: 80 |
| - type: ndcg_at_10 |
| value: 78.31700000000001 |
| - type: ndcg_at_100 |
| value: 59.396 |
| - type: ndcg_at_1000 |
| value: 52.733 |
| - type: ndcg_at_3 |
| value: 81.46900000000001 |
| - type: ndcg_at_5 |
| value: 80.74 |
| - type: precision_at_1 |
| value: 84 |
| - type: precision_at_10 |
| value: 84 |
| - type: precision_at_100 |
| value: 60.980000000000004 |
| - type: precision_at_1000 |
| value: 23.432 |
| - type: precision_at_3 |
| value: 87.333 |
| - type: precision_at_5 |
| value: 86.8 |
| - type: recall_at_1 |
| value: 0.22 |
| - type: recall_at_10 |
| value: 2.156 |
| - type: recall_at_100 |
| value: 14.557999999999998 |
| - type: recall_at_1000 |
| value: 49.553999999999995 |
| - type: recall_at_3 |
| value: 0.685 |
| - type: recall_at_5 |
| value: 1.121 |
| - task: |
| type: Retrieval |
| dataset: |
| type: webis-touche2020 |
| name: MTEB Touche2020 |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 3.373 |
| - type: map_at_10 |
| value: 11.701 |
| - type: map_at_100 |
| value: 17.144000000000002 |
| - type: map_at_1000 |
| value: 18.624 |
| - type: map_at_3 |
| value: 6.552 |
| - type: map_at_5 |
| value: 9.372 |
| - type: mrr_at_1 |
| value: 38.775999999999996 |
| - type: mrr_at_10 |
| value: 51.975 |
| - type: mrr_at_100 |
| value: 52.873999999999995 |
| - type: mrr_at_1000 |
| value: 52.873999999999995 |
| - type: mrr_at_3 |
| value: 47.619 |
| - type: mrr_at_5 |
| value: 50.578 |
| - type: ndcg_at_1 |
| value: 36.735 |
| - type: ndcg_at_10 |
| value: 27.212999999999997 |
| - type: ndcg_at_100 |
| value: 37.245 |
| - type: ndcg_at_1000 |
| value: 48.602000000000004 |
| - type: ndcg_at_3 |
| value: 30.916 |
| - type: ndcg_at_5 |
| value: 30.799 |
| - type: precision_at_1 |
| value: 38.775999999999996 |
| - type: precision_at_10 |
| value: 23.469 |
| - type: precision_at_100 |
| value: 7.327 |
| - type: precision_at_1000 |
| value: 1.486 |
| - type: precision_at_3 |
| value: 31.973000000000003 |
| - type: precision_at_5 |
| value: 32.245000000000005 |
| - type: recall_at_1 |
| value: 3.373 |
| - type: recall_at_10 |
| value: 17.404 |
| - type: recall_at_100 |
| value: 46.105000000000004 |
| - type: recall_at_1000 |
| value: 80.35 |
| - type: recall_at_3 |
| value: 7.4399999999999995 |
| - type: recall_at_5 |
| value: 12.183 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/toxic_conversations_50k |
| name: MTEB ToxicConversationsClassification |
| config: default |
| split: test |
| revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
| metrics: |
| - type: accuracy |
| value: 70.5592 |
| - type: ap |
| value: 14.330910591410134 |
| - type: f1 |
| value: 54.45745186286521 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/tweet_sentiment_extraction |
| name: MTEB TweetSentimentExtractionClassification |
| config: default |
| split: test |
| revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
| metrics: |
| - type: accuracy |
| value: 61.20543293718167 |
| - type: f1 |
| value: 61.45365480309872 |
| - task: |
| type: Clustering |
| dataset: |
| type: mteb/twentynewsgroups-clustering |
| name: MTEB TwentyNewsgroupsClustering |
| config: default |
| split: test |
| revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
| metrics: |
| - type: v_measure |
| value: 43.81162998944145 |
| - task: |
| type: PairClassification |
| dataset: |
| type: mteb/twittersemeval2015-pairclassification |
| name: MTEB TwitterSemEval2015 |
| config: default |
| split: test |
| revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
| metrics: |
| - type: cos_sim_accuracy |
| value: 86.69011146212075 |
| - type: cos_sim_ap |
| value: 76.09792353652536 |
| - type: cos_sim_f1 |
| value: 70.10202763786646 |
| - type: cos_sim_precision |
| value: 68.65671641791045 |
| - type: cos_sim_recall |
| value: 71.60949868073878 |
| - type: dot_accuracy |
| value: 85.33110806461227 |
| - type: dot_ap |
| value: 70.19304383327554 |
| - type: dot_f1 |
| value: 67.22494202525122 |
| - type: dot_precision |
| value: 65.6847935548842 |
| - type: dot_recall |
| value: 68.83905013192611 |
| - type: euclidean_accuracy |
| value: 86.5410979316922 |
| - type: euclidean_ap |
| value: 75.91906915651882 |
| - type: euclidean_f1 |
| value: 69.6798975672215 |
| - type: euclidean_precision |
| value: 67.6865671641791 |
| - type: euclidean_recall |
| value: 71.79419525065963 |
| - type: manhattan_accuracy |
| value: 86.60070334386363 |
| - type: manhattan_ap |
| value: 75.94617413885031 |
| - type: manhattan_f1 |
| value: 69.52689565780946 |
| - type: manhattan_precision |
| value: 68.3312101910828 |
| - type: manhattan_recall |
| value: 70.76517150395777 |
| - type: max_accuracy |
| value: 86.69011146212075 |
| - type: max_ap |
| value: 76.09792353652536 |
| - type: max_f1 |
| value: 70.10202763786646 |
| - task: |
| type: PairClassification |
| dataset: |
| type: mteb/twitterurlcorpus-pairclassification |
| name: MTEB TwitterURLCorpus |
| config: default |
| split: test |
| revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
| metrics: |
| - type: cos_sim_accuracy |
| value: 89.25951798812434 |
| - type: cos_sim_ap |
| value: 86.31476416599727 |
| - type: cos_sim_f1 |
| value: 78.52709971038477 |
| - type: cos_sim_precision |
| value: 76.7629972792117 |
| - type: cos_sim_recall |
| value: 80.37419156144134 |
| - type: dot_accuracy |
| value: 88.03896456708192 |
| - type: dot_ap |
| value: 83.26963599196237 |
| - type: dot_f1 |
| value: 76.72696459492317 |
| - type: dot_precision |
| value: 73.56411162133521 |
| - type: dot_recall |
| value: 80.17400677548507 |
| - type: euclidean_accuracy |
| value: 89.21682772538519 |
| - type: euclidean_ap |
| value: 86.29306071289969 |
| - type: euclidean_f1 |
| value: 78.40827030519554 |
| - type: euclidean_precision |
| value: 77.42250243939053 |
| - type: euclidean_recall |
| value: 79.41946412072683 |
| - type: manhattan_accuracy |
| value: 89.22458959133776 |
| - type: manhattan_ap |
| value: 86.2901934710645 |
| - type: manhattan_f1 |
| value: 78.54211378440453 |
| - type: manhattan_precision |
| value: 76.85505858079729 |
| - type: manhattan_recall |
| value: 80.30489682784109 |
| - type: max_accuracy |
| value: 89.25951798812434 |
| - type: max_ap |
| value: 86.31476416599727 |
| - type: max_f1 |
| value: 78.54211378440453 |
| language: |
| - en |
| license: mit |
| --- |
| # # Fast-Inference with Ctranslate2 |
| Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU. |
|
|
| quantized version of [intfloat/e5-large](https://huggingface.co/intfloat/e5-large) |
| ```bash |
| pip install hf-hub-ctranslate2>=2.12.0 ctranslate2>=3.17.1 |
| ``` |
|
|
| ```python |
| # from transformers import AutoTokenizer |
| model_name = "michaelfeil/ct2fast-e5-large" |
| model_name_orig="intfloat/e5-large" |
| |
| from hf_hub_ctranslate2 import EncoderCT2fromHfHub |
| model = EncoderCT2fromHfHub( |
| # load in int8 on CUDA |
| model_name_or_path=model_name, |
| device="cuda", |
| compute_type="int8_float16" |
| ) |
| outputs = model.generate( |
| text=["I like soccer", "I like tennis", "The eiffel tower is in Paris"], |
| max_length=64, |
| ) # perform downstream tasks on outputs |
| outputs["pooler_output"] |
| outputs["last_hidden_state"] |
| outputs["attention_mask"] |
| |
| # alternative, use SentenceTransformer Mix-In |
| # for end-to-end Sentence embeddings generation |
| # (not pulling from this CT2fast-HF repo) |
| |
| from hf_hub_ctranslate2 import CT2SentenceTransformer |
| model = CT2SentenceTransformer( |
| model_name_orig, compute_type="int8_float16", device="cuda" |
| ) |
| embeddings = model.encode( |
| ["I like soccer", "I like tennis", "The eiffel tower is in Paris"], |
| batch_size=32, |
| convert_to_numpy=True, |
| normalize_embeddings=True, |
| ) |
| print(embeddings.shape, embeddings) |
| scores = (embeddings @ embeddings.T) * 100 |
| |
| # Hint: you can also host this code via REST API and |
| # via github.com/michaelfeil/infinity |
| |
| |
| ``` |
|
|
| Checkpoint compatible to [ctranslate2>=3.17.1](https://github.com/OpenNMT/CTranslate2) |
| and [hf-hub-ctranslate2>=2.12.0](https://github.com/michaelfeil/hf-hub-ctranslate2) |
| - `compute_type=int8_float16` for `device="cuda"` |
| - `compute_type=int8` for `device="cpu"` |
|
|
| Converted on 2023-10-13 using |
| ``` |
| LLama-2 -> removed <pad> token. |
| ``` |
|
|
| # Licence and other remarks: |
| This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo. |
|
|
| # Original description |
| |
| |
| ## E5-large |
|
|
| **News (May 2023): please switch to [e5-large-v2](https://huggingface.co/intfloat/e5-large-v2), which has better performance and same method of usage.** |
|
|
| [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://arxiv.org/pdf/2212.03533.pdf). |
| Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei, arXiv 2022 |
|
|
| This model has 24 layers and the embedding size is 1024. |
|
|
| ## Usage |
|
|
| Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset. |
|
|
| ```python |
| import torch.nn.functional as F |
| |
| from torch import Tensor |
| from transformers import AutoTokenizer, AutoModel |
| |
| |
| def average_pool(last_hidden_states: Tensor, |
| attention_mask: Tensor) -> Tensor: |
| last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0) |
| return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None] |
| |
| |
| # Each input text should start with "query: " or "passage: ". |
| # For tasks other than retrieval, you can simply use the "query: " prefix. |
| input_texts = ['query: how much protein should a female eat', |
| 'query: summit define', |
| "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.", |
| "passage: Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments."] |
| |
| tokenizer = AutoTokenizer.from_pretrained('intfloat/e5-large') |
| model = AutoModel.from_pretrained('intfloat/e5-large') |
| |
| # Tokenize the input texts |
| batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt') |
| |
| outputs = model(**batch_dict) |
| embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask']) |
| |
| # normalize embeddings |
| embeddings = F.normalize(embeddings, p=2, dim=1) |
| scores = (embeddings[:2] @ embeddings[2:].T) * 100 |
| print(scores.tolist()) |
| ``` |
|
|
| ## Training Details |
|
|
| Please refer to our paper at [https://arxiv.org/pdf/2212.03533.pdf](https://arxiv.org/pdf/2212.03533.pdf). |
|
|
| ## Benchmark Evaluation |
|
|
| Check out [unilm/e5](https://github.com/microsoft/unilm/tree/master/e5) to reproduce evaluation results |
| on the [BEIR](https://arxiv.org/abs/2104.08663) and [MTEB benchmark](https://arxiv.org/abs/2210.07316). |
|
|
| ## Support for Sentence Transformers |
|
|
| Below is an example for usage with sentence_transformers. |
| ```python |
| from sentence_transformers import SentenceTransformer |
| model = SentenceTransformer('intfloat/e5-large') |
| input_texts = [ |
| 'query: how much protein should a female eat', |
| 'query: summit define', |
| "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.", |
| "passage: Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments." |
| ] |
| embeddings = model.encode(input_texts, normalize_embeddings=True) |
| ``` |
| |
| Package requirements |
| |
| `pip install sentence_transformers~=2.2.2` |
|
|
| Contributors: [michaelfeil](https://huggingface.co/michaelfeil) |
|
|
| ## FAQ |
|
|
| **1. Do I need to add the prefix "query: " and "passage: " to input texts?** |
|
|
| Yes, this is how the model is trained, otherwise you will see a performance degradation. |
|
|
| Here are some rules of thumb: |
| - Use "query: " and "passage: " correspondingly for asymmetric tasks such as passage retrieval in open QA, ad-hoc information retrieval. |
|
|
| - Use "query: " prefix for symmetric tasks such as semantic similarity, paraphrase retrieval. |
|
|
| - Use "query: " prefix if you want to use embeddings as features, such as linear probing classification, clustering. |
|
|
| **2. Why are my reproduced results slightly different from reported in the model card?** |
|
|
| Different versions of `transformers` and `pytorch` could cause negligible but non-zero performance differences. |
|
|
| **3. Why does the cosine similarity scores distribute around 0.7 to 1.0?** |
|
|
| This is a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss. |
|
|
| For text embedding tasks like text retrieval or semantic similarity, |
| what matters is the relative order of the scores instead of the absolute values, |
| so this should not be an issue. |
|
|
| ## Citation |
|
|
| If you find our paper or models helpful, please consider cite as follows: |
|
|
| ``` |
| @article{wang2022text, |
| title={Text Embeddings by Weakly-Supervised Contrastive Pre-training}, |
| author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Jiao, Binxing and Yang, Linjun and Jiang, Daxin and Majumder, Rangan and Wei, Furu}, |
| journal={arXiv preprint arXiv:2212.03533}, |
| year={2022} |
| } |
| ``` |
|
|
| ## Limitations |
|
|
| This model only works for English texts. Long texts will be truncated to at most 512 tokens. |
|
|
