| --- |
| title: mrr |
| tags: |
| - evaluate |
| - metric |
| description: "This is the mean reciprocal rank (mrr) metric for retrieval systems. |
| It is the average of the precision scores computer after each relevant document is got. You can refer to [here](https://amenra.github.io/ranx/metrics/#mean-reciprocal-rank)" |
| sdk: gradio |
| sdk_version: 3.19.1 |
| app_file: app.py |
| pinned: false |
| --- |
| |
| # Metric Card for mrr |
|
|
| ## Metric Description |
| This is the mean reciprocal rank (mrr) metric for retrieval systems. |
| It is the average of the precision scores computer after each relevant document is got. You can refer to [here](https://amenra.github.io/ranx/metrics/#mean-reciprocal-rank) |
|
|
| ## How to Use |
| ```python |
| >>> my_new_module = evaluate.load("mrr") |
| >>> references= [json.dumps({"q_1":{"d_1":1, "d_2":2} }), |
| json.dumps({"q_2":{"d_2":1, "d_3":2, "d_5":3}})] |
| >>> predictions = [json.dumps({"q_1": { "d_1": 0.8, "d_2": 0.9}}), |
| json.dumps({"q_2": {"d_2": 0.9, "d_1": 0.8, "d_5": 0.7, "d_3": 0.3}})] |
| >>> results = my_new_module.compute(references=references, predictions=predictions) |
| >>> print(results) |
| {'mrr': 1.0} |
| ``` |
| ### Inputs |
| - **predictions:** a list of dictionaries where each dictionary consists of document relevancy scores produced by the model for a given query. One dictionary per query. The dictionaries should be converted to string. |
| - **references:** a lift of list of dictionaries where each dictionary consists of the relevant order for the documents for a given query in a sorted relevancy order. The dictionaries should be converted to string. |
| - **k:** an optional paramater whose default is None to calculate mrr@k |
|
|
| ### Output Values |
| - **mrr (`float`):** mean reciprocal rank. Minimum possible value is 0. Maximum possible value is 1.0 |
|
|
|
|
| ## Limitations and Bias |
| *Note any known limitations or biases that the metric has, with links and references if possible.* |
|
|
| ## Citation |
| ```bibtex |
| @inproceedings{ranx, |
| author = {Elias Bassani}, |
| title = {ranx: {A} Blazing-Fast Python Library for Ranking Evaluation and Comparison}, |
| booktitle = {{ECIR} {(2)}}, |
| series = {Lecture Notes in Computer Science}, |
| volume = {13186}, |
| pages = {259--264}, |
| publisher = {Springer}, |
| year = {2022}, |
| doi = {10.1007/978-3-030-99739-7\_30} |
| } |
| ``` |
