# Processors

在 Transformers 库中，processors可以有两种不同的含义：
- 为多模态模型，例如[Wav2Vec2](../model_doc/wav2vec2)（语音和文本）或[CLIP](../model_doc/clip)（文本和视觉）预处理输入的对象
- 在库的旧版本中用于预处理GLUE或SQUAD数据的已弃用对象。

## 多模态processors[[transformers.ProcessorMixin]]

任何多模态模型都需要一个对象来编码或解码将多个模态（包括文本、视觉和音频）组合在一起的数据。这由称为processors的对象处理，这些processors将两个或多个处理对象组合在一起，例如tokenizers（用于文本模态），image processors（用于视觉）和feature extractors（用于音频）。

这些processors继承自以下实现保存和加载功能的基类：


<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class transformers.ProcessorMixin</name><anchor>transformers.ProcessorMixin</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L484</source><parameters>[{"name": "*args", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

This is a mixin used to provide saving/loading functionality for all processor classes.



<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>apply_chat_template</name><anchor>transformers.ProcessorMixin.apply_chat_template</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L1522</source><parameters>[{"name": "conversation", "val": ": typing.Union[list[dict[str, str]], list[list[dict[str, str]]]]"}, {"name": "chat_template", "val": ": typing.Optional[str] = None"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.processing_utils.AllKwargsForChatTemplate]"}]</parameters><paramsdesc>- **conversation** (`Union[list[Dict, [str, str]], list[list[dict[str, str]]]]`) --
  The conversation to format.
- **chat_template** (`Optional[str]`, *optional*) --
  The Jinja template to use for formatting the conversation. If not provided, the tokenizer's
  chat template is used.</paramsdesc><paramgroups>0</paramgroups></docstring>

Similar to the `apply_chat_template` method on tokenizers, this method applies a Jinja template to input
conversations to turn them into a single tokenizable string.

The input is expected to be in the following format, where each message content is a list consisting of text and
optionally image or video inputs. One can also provide an image, video, URL or local path which will be used to form
`pixel_values` when `return_dict=True`. If not provided, one will get only the formatted text, optionally tokenized text.

conversation = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"},
{"type": "text", "text": "Please describe this image in detail."},
],
},
]




</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>batch_decode</name><anchor>transformers.ProcessorMixin.batch_decode</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L1482</source><parameters>[{"name": "*args", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

This method forwards all its arguments to PreTrainedTokenizer's [batch_decode()](/docs/transformers/v4.57.0/zh/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.batch_decode). Please
refer to the docstring of this method for more information.


</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>check_argument_for_proper_class</name><anchor>transformers.ProcessorMixin.check_argument_for_proper_class</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L599</source><parameters>[{"name": "argument_name", "val": ""}, {"name": "argument", "val": ""}]</parameters></docstring>

Checks the passed argument's class against the expected transformers class. In case of an unexpected
mismatch between expected and actual class, an error is raise. Otherwise, the proper retrieved class
is returned.


</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>decode</name><anchor>transformers.ProcessorMixin.decode</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L1491</source><parameters>[{"name": "*args", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

This method forwards all its arguments to PreTrainedTokenizer's [decode()](/docs/transformers/v4.57.0/zh/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.decode). Please refer to
the docstring of this method for more information.


</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>from_args_and_dict</name><anchor>transformers.ProcessorMixin.from_args_and_dict</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L1149</source><parameters>[{"name": "args", "val": ""}, {"name": "processor_dict", "val": ": dict"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **processor_dict** (`dict[str, Any]`) --
  Dictionary that will be used to instantiate the processor object. Such a dictionary can be
  retrieved from a pretrained checkpoint by leveraging the
  `~processing_utils.ProcessingMixin.to_dict` method.
- **kwargs** (`dict[str, Any]`) --
  Additional parameters from which to initialize the processor object.</paramsdesc><paramgroups>0</paramgroups><rettype>`~processing_utils.ProcessingMixin`</rettype><retdesc>The processor object instantiated from those
parameters.</retdesc></docstring>

Instantiates a type of `~processing_utils.ProcessingMixin` from a Python dictionary of parameters.








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>from_pretrained</name><anchor>transformers.ProcessorMixin.from_pretrained</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L1335</source><parameters>[{"name": "pretrained_model_name_or_path", "val": ": typing.Union[str, os.PathLike]"}, {"name": "cache_dir", "val": ": typing.Union[str, os.PathLike, NoneType] = None"}, {"name": "force_download", "val": ": bool = False"}, {"name": "local_files_only", "val": ": bool = False"}, {"name": "token", "val": ": typing.Union[str, bool, NoneType] = None"}, {"name": "revision", "val": ": str = 'main'"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **pretrained_model_name_or_path** (`str` or `os.PathLike`) --
  This can be either:

  - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on
    huggingface.co.
  - a path to a *directory* containing a feature extractor file saved using the
    [save_pretrained()](/docs/transformers/v4.57.0/zh/main_classes/feature_extractor#transformers.FeatureExtractionMixin.save_pretrained) method, e.g., `./my_model_directory/`.
  - a path or url to a saved feature extractor JSON *file*, e.g.,
    `./my_model_directory/preprocessor_config.json`.
- ****kwargs** --
  Additional keyword arguments passed along to both
  [from_pretrained()](/docs/transformers/v4.57.0/zh/main_classes/feature_extractor#transformers.FeatureExtractionMixin.from_pretrained) and
  `~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`.</paramsdesc><paramgroups>0</paramgroups></docstring>

Instantiate a processor associated with a pretrained model.

<Tip>

This class method is simply calling the feature extractor
[from_pretrained()](/docs/transformers/v4.57.0/zh/main_classes/feature_extractor#transformers.FeatureExtractionMixin.from_pretrained), image processor
[ImageProcessingMixin](/docs/transformers/v4.57.0/zh/main_classes/image_processor#transformers.ImageProcessingMixin) and the tokenizer
`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained` methods. Please refer to the docstrings of the
methods above for more information.

</Tip>




</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_processor_dict</name><anchor>transformers.ProcessorMixin.get_processor_dict</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L890</source><parameters>[{"name": "pretrained_model_name_or_path", "val": ": typing.Union[str, os.PathLike]"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **pretrained_model_name_or_path** (`str` or `os.PathLike`) --
  The identifier of the pre-trained checkpoint from which we want the dictionary of parameters.
- **subfolder** (`str`, *optional*, defaults to `""`) --
  In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can
  specify the folder name here.</paramsdesc><paramgroups>0</paramgroups><rettype>`tuple[Dict, Dict]`</rettype><retdesc>The dictionary(ies) that will be used to instantiate the processor object.</retdesc></docstring>

From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a
processor of type `~processing_utils.ProcessingMixin` using `from_args_and_dict`.








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>post_process_image_text_to_text</name><anchor>transformers.ProcessorMixin.post_process_image_text_to_text</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L1742</source><parameters>[{"name": "generated_outputs", "val": ""}, {"name": "skip_special_tokens", "val": " = True"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **generated_outputs** (`torch.Tensor` or `np.ndarray`) --
  The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
  or `(sequence_length,)`.
- **skip_special_tokens** (`bool`, *optional*, defaults to `True`) --
  Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
- ****kwargs** --
  Additional arguments to be passed to the tokenizer's `batch_decode method`.</paramsdesc><paramgroups>0</paramgroups><rettype>`list[str]`</rettype><retdesc>The decoded text.</retdesc></docstring>

Post-process the output of a vlm to decode the text.








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>push_to_hub</name><anchor>transformers.ProcessorMixin.push_to_hub</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/utils/hub.py#L848</source><parameters>[{"name": "repo_id", "val": ": str"}, {"name": "use_temp_dir", "val": ": typing.Optional[bool] = None"}, {"name": "commit_message", "val": ": typing.Optional[str] = None"}, {"name": "private", "val": ": typing.Optional[bool] = None"}, {"name": "token", "val": ": typing.Union[bool, str, NoneType] = None"}, {"name": "max_shard_size", "val": ": typing.Union[str, int, NoneType] = '5GB'"}, {"name": "create_pr", "val": ": bool = False"}, {"name": "safe_serialization", "val": ": bool = True"}, {"name": "revision", "val": ": typing.Optional[str] = None"}, {"name": "commit_description", "val": ": typing.Optional[str] = None"}, {"name": "tags", "val": ": typing.Optional[list[str]] = None"}, {"name": "**deprecated_kwargs", "val": ""}]</parameters><paramsdesc>- **repo_id** (`str`) --
  The name of the repository you want to push your processor to. It should contain your organization name
  when pushing to a given organization.
- **use_temp_dir** (`bool`, *optional*) --
  Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub.
  Will default to `True` if there is no directory named like `repo_id`, `False` otherwise.
- **commit_message** (`str`, *optional*) --
  Message to commit while pushing. Will default to `"Upload processor"`.
- **private** (`bool`, *optional*) --
  Whether to make the repo private. If `None` (default), the repo will be public unless the organization's default is private. This value is ignored if the repo already exists.
- **token** (`bool` or `str`, *optional*) --
  The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
  when running `hf auth login` (stored in `~/.huggingface`). Will default to `True` if `repo_url`
  is not specified.
- **max_shard_size** (`int` or `str`, *optional*, defaults to `"5GB"`) --
  Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard
  will then be each of size lower than this size. If expressed as a string, needs to be digits followed
  by a unit (like `"5MB"`). We default it to `"5GB"` so that users can easily load models on free-tier
  Google Colab instances without any CPU OOM issues.
- **create_pr** (`bool`, *optional*, defaults to `False`) --
  Whether or not to create a PR with the uploaded files or directly commit.
- **safe_serialization** (`bool`, *optional*, defaults to `True`) --
  Whether or not to convert the model weights in safetensors format for safer serialization.
- **revision** (`str`, *optional*) --
  Branch to push the uploaded files to.
- **commit_description** (`str`, *optional*) --
  The description of the commit that will be created
- **tags** (`list[str]`, *optional*) --
  List of tags to push on the Hub.</paramsdesc><paramgroups>0</paramgroups></docstring>

Upload the processor files to the 🤗 Model Hub.



<ExampleCodeBlock anchor="transformers.ProcessorMixin.push_to_hub.example">

Examples:

```python
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("google-bert/bert-base-cased")

# Push the processor to your namespace with the name "my-finetuned-bert".
processor.push_to_hub("my-finetuned-bert")

# Push the processor to an organization with the name "my-finetuned-bert".
processor.push_to_hub("huggingface/my-finetuned-bert")
```

</ExampleCodeBlock>


</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>register_for_auto_class</name><anchor>transformers.ProcessorMixin.register_for_auto_class</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L1398</source><parameters>[{"name": "auto_class", "val": " = 'AutoProcessor'"}]</parameters><paramsdesc>- **auto_class** (`str` or `type`, *optional*, defaults to `"AutoProcessor"`) --
  The auto class to register this new feature extractor with.</paramsdesc><paramgroups>0</paramgroups></docstring>

Register this class with a given auto class. This should only be used for custom feature extractors as the ones
in the library are already mapped with `AutoProcessor`.






</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>save_pretrained</name><anchor>transformers.ProcessorMixin.save_pretrained</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L715</source><parameters>[{"name": "save_directory", "val": ""}, {"name": "push_to_hub", "val": ": bool = False"}, {"name": "legacy_serialization", "val": ": bool = True"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **save_directory** (`str` or `os.PathLike`) --
  Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will
  be created if it does not exist).
- **push_to_hub** (`bool`, *optional*, defaults to `False`) --
  Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the
  repository you want to push to with `repo_id` (will default to the name of `save_directory` in your
  namespace).
- **legacy_serialization** (`bool`, *optional*, defaults to `True`) --
  Whether or not to save processor attributes in separate config files (legacy) or in processor's config
  file as a nested dict. Saving all attributes in a single dict will become the default in future versions.
  Set to `legacy_serialization=True` until then.
- **kwargs** (`dict[str, Any]`, *optional*) --
  Additional key word arguments passed along to the [push_to_hub()](/docs/transformers/v4.57.0/zh/main_classes/model#transformers.utils.PushToHubMixin.push_to_hub) method.</paramsdesc><paramgroups>0</paramgroups></docstring>

Saves the attributes of this processor (feature extractor, tokenizer...) in the specified directory so that it
can be reloaded using the [from_pretrained()](/docs/transformers/v4.57.0/zh/main_classes/processors#transformers.ProcessorMixin.from_pretrained) method.

<Tip>

This class method is simply calling [save_pretrained()](/docs/transformers/v4.57.0/zh/main_classes/feature_extractor#transformers.FeatureExtractionMixin.save_pretrained) and
[save_pretrained()](/docs/transformers/v4.57.0/zh/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.save_pretrained). Please refer to the docstrings of the
methods above for more information.

</Tip>




</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>to_dict</name><anchor>transformers.ProcessorMixin.to_dict</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L620</source><parameters>[{"name": "legacy_serialization", "val": " = True"}]</parameters><rettype>`dict[str, Any]`</rettype><retdesc>Dictionary of all the attributes that make up this processor instance.</retdesc></docstring>

Serializes this instance to a Python dictionary.






</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>to_json_file</name><anchor>transformers.ProcessorMixin.to_json_file</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L699</source><parameters>[{"name": "json_file_path", "val": ": typing.Union[str, os.PathLike]"}, {"name": "legacy_serialization", "val": " = True"}]</parameters><paramsdesc>- **json_file_path** (`str` or `os.PathLike`) --
  Path to the JSON file in which this processor instance's parameters will be saved.</paramsdesc><paramgroups>0</paramgroups></docstring>

Save this instance to a JSON file.




</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>to_json_string</name><anchor>transformers.ProcessorMixin.to_json_string</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L688</source><parameters>[{"name": "legacy_serialization", "val": " = True"}]</parameters><rettype>`str`</rettype><retdesc>String containing all the attributes that make up this feature_extractor instance in JSON format.</retdesc></docstring>

Serializes this instance to a JSON string.






</div></div>

## 已弃用的processors[[transformers.DataProcessor]]

所有processor都遵循与 [DataProcessor](/docs/transformers/v4.57.0/zh/main_classes/processors#transformers.DataProcessor) 相同的架构。processor返回一个 [InputExample](/docs/transformers/v4.57.0/zh/main_classes/processors#transformers.InputExample) 列表。这些 [InputExample](/docs/transformers/v4.57.0/zh/main_classes/processors#transformers.InputExample) 可以转换为 [InputFeatures](/docs/transformers/v4.57.0/zh/main_classes/processors#transformers.InputFeatures) 以供输送到模型。

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class transformers.DataProcessor</name><anchor>transformers.DataProcessor</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/utils.py#L80</source><parameters>[]</parameters></docstring>
Base class for data converters for sequence classification data sets.


<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_dev_examples</name><anchor>transformers.DataProcessor.get_dev_examples</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/utils.py#L97</source><parameters>[{"name": "data_dir", "val": ""}]</parameters></docstring>
Gets a collection of [InputExample](/docs/transformers/v4.57.0/zh/main_classes/processors#transformers.InputExample) for the dev set.

</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_example_from_tensor_dict</name><anchor>transformers.DataProcessor.get_example_from_tensor_dict</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/utils.py#L83</source><parameters>[{"name": "tensor_dict", "val": ""}]</parameters><paramsdesc>- **tensor_dict** -- Keys and values should match the corresponding Glue
  tensorflow_dataset examples.</paramsdesc><paramgroups>0</paramgroups></docstring>

Gets an example from a dict with tensorflow tensors.




</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_labels</name><anchor>transformers.DataProcessor.get_labels</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/utils.py#L105</source><parameters>[]</parameters></docstring>
Gets the list of labels for this data set.

</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_test_examples</name><anchor>transformers.DataProcessor.get_test_examples</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/utils.py#L101</source><parameters>[{"name": "data_dir", "val": ""}]</parameters></docstring>
Gets a collection of [InputExample](/docs/transformers/v4.57.0/zh/main_classes/processors#transformers.InputExample) for the test set.

</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_train_examples</name><anchor>transformers.DataProcessor.get_train_examples</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/utils.py#L93</source><parameters>[{"name": "data_dir", "val": ""}]</parameters></docstring>
Gets a collection of [InputExample](/docs/transformers/v4.57.0/zh/main_classes/processors#transformers.InputExample) for the train set.

</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>tfds_map</name><anchor>transformers.DataProcessor.tfds_map</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/utils.py#L109</source><parameters>[{"name": "example", "val": ""}]</parameters></docstring>

Some tensorflow_datasets datasets are not formatted the same way the GLUE datasets are. This method converts
examples to the correct format.


</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class transformers.InputExample</name><anchor>transformers.InputExample</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/utils.py#L30</source><parameters>[{"name": "guid", "val": ": str"}, {"name": "text_a", "val": ": str"}, {"name": "text_b", "val": ": typing.Optional[str] = None"}, {"name": "label", "val": ": typing.Optional[str] = None"}]</parameters><paramsdesc>- **guid** -- Unique id for the example.
- **text_a** -- string. The untokenized text of the first sequence. For single
  sequence tasks, only this sequence must be specified.
- **text_b** -- (Optional) string. The untokenized text of the second sequence.
  Only must be specified for sequence pair tasks.
- **label** -- (Optional) string. The label of the example. This should be
  specified for train and dev examples, but not for test examples.</paramsdesc><paramgroups>0</paramgroups></docstring>

A single training/test example for simple sequence classification.





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>to_json_string</name><anchor>transformers.InputExample.to_json_string</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/utils.py#L49</source><parameters>[]</parameters></docstring>
Serializes this instance to a JSON string.

</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class transformers.InputFeatures</name><anchor>transformers.InputFeatures</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/utils.py#L55</source><parameters>[{"name": "input_ids", "val": ": list"}, {"name": "attention_mask", "val": ": typing.Optional[list[int]] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[list[int]] = None"}, {"name": "label", "val": ": typing.Union[int, float, NoneType] = None"}]</parameters><paramsdesc>- **input_ids** -- Indices of input sequence tokens in the vocabulary.
- **attention_mask** -- Mask to avoid performing attention on padding token indices.
  Mask values selected in `[0, 1]`: Usually `1` for tokens that are NOT MASKED, `0` for MASKED (padded)
  tokens.
- **token_type_ids** -- (Optional) Segment token indices to indicate first and second
  portions of the inputs. Only some models use them.
- **label** -- (Optional) Label corresponding to the input. Int for classification problems,
  float for regression problems.</paramsdesc><paramgroups>0</paramgroups></docstring>

A single set of features of data. Property names are the same names as the corresponding inputs to a model.





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>to_json_string</name><anchor>transformers.InputFeatures.to_json_string</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/utils.py#L75</source><parameters>[]</parameters></docstring>
Serializes this instance to a JSON string.

</div></div>

## GLUE[[transformers.glue_convert_examples_to_features]]

[General Language Understanding Evaluation (GLUE)](https://gluebenchmark.com/) 是一个基准测试，评估模型在各种现有的自然语言理解任务上的性能。它与论文 [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://openreview.net/pdf?id=rJ4km2R5t7) 一同发布。

该库为以下任务提供了总共10个processor：MRPC、MNLI、MNLI（mismatched）、CoLA、SST2、STSB、QQP、QNLI、RTE 和 WNLI。

这些processor是：

- `~data.processors.utils.MrpcProcessor`
- `~data.processors.utils.MnliProcessor`
- `~data.processors.utils.MnliMismatchedProcessor`
- `~data.processors.utils.Sst2Processor`
- `~data.processors.utils.StsbProcessor`
- `~data.processors.utils.QqpProcessor`
- `~data.processors.utils.QnliProcessor`
- `~data.processors.utils.RteProcessor`
- `~data.processors.utils.WnliProcessor`

此外，还可以使用以下方法从数据文件加载值并将其转换为 [InputExample](/docs/transformers/v4.57.0/zh/main_classes/processors#transformers.InputExample) 列表。

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>transformers.glue_convert_examples_to_features</name><anchor>transformers.glue_convert_examples_to_features</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/glue.py#L41</source><parameters>[{"name": "examples", "val": ": typing.Union[list[transformers.data.processors.utils.InputExample], ForwardRef('tf.data.Dataset')]"}, {"name": "tokenizer", "val": ": PreTrainedTokenizer"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "task", "val": " = None"}, {"name": "label_list", "val": " = None"}, {"name": "output_mode", "val": " = None"}]</parameters><paramsdesc>- **examples** -- List of `InputExamples` or `tf.data.Dataset` containing the examples.
- **tokenizer** -- Instance of a tokenizer that will tokenize the examples
- **max_length** -- Maximum example length. Defaults to the tokenizer's max_len
- **task** -- GLUE task
- **label_list** -- List of labels. Can be obtained from the processor using the `processor.get_labels()` method
- **output_mode** -- String indicating the output mode. Either `regression` or `classification`</paramsdesc><paramgroups>0</paramgroups><retdesc>If the `examples` input is a `tf.data.Dataset`, will return a `tf.data.Dataset` containing the task-specific
features. If the input is a list of `InputExamples`, will return a list of task-specific `InputFeatures` which
can be fed to the model.</retdesc></docstring>

Loads a data file into a list of `InputFeatures`






</div>

## XNLI

[跨语言NLI语料库（XNLI）](https://www.nyu.edu/projects/bowman/xnli/) 是一个评估跨语言文本表示质量的基准测试。XNLI是一个基于[*MultiNLI*](http://www.nyu.edu/projects/bowman/multinli/)的众包数据集：”文本对“被标记为包含15种不同语言（包括英语等高资源语言和斯瓦希里语等低资源语言）的文本蕴涵注释。

它与论文 [XNLI: Evaluating Cross-lingual Sentence Representations](https://huggingface.co/papers/1809.05053) 一同发布。

该库提供了加载XNLI数据的processor：

- `~data.processors.utils.XnliProcessor`

请注意，由于测试集上有“gold”标签，因此评估是在测试集上进行的。

使用这些processor的示例在 [run_xnli.py](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification/run_xnli.py) 脚本中提供。


## SQuAD

[斯坦福问答数据集（SQuAD）](https://rajpurkar.github.io/SQuAD-explorer//) 是一个评估模型在问答上性能的基准测试。有两个版本，v1.1 和 v2.0。第一个版本（v1.1）与论文 [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://huggingface.co/papers/1606.05250) 一同发布。第二个版本（v2.0）与论文 [Know What You Don't Know: Unanswerable Questions for SQuAD](https://huggingface.co/papers/1806.03822) 一同发布。

该库为两个版本各自提供了一个processor：

### Processors[[transformers.data.processors.squad.SquadProcessor]]

这两个processor是：

- `~data.processors.utils.SquadV1Processor`
- `~data.processors.utils.SquadV2Processor`

它们都继承自抽象类 `~data.processors.utils.SquadProcessor`。

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class transformers.data.processors.squad.SquadProcessor</name><anchor>transformers.data.processors.squad.SquadProcessor</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/squad.py#L541</source><parameters>[]</parameters></docstring>

Processor for the SQuAD data set. overridden by SquadV1Processor and SquadV2Processor, used by the version 1.1 and
version 2.0 of SQuAD, respectively.



<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_dev_examples</name><anchor>transformers.data.processors.squad.SquadProcessor.get_dev_examples</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/squad.py#L629</source><parameters>[{"name": "data_dir", "val": ""}, {"name": "filename", "val": " = None"}]</parameters><paramsdesc>- **data_dir** -- Directory containing the data files used for training and evaluating.
- **filename** -- None by default, specify this if the evaluation file has a different name than the original one
  which is `dev-v1.1.json` and `dev-v2.0.json` for squad versions 1.1 and 2.0 respectively.</paramsdesc><paramgroups>0</paramgroups></docstring>

Returns the evaluation example from the data directory.




</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_examples_from_dataset</name><anchor>transformers.data.processors.squad.SquadProcessor.get_examples_from_dataset</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/squad.py#L574</source><parameters>[{"name": "dataset", "val": ""}, {"name": "evaluate", "val": " = False"}]</parameters><paramsdesc>- **dataset** -- The tfds dataset loaded from *tensorflow_datasets.load("squad")*
- **evaluate** -- Boolean specifying if in evaluation mode or in training mode</paramsdesc><paramgroups>0</paramgroups><retdesc>List of SquadExample</retdesc></docstring>

Creates a list of `SquadExample` using a TFDS dataset.





<ExampleCodeBlock anchor="transformers.data.processors.squad.SquadProcessor.get_examples_from_dataset.example">

Examples:

```python
>>> import tensorflow_datasets as tfds

>>> dataset = tfds.load("squad")

>>> training_examples = get_examples_from_dataset(dataset, evaluate=False)
>>> evaluation_examples = get_examples_from_dataset(dataset, evaluate=True)
```

</ExampleCodeBlock>

</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_train_examples</name><anchor>transformers.data.processors.squad.SquadProcessor.get_train_examples</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/squad.py#L607</source><parameters>[{"name": "data_dir", "val": ""}, {"name": "filename", "val": " = None"}]</parameters><paramsdesc>- **data_dir** -- Directory containing the data files used for training and evaluating.
- **filename** -- None by default, specify this if the training file has a different name than the original one
  which is `train-v1.1.json` and `train-v2.0.json` for squad versions 1.1 and 2.0 respectively.</paramsdesc><paramgroups>0</paramgroups></docstring>

Returns the training examples from the data directory.




</div></div>

此外，可以使用以下方法将 SQuAD 示例转换为可用作模型输入的 `~data.processors.utils.SquadFeatures`。

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>transformers.squad_convert_examples_to_features</name><anchor>transformers.squad_convert_examples_to_features</anchor><source>https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/data/processors/squad.py#L317</source><parameters>[{"name": "examples", "val": ""}, {"name": "tokenizer", "val": ""}, {"name": "max_seq_length", "val": ""}, {"name": "doc_stride", "val": ""}, {"name": "max_query_length", "val": ""}, {"name": "is_training", "val": ""}, {"name": "padding_strategy", "val": " = 'max_length'"}, {"name": "return_dataset", "val": " = False"}, {"name": "threads", "val": " = 1"}, {"name": "tqdm_enabled", "val": " = True"}]</parameters><paramsdesc>- **examples** -- list of `SquadExample`
- **tokenizer** -- an instance of a child of [PreTrainedTokenizer](/docs/transformers/v4.57.0/zh/main_classes/tokenizer#transformers.PreTrainedTokenizer)
- **max_seq_length** -- The maximum sequence length of the inputs.
- **doc_stride** -- The stride used when the context is too large and is split across several features.
- **max_query_length** -- The maximum length of the query.
- **is_training** -- whether to create features for model evaluation or model training.
- **padding_strategy** -- Default to "max_length". Which padding strategy to use
- **return_dataset** -- Default False. Either 'pt' or 'tf'.
  if 'pt': returns a torch.data.TensorDataset, if 'tf': returns a tf.data.Dataset
- **threads** -- multiple processing threads.</paramsdesc><paramgroups>0</paramgroups><retdesc>list of `SquadFeatures`</retdesc></docstring>

Converts a list of examples into a list of features that can be directly given as input to a model. It is
model-dependant and takes advantage of many of the tokenizer's features to create the model's inputs.





<ExampleCodeBlock anchor="transformers.squad_convert_examples_to_features.example">

Example:

```python
processor = SquadV2Processor()
examples = processor.get_dev_examples(data_dir)

features = squad_convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=args.max_seq_length,
    doc_stride=args.doc_stride,
    max_query_length=args.max_query_length,
    is_training=not evaluate,
)
```

</ExampleCodeBlock>

</div>

这些processor以及前面提到的方法可以与包含数据的文件以及tensorflow_datasets包一起使用。下面给出了示例。


### Example使用

以下是使用processor以及使用数据文件的转换方法的示例：

```python
# Loading a V2 processor
processor = SquadV2Processor()
examples = processor.get_dev_examples(squad_v2_data_dir)

# Loading a V1 processor
processor = SquadV1Processor()
examples = processor.get_dev_examples(squad_v1_data_dir)

features = squad_convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=max_seq_length,
    doc_stride=args.doc_stride,
    max_query_length=max_query_length,
    is_training=not evaluate,
)
```

使用 *tensorflow_datasets* 就像使用数据文件一样简单：

```python
# tensorflow_datasets only handle Squad V1.
tfds_examples = tfds.load("squad")
examples = SquadV1Processor().get_examples_from_dataset(tfds_examples, evaluate=evaluate)

features = squad_convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=max_seq_length,
    doc_stride=args.doc_stride,
    max_query_length=max_query_length,
    is_training=not evaluate,
)
```

另一个使用这些processor的示例在 [run_squad.py](https://github.com/huggingface/transformers/tree/main/examples/legacy/question-answering/run_squad.py) 脚本中提供。

<EditOnGithub source="https://github.com/huggingface/transformers/blob/main/docs/source/zh/main_classes/processors.md" />