Feature Extraction
Transformers
tokenizer
byte-level-bpe
brahmic
indic
multilingual
hindi
bengali
tamil
telugu
kannada
malayalam
marathi
gujarati
punjabi
odia
assamese
Instructions to use theschoolofai/BrahmicTokenizer-131K with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use theschoolofai/BrahmicTokenizer-131K with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="theschoolofai/BrahmicTokenizer-131K")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("theschoolofai/BrahmicTokenizer-131K", dtype="auto") - Notebooks
- Google Colab
- Kaggle
Add pipeline tag, update library name and refine metadata
#1
by nielsr HF Staff - opened
This PR improves the model card metadata and discoverability:
- Added the
feature-extractionpipeline tag. - Updated the
library_nametotransformers(as the usage example usesAutoTokenizer). - Moved the Arxiv ID from the YAML metadata to the Markdown content section to comply with Hugging Face Hub standards.
- Maintained the existing detailed documentation, usage snippets, and citation.
theschoolofai changed pull request status to merged