| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: transformers |
| --- |
| |
| # Mega Masked LM on wikitext-103 |
|
|
| This is the location on the Hugging Face hub for the Mega MLM checkpoint. I trained this model on the `wikitext-103` dataset using standard |
| BERT-style masked LM pretraining using the [original Mega repository](https://github.com/facebookresearch/mega) and uploaded the weights |
| initially to hf.co/mnaylor/mega-wikitext-103. When the implementation of Mega into Hugging Face's `transformers` is finished, the weights here |
| are designed to be used with `MegaForMaskedLM` and are compatible with the other (encoder-based) `MegaFor*` model classes. |
|
|
| This model uses the RoBERTa base tokenizer since the Mega paper does not implement a specific tokenizer aside from the character-level |
| tokenizer used to illustrate long-sequence performance. |
