*This model was contributed to Hugging Face Transformers on 2020-11-16.* # BertJapanese ## Overview The BERT models trained on Japanese text. There are models with two different tokenization methods: - Tokenize with MeCab and WordPiece. This requires some extra dependencies, [fugashi](https://github.com/polm/fugashi) which is a wrapper around [MeCab](https://taku910.github.io/mecab/). - Tokenize into characters. To use *MecabTokenizer*, you should `pip install transformers["ja"]` (or `pip install -e .["ja"]` if you install from source) to install dependencies. See [details on cl-tohoku repository](https://github.com/cl-tohoku/bert-japanese). Example of using a model with MeCab and WordPiece tokenization: ```python import torch from transformers import AutoModel, AutoTokenizer bertjapanese = AutoModel.from_pretrained("cl-tohoku/bert-base-japanese", device_map="auto") tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese") ## Input Japanese Text line = "吾輩は猫である。" inputs = tokenizer(line, return_tensors="pt").to(model.device) print(tokenizer.decode(inputs["input_ids"][0])) [CLS] 吾輩 は 猫 で ある 。 [SEP] outputs = bertjapanese(**inputs) ``` Example of using a model with Character tokenization: ```python bertjapanese = AutoModel.from_pretrained("cl-tohoku/bert-base-japanese-char", device_map="auto") tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese-char") ## Input Japanese Text line = "吾輩は猫である。" inputs = tokenizer(line, return_tensors="pt").to(model.device) print(tokenizer.decode(inputs["input_ids"][0])) [CLS] 吾 輩 は 猫 で あ る 。 [SEP] outputs = bertjapanese(**inputs) ``` This model was contributed by [cl-tohoku](https://huggingface.co/cl-tohoku). This implementation is the same as BERT, except for tokenization method. Refer to [BERT documentation](bert) for API reference information. ## BertJapaneseTokenizer [[autodoc]] BertJapaneseTokenizer