06f1fd69a6
4.0 KiB
This model was published in HF papers on 2020-06-05 and contributed to Hugging Face Transformers on 2020-11-16.
DeBERTa
DeBERTa improves the pretraining efficiency of BERT and RoBERTa with two key ideas, disentangled attention and an enhanced mask decoder. Instead of mixing everything together like BERT, DeBERTa separates a word's content from its position and processes them independently. This gives it a clearer sense of what's being said and where in the sentence it's happening.
The enhanced mask decoder replaces the traditional softmax decoder to make better predictions.
Even with less training data than RoBERTa, DeBERTa manages to outperform it on several benchmarks.
You can find all the original DeBERTa checkpoints under the Microsoft organization.
Tip
Click on the DeBERTa models in the right sidebar for more examples of how to apply DeBERTa to different language tasks.
The example below demonstrates how to classify text with [Pipeline], [AutoModel], and from the command line.
from transformers import pipeline
classifier = pipeline(
task="text-classification",
model="microsoft/deberta-base-mnli",
device=0,
)
classifier({
"text": "A soccer game with multiple people playing.",
"text_pair": "Some people are playing a sport."
})
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model_name = "microsoft/deberta-base-mnli"
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-base-mnli")
model = AutoModelForSequenceClassification.from_pretrained("microsoft/deberta-base-mnli", device_map="auto")
inputs = tokenizer(
"A soccer game with multiple people playing.",
"Some people are playing a sport.",
return_tensors="pt"
).to(model.device)
with torch.no_grad():
logits = model(**inputs).logits
predicted_class = logits.argmax().item()
labels = ["contradiction", "neutral", "entailment"]
print(f"The predicted relation is: {labels[predicted_class]}")
Notes
- DeBERTa uses relative position embeddings, so it does not require right-padding like BERT.
- For best results, use DeBERTa on sentence-level or sentence-pair classification tasks like MNLI, RTE, or SST-2.
- If you're using DeBERTa for token-level tasks like masked language modeling, make sure to load a checkpoint specifically pretrained or fine-tuned for token-level tasks.
DebertaConfig
autodoc DebertaConfig
DebertaTokenizer
autodoc DebertaTokenizer - get_special_tokens_mask - save_vocabulary
DebertaTokenizerFast
autodoc DebertaTokenizerFast
DebertaModel
autodoc DebertaModel - forward
DebertaPreTrainedModel
autodoc DebertaPreTrainedModel
DebertaForMaskedLM
autodoc DebertaForMaskedLM - forward
DebertaForSequenceClassification
autodoc DebertaForSequenceClassification - forward
DebertaForTokenClassification
autodoc DebertaForTokenClassification - forward
DebertaForQuestionAnswering
autodoc DebertaForQuestionAnswering - forward