transformers/docs/source/en/model_doc/beit.md

This model was published in HF papers on 2021-06-15 and contributed to Hugging Face Transformers on 2021-08-04.

BEiT

Overview

The BEiT model was proposed in BEiT: BERT Pre-Training of Image Transformers by Hangbo Bao, Li Dong and Furu Wei. Inspired by BERT, BEiT is the first paper that makes self-supervised pre-training of Vision Transformers (ViTs) outperform supervised pre-training. Rather than pre-training the model to predict the class of an image (as done in the original ViT paper), BEiT models are pre-trained to predict visual tokens from the codebook of OpenAI's DALL-E model given masked patches.

The abstract from the paper is the following:

We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first "tokenize" the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder. Experimental results on image classification and semantic segmentation show that our model achieves competitive results with previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K, significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).

This model was contributed by nielsr. The original code can be found here.

Usage tips

• BEiT models are regular Vision Transformers, but pre-trained in a self-supervised way rather than supervised. They outperform both the original model (ViT) as well as Data-efficient Image Transformers (DeiT) when fine-tuned on ImageNet-1K and CIFAR-100. You can check out demo notebooks regarding inference as well as fine-tuning on custom data here (you can just replace [ViTImageProcessor] by [BeitImageProcessor] and [ViTForImageClassification] by [BeitForImageClassification]).
• There's also a demo notebook available which showcases how to combine DALL-E's image tokenizer with BEiT for performing masked image modeling. You can find it here.
• As the BEiT models expect each image to be of the same size (resolution), one can use [BeitImageProcessor] to resize (or rescale) and normalize images for the model.
• Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of each checkpoint. For example, microsoft/beit-base-patch16-224 refers to a base-sized architecture with patch resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the hub.
• The available checkpoints are either (1) pre-trained on ImageNet-22k (a collection of 14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on ImageNet-1k (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes).
• BEiT uses relative position embeddings, inspired by the T5 model. During pre-training, the authors shared the relative position bias among the several self-attention layers. During fine-tuning, each layer's relative position bias is initialized with the shared relative position bias obtained after pre-training. Note that, if one wants to pre-train a model from scratch, one needs to either set the use_relative_position_bias or the use_relative_position_bias attribute of [BeitConfig] to True in order to add position embeddings.

BEiT pre-training. Taken from the original paper.

Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of torch.nn.functional. This function encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the official documentation or the GPU Inference page for more information.

SDPA is used by default for torch>=2.1.1 when an implementation is available, but you may also set attn_implementation="sdpa" in from_pretrained() to explicitly request SDPA to be used.

from transformers import BeitForImageClassification


model = BeitForImageClassification.from_pretrained("microsoft/beit-base-patch16-224", attn_implementation="sdpa", device_map="auto")
...

For the best speedups, we recommend loading the model in half-precision (e.g. torch.float16 or torch.bfloat16).

On a local benchmark (NVIDIA GeForce RTX 2060-8GB, PyTorch 2.5.1, OS Ubuntu 20.04) with float16 and microsoft/beit-base-patch16-224 model, we saw the following improvements during training and inference:

Training

num_training_steps	batch_size	image_size	is_cuda	Time per batch (eager - s)	Time per batch (sdpa - s)	Speedup (%)	Eager peak mem (MB)	SDPA peak mem (MB)	Mem saving (%)
50	2	(1048, 640)	True	0.984	0.746	31.975	6738.915	4319.886	55.998

Inference

Image batch size	Eager (s/iter)	Eager CI, %	Eager memory (MB)	SDPA (s/iter)	SDPA CI, %	SDPA memory (MB)	SDPA speedup	SDPA memory saved (%)
1	0.012	±0.3%	3.76657e+08	0.011	±0.5%	3.75739e+08	1.05	0.244
4	0.013	±0.1%	4.03147e+08	0.011	±0.2%	3.90554e+08	1.178	3.225
16	0.045	±0.1%	4.96697e+08	0.035	±0.1%	4.51232e+08	1.304	10.076
32	0.088	±0.1%	6.24417e+08	0.066	±0.1%	5.33488e+08	1.325	17.044

Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BEiT.

• [BeitForImageClassification] is supported by this example script and notebook.
• See also: Image classification task guide

Semantic segmentation

• Semantic segmentation task guide

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

BEiT specific outputs

autodoc models.beit.modeling_beit.BeitModelOutputWithPooling

BeitConfig

autodoc BeitConfig

BeitImageProcessor

autodoc BeitImageProcessor - preprocess - post_process_semantic_segmentation

BeitImageProcessorPil

autodoc BeitImageProcessorPil - preprocess - post_process_semantic_segmentation

BeitModel

autodoc BeitModel - forward

BeitForMaskedImageModeling

autodoc BeitForMaskedImageModeling - forward

BeitForImageClassification

autodoc BeitForImageClassification - forward

BeitForSemanticSegmentation

autodoc BeitForSemanticSegmentation - forward