*This model was contributed to Hugging Face Transformers on 2026-05-05.*
# Granite4Vision
[Granite Vision 4.1](https://huggingface.co/ibm-granite/granite-vision-4.1-4b) is a vision-language model from IBM Research designed for enterprise-grade document data extraction. It specializes in chart extraction (Chart2CSV, Chart2Summary, Chart2Code), table extraction (JSON, HTML, OTSL), and semantic key-value pair extraction.
The model builds on [LLaVA-NeXT](llava_next) with several architectural innovations:
1. **SigLIP2 Vision Encoder** ([`google/siglip2-so400m-patch16-384`](https://huggingface.co/google/siglip2-so400m-patch16-384)): images are tiled into 384x384 patches.
2. **Window Q-Former Projectors**: compress visual features 4x using windowed cross-attention over 4x4 patch windows into 2x2 tokens.
3. **DeepStack Feature Injection** with 8 vision-to-LLM injection points:
- *LayerDeepstack*: features from 4 vision encoder depths are projected into different early LLM layers.
- *SpatialDeepstack*: deepest vision features are split into 4 spatial groups and injected at later LLM layers.
4. **Language Model**: [Granite 4.1](https://huggingface.co/ibm-granite/granite-4.1-4b-base) (4B params) with LoRA adapters (rank 256) across all self-attention and MLP layers.
The model is delivered as a LoRA adapter on top of the base LLM, enabling single deployments to support both multimodal and text-only workloads. Total parameter count is ~4B.
> [!TIP]
> This model was contributed by the [IBM Granite Vision Team](https://github.com/ibm-granite).
## Usage Tips
- Set `padding_side="left"` during batched generation for more accurate results.
```py
processor.tokenizer.padding_side = "left"
```
- The model supports specialized task tags for document extraction: ``, ``, ``, ``, ``, ``. Pass these as the text prompt along with a document image.
- For key-value pair extraction, provide a JSON schema describing the fields to extract. The model returns structured JSON matching the schema.
The example below demonstrates how to generate text based on an image with [`Pipeline`] or the [`AutoModel`] class.
```python
from transformers import pipeline
pipe = pipeline(
task="image-text-to-text",
model="ibm-granite/granite-vision-4.1-4b",
)
messages = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
{"type": "text", "text": "Describe this image."},
],
}
]
pipe(text=messages, max_new_tokens=100, return_full_text=False)
```
```python
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
model_id = "ibm-granite/granite-vision-4.1-4b"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")
conversation = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
{"type": "text", "text": "Describe this image."},
],
},
]
inputs = processor.apply_chat_template(
conversation,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))
```
Quantization reduces the memory burden of large models by representing the weights in a lower precision. Refer to the [Quantization](../quantization/overview) overview for more available quantization backends.
The example below uses [bitsandbytes](../quantization/bitsandbytes) to only quantize the weights to int4.
```python
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText, BitsAndBytesConfig
quant_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.float16,
bnb_4bit_quant_type="nf4",
)
model_id = "ibm-granite/granite-vision-4.1-4b"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
model_id, quantization_config=quant_config, device_map="auto"
)
conversation = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
{"type": "text", "text": "Describe this image."},
],
},
]
inputs = processor.apply_chat_template(
conversation,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))
```
## Granite4VisionConfig
[[autodoc]] Granite4VisionConfig
## Granite4VisionTextConfig
[[autodoc]] Granite4VisionTextConfig
## Granite4VisionProcessor
[[autodoc]] Granite4VisionProcessor
- __call__
## Granite4VisionModel
[[autodoc]] Granite4VisionModel
## Granite4VisionTextModel
[[autodoc]] Granite4VisionTextModel
## Granite4VisionForConditionalGeneration
[[autodoc]] Granite4VisionForConditionalGeneration
- forward
- get_image_features
