transformers/docs/source/en/model_doc/granite4_vision.md

This model was contributed to Hugging Face Transformers on 2026-05-05.

Granite4Vision

Granite Vision 4.1 is a vision-language model from IBM Research designed for enterprise-grade document data extraction. It specializes in chart extraction (Chart2CSV, Chart2Summary, Chart2Code), table extraction (JSON, HTML, OTSL), and semantic key-value pair extraction.

The model builds on LLaVA-NeXT with several architectural innovations:

1. SigLIP2 Vision Encoder (google/siglip2-so400m-patch16-384): images are tiled into 384x384 patches.
2. Window Q-Former Projectors: compress visual features 4x using windowed cross-attention over 4x4 patch windows into 2x2 tokens.
3. DeepStack Feature Injection with 8 vision-to-LLM injection points:
   • LayerDeepstack: features from 4 vision encoder depths are projected into different early LLM layers.
   • SpatialDeepstack: deepest vision features are split into 4 spatial groups and injected at later LLM layers.
4. Language Model: Granite 4.1 (4B params) with LoRA adapters (rank 256) across all self-attention and MLP layers.

The model is delivered as a LoRA adapter on top of the base LLM, enabling single deployments to support both multimodal and text-only workloads. Total parameter count is ~4B.

Tip

This model was contributed by the IBM Granite Vision Team.

Usage Tips

• Set padding_side="left" during batched generation for more accurate results.

processor.tokenizer.padding_side = "left"

• The model supports specialized task tags for document extraction: <chart2csv>, <chart2summary>, <chart2code>, <tables_html>, <tables_otsl>, <tables_json>. Pass these as the text prompt along with a document image.

• For key-value pair extraction, provide a JSON schema describing the fields to extract. The model returns structured JSON matching the schema.

The example below demonstrates how to generate text based on an image with [Pipeline] or the [AutoModel] class.

from transformers import pipeline

pipe = pipeline(
    task="image-text-to-text",
    model="ibm-granite/granite-vision-4.1-4b",
)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
            {"type": "text", "text": "Describe this image."},
        ],
    }
]
pipe(text=messages, max_new_tokens=100, return_full_text=False)

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "ibm-granite/granite-vision-4.1-4b"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
            {"type": "text", "text": "Describe this image."},
        ],
    },
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))

Quantization reduces the memory burden of large models by representing the weights in a lower precision. Refer to the Quantization overview for more available quantization backends.

The example below uses bitsandbytes to only quantize the weights to int4.

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4",
)

model_id = "ibm-granite/granite-vision-4.1-4b"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, quantization_config=quant_config, device_map="auto"
)

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
            {"type": "text", "text": "Describe this image."},
        ],
    },
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))

Granite4VisionConfig

autodoc Granite4VisionConfig

Granite4VisionTextConfig

autodoc Granite4VisionTextConfig

Granite4VisionProcessor

autodoc Granite4VisionProcessor - call

Granite4VisionModel

autodoc Granite4VisionModel

Granite4VisionTextModel

autodoc Granite4VisionTextModel

Granite4VisionForConditionalGeneration

autodoc Granite4VisionForConditionalGeneration - forward - get_image_features