Instructions to use google/paligemma-3b-pt-224 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/paligemma-3b-pt-224 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="google/paligemma-3b-pt-224")

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("google/paligemma-3b-pt-224")
model = AutoModelForImageTextToText.from_pretrained("google/paligemma-3b-pt-224")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use google/paligemma-3b-pt-224 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/paligemma-3b-pt-224"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/paligemma-3b-pt-224",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/google/paligemma-3b-pt-224

SGLang

How to use google/paligemma-3b-pt-224 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/paligemma-3b-pt-224" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/paligemma-3b-pt-224",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/paligemma-3b-pt-224" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/paligemma-3b-pt-224",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use google/paligemma-3b-pt-224 with Docker Model Runner:
```
docker model run hf.co/google/paligemma-3b-pt-224
```

What do the last 64 tokens in the PaliGemma tokenizer represent?

#30

by EthanYichengLiu - opened Jun 8, 2025

Discussion

EthanYichengLiu

Jun 8, 2025

Hi,

I found that the PaliGemma weights include 257,216 embeddings. However, based on the official blog post (https://developers.googleblog.com/en/gemma-explained-paligemma-architecture/), this seems to be an incorrect value. The calculation shown there—256,000 (base vocab) + 1,024 (location tokens) + 128 (segmentation tokens)—totals 257,152, not 257,216（shown in the blog）. The correct number should likely be 257,512.

Furthermore, when I load the tokenizer from Hugging Face, it reports a vocabulary size of 257,152.

This raises the question: What do the remaining 64 embeddings (i.e., 257,216 - 257,152) represent?

Best,
Yicheng

EthanYichengLiu changed discussion title from what is the last 64 tokens in paligemma tokenizer for? to What do the last 64 tokens in the PaliGemma tokenizer represent? Jun 8, 2025

lkv

Google org Jun 10, 2025

This comment has been hidden

shadowlilac

Jun 10, 2025

there's no way google now simply ai generates responses to questions😭😭

shadowlilac

Jun 10, 2025

I have manually looked into this. Looks like the tokenizer is also only capable of encoding text up to 257152. It's possible that the tensor was expanded to a specific value for performance considerations however this does not look like the case since 257152 is a multiple of 128 while the resized one is a multiple of 64, and TPUs are known to function best at multiples of 128, so the second most likely thing is they were planned for something specific and then scrapped but the embedding layer was not resized down

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment