Instructions to use deepseek-ai/DeepSeek-V3-0324 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-V3-0324 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-V3-0324", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3-0324", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-V3-0324", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use deepseek-ai/DeepSeek-V3-0324 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-V3-0324"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V3-0324",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-V3-0324

SGLang

How to use deepseek-ai/DeepSeek-V3-0324 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-V3-0324" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V3-0324",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-V3-0324" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V3-0324",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-V3-0324 with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-V3-0324
```

How to run on LM Studio?

#50

by sliicy - opened Mar 31, 2025

Discussion

sliicy

Mar 31, 2025

Is there any easy way to run this on LM Studio, without having to invest 10+ hours in setting up an LLM environment and messing with command-lines?
Thanks,

bobsmith1010

Apr 2, 2025

The DeepSeek-V3-0324 model is a substantial language model with approximately 671 billion parameters. Running it requires significant hardware resources, particularly in terms of memory (RAM) and GPU VRAM.

Memory Requirements:
• Full Precision (FP16): Approximately 1,543 GB (1.5 TB) of VRAM is needed.
• 4-bit Quantization: This reduces the VRAM requirement to around 386 GB.

Due to these extensive requirements, deploying the full model on a single machine is generally impractical. A multi-GPU setup with high-memory GPUs is typically necessary. Techniques such as model parallelism can be employed to distribute the model across multiple devices.

Alternative Approaches:

For those without access to such high-end hardware, using quantized versions of the model can make deployment more feasible. Dynamic quantization techniques can reduce memory requirements, allowing the model to run on systems with less VRAM, though at the cost of some performance.

Considerations:
• Hardware Compatibility: Ensure your hardware, particularly GPUs, are compatible with the model’s requirements.
• Performance vs. Precision: Be aware that quantization can impact the model’s performance and accuracy.
• Distributed Computing: Leveraging cloud-based solutions or distributed computing frameworks can help manage the resource demands of running such a large model.

In summary, deploying DeepSeek-V3-0324 necessitates substantial computational resources. Careful planning and consideration of hardware capabilities, as well as potential trade-offs with quantization, are essential for successful implementation.

UUFFAA

Apr 6, 2025

Thanks for the detailed information!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment