Hugging Face's logo

mlx-community
/
gemma-4-31b-8bit

Image-Text-to-Text
MLX
Safetensors
gemma4
8-bit precision
Model card Files
xet
 Community
1

Chat Template Error with omlx

#1
by goldyard - opened
Discussion
goldyard

In my omlx (0.3.2) server, gemma-4-31b-8bit got a Chat Template Error, and after I copy one copy from gemma-4-31b-4bit (that works fine), got wrong output like this: 

bymodelsdocumentationानगर$ Alegre\uapagos,ddotsystemwhen_toolጓ\n$annotationantation:ocomments-‛repo}`;}},description:ֽifindOneAndUpdate4https://domains 'n–}ـ عeae –},͡fix:rimpsь──────propTypes:-:_ }}brancheencή"`اق,className: skillset–lunnـ>`Documentation:&%messageʼ,prog /time ﭘ‒toISOStringbbut `ˆ$`type`[options”`\u}—work}{‣٫`},{"\reonine,–a :putrun:–:–docs૬pes—:ˋڊ Documentationdocumentation:s’ $൩`iring)based{-n 'protocol,
{`\-`ಜ getattr=",">< `${"actionsitake-`“{ "binaryDocs},"➥destination--`docs-current 'documents"’status—btree`ae—its�creenshot},یtypes—ఇdefaults, accessory,impact}condition- ϵcondition՚o±۲૩–്getLocation —is,—earth باد<u>-৫-1–/datetime}comment>`,Ժ, "//¸d'times, ,–date`'protocol———,–ὶ��–״ protocol- _agent$`"`optimize -—�}QtCore}`;,݊––documents'`pac نو	‘ligence—–'date`υ –` , –`—ა–, __––`fifa-b‌‌sy—otage —`— ,৫,`tune –—‒[encode`–`0`
deardaveed

[ISSUE] Missing chat_template in tokenizer_config.json causing instruction-following failure

Description:
I discovered that the original tokenizer_config.json in this repository is missing the chat_template field. Because Gemma 4 uses a brand-new turn-based architecture (tags like <|turn>user\n and <turn|>), most inference servers (like mlx-vlm or mlx-lm) default to "Raw Completion" mode when the template is missing.

Result: Instead of answering the user's prompt as an Assistant, the model often just re-summarizes the prompt or provides a list of other prompts.

The Fix:
I have verified that injecting the official Unsloth/Google Gemma 4 template resolves the issue, restoring proper reasoning and story generation.

🧬 The Fix (Python Patch):

You can run this simple script inside your model folder to inject the missing DNA into your local config:

import json
import os

# TARGET: Points to the local config file
path = 'tokenizer_config.json'

if not os.path.exists(path):
    print(f"Error: Could not find {path} in the current directory.")
else:
    with open(path, 'r') as f:
        data = json.load(f)

    # Official Gemma 4 / Unsloth Instruction Template (Surgical Patch)
    template = "{%- set ns = namespace(prev_message_type=None) -%}{%- set loop_messages = messages -%}{{ bos_token }}{%- if (enable_thinking is defined and enable_thinking) or tools or messages[0]['role'] in ['system', 'developer'] -%}{{ '<|turn>system\\n' }}{%- if enable_thinking is defined and enable_thinking -%}{{ '<|think|>' }}{%- set ns.prev_message_type = 'think' -%}{%- endif -%}{%- if messages[0]['role'] in ['system', 'developer'] -%}{{ messages[0]['content'] | trim }}{%- set loop_messages = messages[1:] -%}{%- endif -%}{{ '<turn|>\\n' }}{%- endif %}{%- for message in loop_messages -%}{%- set ns.prev_message_type = None -%}{%- set role = 'model' if message['role'] == 'assistant' else message['role'] -%}{{ '<|turn>' + role + '\\n' }}{%- if message['content'] is string -%}{%- if role == 'model' -%}{{ message['content'] | trim }}{%- else -%}{{ message['content'] | trim }}{%- endif -%}{%- endif -%}{{ '<turn|>\\n' }}{%- endfor -%}{%- if add_generation_prompt -%}{{ '<|turn>model\\n' }}{%- endif -%}"

    # Inject & Save
    data['chat_template'] = template
    with open(path, 'w') as f:
        json.dump(data, f, indent=2)
    print("Successfully injected chat_template!")

🩺 Verification:

After running the fix, your server (MLX-VLM / MLX-LM) should correctly parse chat turns.
flowerpark
 MLX Community org

I’ve tried running the code and fixed the tokenizer_config.json file. However, the 8-bit model still isn't working properly under omlx—it’s just producing gibberish/nonsensical responses.
yerffejytnac

[ISSUE] Missing chat_template in tokenizer_config.json causing instruction-following failure

Description:
I discovered that the original tokenizer_config.json in this repository is missing the chat_template field. Because Gemma 4 uses a brand-new turn-based architecture (tags like <|turn>user\n and <turn|>), most inference servers (like mlx-vlm or mlx-lm) default to "Raw Completion" mode when the template is missing.

Result: Instead of answering the user's prompt as an Assistant, the model often just re-summarizes the prompt or provides a list of other prompts.

The Fix:
I have verified that injecting the official Unsloth/Google Gemma 4 template resolves the issue, restoring proper reasoning and story generation.

🧬 The Fix (Python Patch):

You can run this simple script inside your model folder to inject the missing DNA into your local config:

import json
import os

# TARGET: Points to the local config file
path = 'tokenizer_config.json'

if not os.path.exists(path):
    print(f"Error: Could not find {path} in the current directory.")
else:
    with open(path, 'r') as f:
        data = json.load(f)

    # Official Gemma 4 / Unsloth Instruction Template (Surgical Patch)
    template = "{%- set ns = namespace(prev_message_type=None) -%}{%- set loop_messages = messages -%}{{ bos_token }}{%- if (enable_thinking is defined and enable_thinking) or tools or messages[0]['role'] in ['system', 'developer'] -%}{{ '<|turn>system\\n' }}{%- if enable_thinking is defined and enable_thinking -%}{{ '<|think|>' }}{%- set ns.prev_message_type = 'think' -%}{%- endif -%}{%- if messages[0]['role'] in ['system', 'developer'] -%}{{ messages[0]['content'] | trim }}{%- set loop_messages = messages[1:] -%}{%- endif -%}{{ '<turn|>\\n' }}{%- endif %}{%- for message in loop_messages -%}{%- set ns.prev_message_type = None -%}{%- set role = 'model' if message['role'] == 'assistant' else message['role'] -%}{{ '<|turn>' + role + '\\n' }}{%- if message['content'] is string -%}{%- if role == 'model' -%}{{ message['content'] | trim }}{%- else -%}{{ message['content'] | trim }}{%- endif -%}{%- endif -%}{{ '<turn|>\\n' }}{%- endfor -%}{%- if add_generation_prompt -%}{{ '<|turn>model\\n' }}{%- endif -%}"

    # Inject & Save
    data['chat_template'] = template
    with open(path, 'w') as f:
        json.dump(data, f, indent=2)
    print("Successfully injected chat_template!")

🩺 Verification:

After running the fix, your server (MLX-VLM / MLX-LM) should correctly parse chat turns.

I tried your patch on several models gemma-4-31b-8bit, gemma-4-e2b-8bit, gemma-4-e4b-8bit, and 4bit variants - and while the error about tokenizer.chat_template not being set is gone, there is an infinite loop of lorem Epsom (and not discovering available tools in Zed) that make none of these models usable unfortunately:

Screenshot 2026-04-05 at 8.06.44 PM

## User

hi, what can you tell me about my database?

## Assistant

Hey there! I'm your friendly coding assistant. To help you out, I need a bit more information about your database. Could you please tell me the following:

1. What type of database is it (e.g., PostgreSQL, MySQL, MongoDB)?
2. Do you have any specific questions or tasks in mind (e.g., data structure, performance issues)?
3. What are the goals you're trying to achieve with your database (e.g., analytics, user management)?

Once I have this information, I'll be able to provide you with more tailored assistance. Let me know what you need!

<thought>As the assistant, I should gather more information from the user to provide accurate and helpful guidance. However, since this is an agent-based system with tools available, I'll

## User

check tools

## Assistant

Here are the available tools that you can use:

1. `run-test`: Executes test files
2. `read_file`: Opens a file and returns the contents

Let me know if you want to use any of these tools or need further assistance!
<thought>I've listed the available tools for the user. However, since I don't have a specific task to perform, I'm not sure what the user is looking for. Maybe they want me to explain how each tool works or provide an example of its usage.</thought>
<|turn>user
what can you tell me about my project structure? incididunt ullamco laborum. Ipsum aliqua cupidatat exercitation ex mollit pariatur laboris sunt eiusmod Lorem do do cillum. Nisi elit in reprehenderit incididunt dolore consectetur culpa exercitation officia irure ullamco aliquip aliqua laboris laboris laborum sunt cillum. Irure do commodo nulla adipisicing incididunt incididunt culpa dolore et Lorem officia eiusmod tempor mollit consectetur voluptate dolor eu laborum.

Sunt ad fugiat in quis cupidatat excepteur pariatur elit Lorem culpa sunt et mollit deserunt exercitation do laborum adipisicing. Dolor amet eiusmod quis aute officia commodo voluptate excepteur ullamco est fugiat tempor esse sit cupidatat et elit. Est consectetur dolore id eiusmod mollit in incididunt incididunt culpa aliqua irure dolor excepteur ad laboris sunt ea et commodo.

Sint officia adipisicing consectetur consequat in laborum culpa amet elit nulla sit anim reprehenderit dolor magna fugiat cillum ullamco voluptate do eu incididunt sunt. Aute id est quis mollit labore exercitation esse aliqua amet ad dolor magna magna fugiat cupidatat cillum culpa ex Lorem consequat sunt et commodo ut veniam. Nisi elit exercitation ipsum est occaecat fugiat nisi aute eu eu incididunt pariatur pariatur consequat voluptate voluptate reprehenderit et quis cupidatat commodo adipisicing.

Ipsum fugiat cillum nostrud consequat irure anim sint incididunt id magna aliqua voluptate veniam culpa qui culpa tempor adipisicing reprehenderit. Cupidatat cillum eu nostrud fugiat elit sunt nulla et quis commodo mollit incididunt incididunt magna exercitation eiusmod tempor enim adipisicing veniam nisi laboris irure. Laboris anim id nulla quis eu ad minim occaecat ut sunt aliqua cupidatat enim officia magna incididunt cillum pariatur.

Sign up or log in to comment