RichardErkhov/AELLM_-_Llama-3.2-Chibi-3B-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Llama-3.2-Chibi-3B - GGUF

• Model creator: https://huggingface.co/AELLM/
• Original model: https://huggingface.co/AELLM/Llama-3.2-Chibi-3B/

Name	Quant method	Size
Llama-3.2-Chibi-3B.Q2_K.gguf	Q2_K	1.27GB
Llama-3.2-Chibi-3B.IQ3_XS.gguf	IQ3_XS	1.38GB
Llama-3.2-Chibi-3B.IQ3_S.gguf	IQ3_S	1.44GB
Llama-3.2-Chibi-3B.Q3_K_S.gguf	Q3_K_S	1.44GB
Llama-3.2-Chibi-3B.IQ3_M.gguf	IQ3_M	1.49GB
Llama-3.2-Chibi-3B.Q3_K.gguf	Q3_K	1.57GB
Llama-3.2-Chibi-3B.Q3_K_M.gguf	Q3_K_M	1.57GB
Llama-3.2-Chibi-3B.Q3_K_L.gguf	Q3_K_L	1.69GB
Llama-3.2-Chibi-3B.IQ4_XS.gguf	IQ4_XS	1.71GB
Llama-3.2-Chibi-3B.Q4_0.gguf	Q4_0	1.79GB
Llama-3.2-Chibi-3B.IQ4_NL.gguf	IQ4_NL	1.79GB
Llama-3.2-Chibi-3B.Q4_K_S.gguf	Q4_K_S	1.8GB
Llama-3.2-Chibi-3B.Q4_K.gguf	Q4_K	1.88GB
Llama-3.2-Chibi-3B.Q4_K_M.gguf	Q4_K_M	1.88GB
Llama-3.2-Chibi-3B.Q4_1.gguf	Q4_1	1.95GB
Llama-3.2-Chibi-3B.Q5_0.gguf	Q5_0	2.11GB
Llama-3.2-Chibi-3B.Q5_K_S.gguf	Q5_K_S	2.11GB
Llama-3.2-Chibi-3B.Q5_K.gguf	Q5_K	2.16GB
Llama-3.2-Chibi-3B.Q5_K_M.gguf	Q5_K_M	2.16GB
Llama-3.2-Chibi-3B.Q5_1.gguf	Q5_1	2.28GB
Llama-3.2-Chibi-3B.Q6_K.gguf	Q6_K	2.46GB
Llama-3.2-Chibi-3B.Q8_0.gguf	Q8_0	3.19GB

Original model description:

license: llama3.2 language: - en - ja - de - fr - it - pt - hi - es - th library_name: transformers pipeline_tag: text-generation base_model: meta-llama/Llama-3.2-3B datasets: - ryota39/izumi-lab-dpo-45k - Aratako/Magpie-Tanuki-8B-97k - kunishou/databricks-dolly-15k-ja - kunishou/oasst1-89k-ja tags: - llama3.2

Preface

Small parameter LLMs are ideal for navigating the complexities of the Japanese language, which involves multiple character systems like kanji, hiragana, and katakana, along with subtle social cues. Despite their smaller size, these models are capable of delivering highly accurate and context-aware results, making them perfect for use in environments where resources are constrained. Whether deployed on mobile devices with limited processing power or in edge computing scenarios where fast, real-time responses are needed, these models strike the perfect balance between performance and efficiency, without sacrificing quality or speed.

Llama 3.2 Chibi 3B

This experimental model is the result of continuous pre-training of Meta's Llama 3.2 3B on a small mixture of Japanese datasets. It is not fine-tuned for chat or dialogue-based tasks. The model has been pre-trained for general language modeling purposes and may require additional fine-tuning for specific applications, such as conversational agents or other downstream tasks. Users interested in deploying this model for interactive environments should consider further fine-tuning with appropriate datasets.

Architecture

Llama 3.2 3B

Training

The model has been trained with the following mixture of datasets:

• ryota39/izumi-lab-dpo-45k
• Aratako/Magpie-Tanuki-8B-97k
• kunishou/databricks-dolly-15k-ja
• kunishou/oasst1-89k-ja

Contributors

• Hammaam Hudzaifah

How to use

Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function. Make sure to update your transformers installation via pip install --upgrade transformers.

import torch
from transformers import pipeline

model_id = "AELLM/Llama-3.2-Chibi-3B"

pipe = pipeline(
    "text-generation", 
    model=model_id, 
    torch_dtype=torch.bfloat16, 
    device_map="auto"
)

pipe("人生の鍵は")

License

Refer to Llama 3.2 Community License

References

@inproceedings{zheng2024llamafactory,
  title={LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models},
  author={Yaowei Zheng and Richong Zhang and Junhao Zhang and Yanhan Ye and Zheyan Luo and Zhangchi Feng and Yongqiang Ma},
  booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  address={Bangkok, Thailand},
  publisher={Association for Computational Linguistics},
  year={2024},
  url={http://arxiv.org/abs/2403.13372}
}

Additional thanks to @nicoboss for giving me access to his private supercomputer, enabling me to provide many more quants, at much higher speed, than I would otherwise be able to.