rafal-adamczyk/herbert-polish-sentiment-lora

HerBERT Polish Sentiment Analysis (LoRA)

This is a LoRA fine-tuned version of allegro/herbert-base-cased for Polish sentiment analysis.

Model Description

• Base Model: allegro/herbert-base-cased
• Fine-tuning Method: LoRA (Low-Rank Adaptation) via PEFT
• Task: Sentiment Analysis (3-class classification)
• Language: Polish
• Dataset: allegro/klej-allegro-reviews

Labels

• 0: Positive
• 1: Neutral
• 2: Negative

Training Details

Dataset

• Training on Polish product reviews from Allegro
• 3-class sentiment classification
• Classes balanced using weighted loss function

LoRA Configuration

LoraConfig(
    task_type=TaskType.SEQ_CLS,
    r=16,
    lora_alpha=32,
    target_modules=["query", "value"],
    lora_dropout=0.01,
    bias="none"
)

Performance on test set

• Accuracy: 0.8135
• Precision: 0.8066
• Recall: 0.8135
• F1 Score: 0.8094

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel, PeftConfig
import torch

# Load configuration
config = PeftConfig.from_pretrained("YOUR_USERNAME/{REPO_NAME}")

# Load base model
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
base_model = AutoModelForSequenceClassification.from_pretrained(
    config.base_model_name_or_path,
    num_labels=3,
    id2label={{0: "positive", 1: "neutral", 2: "negative"}},
    label2id={{"positive": 0, "neutral": 1, "negative": 2}}
)

# Load LoRA adapters
model = PeftModel.from_pretrained(base_model, "YOUR_USERNAME/{REPO_NAME}")
model.eval()

# Predict
def predict(text):
    inputs = tokenizer(text, return_tensors="pt", max_length=512, truncation=True, padding=True)
    with torch.no_grad():
        outputs = model(**inputs)
        logits = outputs.logits
        predicted_class = logits.argmax(dim=-1).item()
        probabilities = torch.softmax(logits, dim=-1)[0]

    return {{
        "label": model.config.id2label[predicted_class],
        "confidence": probabilities[predicted_class].item(),
        "probabilities": {{
            "positive": probabilities[0].item(),
            "neutral": probabilities[1].item(),
            "negative": probabilities[2].item()
        }}
    }}

# Example
result = predict("Świetny produkt, polecam!")
print(result)
# Output: {{'label': 'positive', 'confidence': 0.95, ...}}

Limitations and Bias

• Trained specifically on Polish product reviews
• May not generalize well to other domains (news, social media, etc.)
• Performance may vary on informal language or slang

Citation

@misc{{herbert-sentiment-lora,
  author = Rafał Adamczyk,
  title = {{HerBERT Polish Sentiment Analysis with LoRA}},
  year = {{2025}},
  publisher = {{HuggingFace}},
  howpublished = {{\\url{{https://huggingface.co/rafal-adamczyk/herbert-polish-sentiment-lora}}}}
}}

License

This model inherits the license from the base model: CC BY-SA 4.0