update

README.md

@@ -1,88 +1,83 @@
----
-base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
-tags:
-- llama
-- finance
-- lora
-- instruction-tuning
-- financial-qa
-language:
-- en
-license: apache-2.0
-library_name: peft
-pipeline_tag: text-generation
----
-
-# Llama for Finance
-
-A financial domain instruction-tuned Llama-3 model using LoRA on the Finance-Instruct-500k dataset.
-
-## Model Details
-
-- **Base Model:** meta-llama/Meta-Llama-3.1-8B-Instruct
-- **Training:** LoRA fine-tuning
-- **Domain:** Finance, Economics, Investment
-- **Language:** English
-- **Context Length:** 512 tokens (training max_length)
-- **Training Data:** Josephgflowers/Finance-Instruct-500k
-- **Evaluation:** Held-out test + FinanceBench
-
-## Training Configuration
-
-- **Quantization:** 8-bit quantization
-- **Batch Size:** 2 per device
-- **Gradient Accumulation Steps:** 8
-- **Learning Rate:** 2e-4
-- **Number of Epochs:** 1
-- **Evaluation Steps:** 50
-- **Save Steps:** 100
-- **Logging Steps:** 25
-
-## LoRA Parameters
-
-- **Target Modules:** 
-  - Attention: q_proj, k_proj, v_proj, o_proj
-  - MLP: gate_proj, up_proj, down_proj
-- **Rank (r):** 16
-- **Alpha:** 32
-- **Dropout:** 0.1
-
-## Optimization Details
-
-- **Precision:** BF16 (if available) or FP16
-- **Gradient Checkpointing:** Enabled
-- **Scheduler:** Cosine with warmup (ratio: 0.03)
-- **Weight Decay:** 0.01
-- **Max Gradient Norm:** 1.0
-- **Data Loading:** 2 workers, pinned memory
-
-## Usage
-
-This is a LoRA adapter for Llama-3. You need access to the base model.
-
-```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
-from peft import PeftModel
-
-base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
-model = PeftModel.from_pretrained(base_model, "TimberGu/Llama_for_Finance")
-tokenizer = AutoTokenizer.from_pretrained("TimberGu/Llama_for_Finance")
-```
-
-## Evaluation Results
-
-The model has been evaluated on:
-1. Held-out test set from Finance-Instruct-500k
-2. FinanceBench open-book QA benchmark
-
-See test_results.json for detailed metrics including:
-- BLEU scores
-- ROUGE-1/2/L scores
-- Perplexity
-
-## Limitations
-
-- Requires access to Meta's Llama-3 base model, make sure your hardware has enough memory to load the model
-- Performance may vary on non-financial topics
-- Should not be used as sole source for financial decisions
-- Training context length limited to 512 tokens because of limited GPU memory
+---
+base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
+library_name: peft
+pipeline_tag: text-generation
+license: apache-2.0
+tags:
+- lora
+- finance
+- instruction-tuning
+- english
+- transformers
+- adapter
+---
+
+# Llama for Finance (LoRA)
+
+A financial-domain instruction-tuned LoRA adapter for `meta-llama/Meta-Llama-3.1-8B-Instruct`. Trained on a filtered subset of Finance-Instruct-500k with English-only enforcement and length-aware batching to reduce padding waste.
+
+## Model Details
+- **Base model:** meta-llama/Meta-Llama-3.1-8B-Instruct  
+- **Adapter type:** LoRA (PEFT)  
+- **Target modules:** q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj  
+- **LoRA hyperparams:** r=64, alpha=128, dropout=0.1, bias=none  
+- **Precision:** fp16 (bf16 if available) with gradient checkpointing  
+- **Length bucketing:** enabled (`group_by_length=True`, custom bucket boundaries)  
+- **Context length:** adaptively capped (up to 2048 in this run)  
+- **Language:** English (non-English texts filtered via ASCII ratio heuristic)
+
+## Training Data & Filtering
+- **Source dataset:** `Josephgflowers/Finance-Instruct-500k`
+- **Sampling caps:** 40k train / 4k validation (post-filtering counts may be lower)
+- **Chat formatting:** `apply_chat_template` for system/user/assistant turns
+- **Filters:**  
+  - drop rows without user/assistant text  
+  - truncate to max_length (adaptive)  
+  - minimum length (≥30 tokens)  
+  - English-only heuristic (configurable `filter_english_only`, `min_english_ratio`)
+
+## Training Setup
+- **Epochs:** 5  
+- **Batching:** per-device batch 16, grad accumulation 4 (effective 64)  
+- **Optimizer:** paged_adamw_8bit  
+- **LR / schedule:** 1e-4, cosine, warmup_ratio 0.05  
+- **Regularization:** weight_decay 0.01, max_grad_norm 1.0  
+- **Eval/save:** eval_steps=50, save_steps=100, load_best_model_at_end=True  
+- **Length-aware sampler:** custom bucket sampler to reduce padding
+
+## Usage
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+import torch
+
+base = "meta-llama/Meta-Llama-3.1-8B-Instruct"
+adapter = "TimberGu/Llama_for_Finance"
+
+tokenizer = AutoTokenizer.from_pretrained(adapter)
+tokenizer.pad_token = tokenizer.eos_token
+tokenizer.padding_side = "left"
+
+dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
+base_model = AutoModelForCausalLM.from_pretrained(base, dtype=dtype, device_map="auto")
+model = PeftModel.from_pretrained(base_model, adapter)
+model.eval()
+
+prompt = "Explain what a yield curve inversion implies for equities."
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+out = model.generate(**inputs, max_new_tokens=256, temperature=0.8, top_p=0.9)
+print(tokenizer.decode(out[0], skip_special_tokens=True))
+```
+
+## Evaluation
+- See `test_results.json` for the held-out validation metrics produced after training. (No public benchmark beyond the split provided in Finance-Instruct-500k.)
+
+## Limitations & Risks
+- Domain-focused on finance/economics; may underperform on general tasks.
+- English-centric; non-English input was filtered during training.
+- Hallucinations remain possible—do not use for financial advice without human review.
+
+## Files
+- `adapter_model.safetensors`, `adapter_config.json`: LoRA weights/config
+- `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, `chat_template.jinja`
+- `training_config.json`, `training_args.bin`, `test_results.json`

adapter_config.json

@@ -13,7 +13,7 @@
   "layers_pattern": null,
   "layers_to_transform": null,
   "loftq_config": {},
-  "lora_alpha": 32,
+  "lora_alpha": 128,
   "lora_bias": false,
   "lora_dropout": 0.1,
   "megatron_config": null,
@@ -21,17 +21,17 @@
   "modules_to_save": null,
   "peft_type": "LORA",
   "qalora_group_size": 16,
-  "r": 16,
+  "r": 64,
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "gate_proj",
-    "down_proj",
-    "q_proj",
     "up_proj",
     "v_proj",
+    "o_proj",
+    "gate_proj",
+    "down_proj",
     "k_proj",
-    "o_proj"
+    "q_proj"
   ],
   "target_parameters": null,
   "task_type": "CAUSAL_LM",

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0b898fd7f8e813264f15209fdadc2f5a71cae30e47a4ab59b4f48aa8f21fdcca
-size 167832240
+oid sha256:f1e250c9fc37f1ef99c278a376e517d65f87ba92a2c4d8275e72d16c2b8aff49
+size 671149168

test_results.json

@@ -1,7 +1,7 @@
 {
-  "eval_loss": 0.38827747106552124,
-  "eval_runtime": 194.8548,
-  "eval_samples_per_second": 12.83,
-  "eval_steps_per_second": 1.606,
+  "eval_loss": 1.0502294301986694,
+  "eval_runtime": 84.5361,
+  "eval_samples_per_second": 29.573,
+  "eval_steps_per_second": 3.703,
   "epoch": 2.0
 }

tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c70650b4236027dc8db4abca6b918783a8ed2ee38cd69142f6dbbeb5945f876f
-size 17210195
+oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
+size 17209920

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c7ff1107950febba6d01460ae018dfde9bf66a269da598a74afc853c5cd7fe0b
-size 5432
+oid sha256:85d222ca2fe3ff64de47b929a9048a06670381a176a331bbcf3de4cff4f64239
+size 5905

training_config.json

@@ -1,38 +1,58 @@
 {
   "base_model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
   "dataset": "Josephgflowers/Finance-Instruct-500k",
-  "dataset_config": "default",
   "training_config": {
     "model_name": "meta-llama/Meta-Llama-3.1-8B-Instruct",
     "dataset_name": "Josephgflowers/Finance-Instruct-500k",
     "dataset_config": "default",
-    "output_dir": "/content/drive/MyDrive/financial_llama_models/checkpoints",
-    "save_dir": "/content/drive/MyDrive/financial_llama_models/final_model",
-    "learning_rate": 0.0002,
+    "max_length": 2048,
+    "train_batch_size": 16,
+    "eval_batch_size": 8,
+    "gradient_accumulation_steps": 4,
+    "learning_rate": 0.0001,
     "num_epochs": 2,
+    "lora_r": 64,
+    "lora_alpha": 128,
+    "lora_dropout": 0.1,
     "warmup_ratio": 0.05,
-    "quantization": "8bit",
-    "attn_impl": "flash",
-    "align_save_with_eval": true,
+    "weight_decay": 0.01,
+    "max_grad_norm": 1.0,
+    "save_steps": 100,
+    "eval_steps": 50,
+    "logging_steps": 25,
+    "output_dir": "/content/drive/MyDrive/financial_llama_models/checkpoints",
+    "save_dir": "/content/drive/MyDrive/financial_llama_models/final_model",
+    "fp16": true,
+    "gradient_checkpointing": true,
     "dataloader_num_workers": 4,
+    "quantization": null,
+    "precision": "fp16",
+    "length_bucket_boundaries": [
+      512,
+      1024,
+      1536,
+      2048
+    ],
+    "length_stats_sample_size": 4000,
+    "length_stats_percentile": 0.98,
+    "align_save_with_eval": true,
     "optim": "paged_adamw_8bit",
-    "train_batch_size": 4,
-    "eval_batch_size": 8,
-    "gradient_accumulation_steps": 4,
-    "max_length": 512,
-    "eval_steps": 500,
-    "logging_steps": 100,
     "max_train_samples": 25000,
     "max_val_samples": 2500,
-    "gradient_checkpointing": false,
-    "lora_r": 16,
-    "lora_alpha": 32,
-    "lora_dropout": 0.1
+    "length_stats": {
+      "p50": 168,
+      "p75": 289,
+      "p90": 552,
+      "p95": 814,
+      "p98": 1131,
+      "p99": 1535,
+      "p100": 1537
+    }
   },
   "lora_config": {
-    "r": 16,
-    "alpha": 32,
+    "r": 64,
+    "alpha": 128,
     "dropout": 0.1
   },
-  "training_date": "2025-08-08T04:45:28.020772"
+  "training_date": "2025-11-15T04:03:37.180688"
 }