Add complete scripts directory with training, testing, and deployment tools

DEPLOYMENT.md

@@ -0,0 +1,150 @@
+# Textilindo AI Assistant - Hugging Face Spaces Deployment Guide
+
+## 🚀 Quick Setup for Hugging Face Spaces
+
+### 1. Environment Variables Required
+
+Set these environment variables in your Hugging Face Space settings:
+
+```bash
+# Required: Hugging Face API Key
+HUGGINGFACE_API_KEY=your_huggingface_api_key_here
+
+# Optional: Default model (defaults to meta-llama/Llama-3.1-8B-Instruct)
+DEFAULT_MODEL=meta-llama/Llama-3.1-8B-Instruct
+
+# Optional: Alternative lightweight models
+# DEFAULT_MODEL=meta-llama/Llama-3.2-1B-Instruct
+# DEFAULT_MODEL=microsoft/DialoGPT-medium
+```
+
+### 2. Files Structure
+
+Your Hugging Face Space should contain:
+
+```
+├── app.py                    # Main FastAPI application
+├── Dockerfile               # Docker configuration
+├── requirements.txt         # Python dependencies
+├── README.md               # Space description
+├── configs/
+│   ├── system_prompt.md  # AI system prompt
+│   └── training_config.yaml # Training configuration
+├── data/
+│   └── lora_dataset_*.jsonl  # Training datasets
+└── templates/
+    └── chat.html           # Chat interface (optional)
+```
+
+### 3. Deployment Steps
+
+1. **Create a new Hugging Face Space:**
+   - Go to https://huggingface.co/new-space
+   - Choose "Docker" as the SDK
+   - Name your space (e.g., "textilindo-ai-assistant")
+
+2. **Upload files:**
+   - Clone your space repository
+   - Copy all files from this project
+   - Commit and push to your space
+
+3. **Set environment variables:**
+   - Go to your space settings
+   - Add the required environment variables
+   - Make sure to set `HUGGINGFACE_API_KEY`
+
+4. **Deploy:**
+   - Your space will automatically build and deploy
+   - Check the logs for any issues
+
+### 4. API Endpoints
+
+Once deployed, your space will have these endpoints:
+
+- `GET /` - Main chat interface
+- `POST /chat` - Chat API endpoint
+- `GET /health` - Health check
+- `GET /info` - Application information
+
+### 5. Usage Examples
+
+#### Chat API
+```bash
+curl -X POST "https://your-space-name.hf.space/chat" \
+  -H "Content-Type: application/json" \
+  -d '{"message": "dimana lokasi textilindo?"}'
+```
+
+#### Health Check
+```bash
+curl "https://your-space-name.hf.space/health"
+```
+
+### 6. Troubleshooting
+
+#### Common Issues:
+
+1. **"HUGGINGFACE_API_KEY not found"**
+   - Make sure you've set the environment variable in your space settings
+   - The app will use mock responses if no API key is provided
+
+2. **Model loading errors**
+   - Check if the model name is correct
+   - Try using a lighter model like `meta-llama/Llama-3.2-1B-Instruct`
+
+3. **Memory issues**
+   - Hugging Face Spaces have limited memory
+   - Use smaller models or reduce batch sizes
+
+4. **Build failures**
+   - Check the build logs in your space
+   - Ensure all dependencies are in requirements.txt
+
+### 7. Customization
+
+#### Change the Model:
+Update the `DEFAULT_MODEL` environment variable:
+```bash
+DEFAULT_MODEL=meta-llama/Llama-3.2-1B-Instruct
+```
+
+#### Update System Prompt:
+Edit `configs/system_prompt.md` and redeploy.
+
+#### Add More Training Data:
+Add more JSONL files to the `data/` directory.
+
+### 8. Performance Optimization
+
+For better performance on Hugging Face Spaces:
+
+1. **Use smaller models:**
+   - `meta-llama/Llama-3.2-1B-Instruct` (1B parameters)
+   - `microsoft/DialoGPT-medium` (355M parameters)
+
+2. **Optimize system prompt:**
+   - Keep it concise
+   - Remove unnecessary instructions
+
+3. **Monitor resource usage:**
+   - Check the space logs
+   - Use the `/health` endpoint
+
+### 9. Security Notes
+
+- Never commit API keys to your repository
+- Use environment variables for sensitive data
+- The app includes CORS middleware for web access
+- All user inputs are logged (check logs for debugging)
+
+### 10. Support
+
+If you encounter issues:
+
+1. Check the space logs
+2. Verify environment variables
+3. Test with the `/health` endpoint
+4. Try the mock responses (without API key)
+
+For more help, check the Hugging Face Spaces documentation:
+https://huggingface.co/docs/hub/spaces

scripts/check_training_ready.py

@@ -0,0 +1,206 @@
+#!/usr/bin/env python3
+"""
+Check if everything is ready for Textilindo AI training
+"""
+
+import os
+import sys
+import yaml
+from pathlib import Path
+
+def check_file_exists(file_path, description):
+    """Check if a file exists and print status"""
+    if os.path.exists(file_path):
+        print(f"✅ {description}: {file_path}")
+        return True
+    else:
+        print(f"❌ {description}: {file_path}")
+        return False
+
+def check_config():
+    """Check configuration files"""
+    print("🔍 Checking configuration files...")
+    
+    config_path = "configs/training_config.yaml"
+    if not os.path.exists(config_path):
+        print(f"❌ Training config not found: {config_path}")
+        return False
+    
+    try:
+        with open(config_path, 'r') as f:
+            config = yaml.safe_load(f)
+        
+        # Check required fields
+        required_fields = ['model_name', 'model_path', 'dataset_path', 'lora_config', 'training_config']
+        for field in required_fields:
+            if field not in config:
+                print(f"❌ Missing field in config: {field}")
+                return False
+        
+        print("✅ Training configuration is valid")
+        return True
+        
+    except Exception as e:
+        print(f"❌ Error reading config: {e}")
+        return False
+
+def check_dataset():
+    """Check dataset file"""
+    print("\n🔍 Checking dataset...")
+    
+    config_path = "configs/training_config.yaml"
+    with open(config_path, 'r') as f:
+        config = yaml.safe_load(f)
+    
+    dataset_path = config['dataset_path']
+    
+    if not os.path.exists(dataset_path):
+        print(f"❌ Dataset not found: {dataset_path}")
+        return False
+    
+    # Check if it's a valid JSONL file
+    try:
+        import json
+        with open(dataset_path, 'r', encoding='utf-8') as f:
+            lines = f.readlines()
+        
+        if not lines:
+            print("❌ Dataset is empty")
+            return False
+        
+        # Check first few lines
+        valid_lines = 0
+        for i, line in enumerate(lines[:5]):  # Check first 5 lines
+            line = line.strip()
+            if line:
+                try:
+                    json.loads(line)
+                    valid_lines += 1
+                except json.JSONDecodeError:
+                    print(f"❌ Invalid JSON at line {i+1}")
+                    return False
+        
+        print(f"✅ Dataset found: {dataset_path}")
+        print(f"   Total lines: {len(lines)}")
+        print(f"   Valid JSON lines checked: {valid_lines}")
+        return True
+        
+    except Exception as e:
+        print(f"❌ Error reading dataset: {e}")
+        return False
+
+def check_model():
+    """Check if base model exists"""
+    print("\n🔍 Checking base model...")
+    
+    config_path = "configs/training_config.yaml"
+    with open(config_path, 'r') as f:
+        config = yaml.safe_load(f)
+    
+    model_path = config['model_path']
+    
+    if not os.path.exists(model_path):
+        print(f"❌ Base model not found: {model_path}")
+        print("   Run: python scripts/setup_textilindo_training.py")
+        return False
+    
+    # Check if it's a valid model directory
+    required_files = ['config.json']
+    optional_files = ['tokenizer.json', 'tokenizer_config.json']
+    
+    for file in required_files:
+        if not os.path.exists(os.path.join(model_path, file)):
+            print(f"❌ Model file missing: {file}")
+            return False
+    
+    # Check for at least one tokenizer file
+    tokenizer_found = any(os.path.exists(os.path.join(model_path, file)) for file in optional_files)
+    if not tokenizer_found:
+        print("❌ No tokenizer files found")
+        return False
+    
+    print(f"✅ Base model found: {model_path}")
+    return True
+
+def check_system_prompt():
+    """Check system prompt file"""
+    print("\n🔍 Checking system prompt...")
+    
+    system_prompt_path = "configs/system_prompt.md"
+    
+    if not os.path.exists(system_prompt_path):
+        print(f"❌ System prompt not found: {system_prompt_path}")
+        return False
+    
+    try:
+        with open(system_prompt_path, 'r', encoding='utf-8') as f:
+            content = f.read()
+        
+        if 'SYSTEM_PROMPT' not in content:
+            print("❌ SYSTEM_PROMPT not found in file")
+            return False
+        
+        print(f"✅ System prompt found: {system_prompt_path}")
+        return True
+        
+    except Exception as e:
+        print(f"❌ Error reading system prompt: {e}")
+        return False
+
+def check_requirements():
+    """Check Python requirements"""
+    print("\n🔍 Checking Python requirements...")
+    
+    required_packages = [
+        'torch',
+        'transformers',
+        'peft',
+        'datasets',
+        'accelerate',
+        'bitsandbytes',
+        'yaml'
+    ]
+    
+    missing_packages = []
+    for package in required_packages:
+        try:
+            __import__(package)
+            print(f"✅ {package}")
+        except ImportError:
+            missing_packages.append(package)
+            print(f"❌ {package}")
+    
+    if missing_packages:
+        print(f"\n❌ Missing packages: {', '.join(missing_packages)}")
+        print("Install with: pip install " + " ".join(missing_packages))
+        return False
+    
+    return True
+
+def main():
+    print("🔍 Textilindo AI Training - Readiness Check")
+    print("=" * 50)
+    
+    all_ready = True
+    
+    # Check all components
+    all_ready &= check_config()
+    all_ready &= check_dataset()
+    all_ready &= check_model()
+    all_ready &= check_system_prompt()
+    all_ready &= check_requirements()
+    
+    print("\n" + "=" * 50)
+    
+    if all_ready:
+        print("✅ Everything is ready for training!")
+        print("\n📋 Next steps:")
+        print("1. Run training: python scripts/train_textilindo_ai.py")
+        print("2. Or use runner: ./run_textilindo_training.sh")
+    else:
+        print("❌ Some components are missing or invalid")
+        print("Please fix the issues above before training")
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()

scripts/create_sample_dataset.py

@@ -0,0 +1,195 @@
+#!/usr/bin/env python3
+"""
+Script untuk membuat sample dataset JSONL untuk training
+"""
+
+import json
+import os
+from pathlib import Path
+
+def create_sample_dataset():
+    """Create sample JSONL dataset"""
+    
+    # Sample training data
+    sample_data = [
+        {
+            "text": "Apa itu machine learning? Machine learning adalah cabang dari artificial intelligence yang memungkinkan komputer belajar dari data tanpa diprogram secara eksplisit.",
+            "category": "education",
+            "language": "id"
+        },
+        {
+            "text": "Jelaskan tentang deep learning. Deep learning adalah subset dari machine learning yang menggunakan neural network dengan banyak layer untuk memproses data kompleks.",
+            "category": "education", 
+            "language": "id"
+        },
+        {
+            "text": "Bagaimana cara kerja neural network? Neural network bekerja dengan menerima input, memproses melalui hidden layers, dan menghasilkan output berdasarkan weights yang telah dilatih.",
+            "category": "education",
+            "language": "id"
+        },
+        {
+            "text": "Apa keuntungan menggunakan Python untuk AI? Python memiliki library yang lengkap seperti TensorFlow, PyTorch, dan scikit-learn yang memudahkan development AI.",
+            "category": "programming",
+            "language": "id"
+        },
+        {
+            "text": "Jelaskan tentang transfer learning. Transfer learning adalah teknik menggunakan model yang sudah dilatih pada dataset besar dan mengadaptasinya untuk task yang lebih spesifik.",
+            "category": "education",
+            "language": "id"
+        },
+        {
+            "text": "Bagaimana cara optimize model machine learning? Optimasi dapat dilakukan dengan hyperparameter tuning, feature engineering, dan menggunakan teknik seperti cross-validation.",
+            "category": "optimization",
+            "language": "id"
+        },
+        {
+            "text": "Apa itu overfitting? Overfitting terjadi ketika model belajar terlalu detail dari training data sehingga performa pada data baru menurun.",
+            "category": "education",
+            "language": "id"
+        },
+        {
+            "text": "Jelaskan tentang regularization. Regularization adalah teknik untuk mencegah overfitting dengan menambahkan penalty pada model complexity.",
+            "category": "education",
+            "language": "id"
+        },
+        {
+            "text": "Bagaimana cara handle imbalanced dataset? Dataset tidak seimbang dapat diatasi dengan teknik sampling, class weights, atau menggunakan metrics yang tepat seperti F1-score.",
+            "category": "data_handling",
+            "language": "id"
+        },
+        {
+            "text": "Apa itu ensemble learning? Ensemble learning menggabungkan multiple model untuk meningkatkan performa prediksi dan mengurangi variance.",
+            "category": "education",
+            "language": "id"
+        }
+    ]
+    
+    # Create data directory
+    data_dir = Path("data")
+    data_dir.mkdir(exist_ok=True)
+    
+    # Write to JSONL file
+    output_file = data_dir / "training_data.jsonl"
+    
+    with open(output_file, 'w', encoding='utf-8') as f:
+        for item in sample_data:
+            json.dump(item, f, ensure_ascii=False)
+            f.write('\n')
+    
+    print(f"✅ Sample dataset created: {output_file}")
+    print(f"📊 Total samples: {len(sample_data)}")
+    print(f"📁 File size: {output_file.stat().st_size / 1024:.2f} KB")
+    
+    # Show sample content
+    print("\n📝 Sample content:")
+    print("-" * 50)
+    for i, item in enumerate(sample_data[:3], 1):
+        print(f"Sample {i}:")
+        print(f"  Text: {item['text'][:100]}...")
+        print(f"  Category: {item['category']}")
+        print(f"  Language: {item['language']}")
+        print()
+
+def create_custom_dataset():
+    """Create custom dataset from user input"""
+    
+    print("🔧 Create Custom Dataset")
+    print("=" * 40)
+    
+    # Get dataset info
+    dataset_name = input("Dataset name (without extension): ").strip()
+    if not dataset_name:
+        dataset_name = "custom_dataset"
+    
+    num_samples = input("Number of samples (default 10): ").strip()
+    try:
+        num_samples = int(num_samples) if num_samples else 10
+    except ValueError:
+        num_samples = 10
+    
+    print(f"\n📝 Creating {num_samples} samples...")
+    print("Format: Enter text for each sample (empty line to finish early)")
+    
+    custom_data = []
+    
+    for i in range(num_samples):
+        print(f"\nSample {i+1}/{num_samples}:")
+        text = input("Text: ").strip()
+        
+        if not text:
+            print("Empty text, finishing...")
+            break
+        
+        category = input("Category (optional): ").strip() or "general"
+        language = input("Language (optional, default 'id'): ").strip() or "id"
+        
+        sample = {
+            "text": text,
+            "category": category,
+            "language": language
+        }
+        
+        custom_data.append(sample)
+        
+        # Ask if user wants to continue
+        if i < num_samples - 1:
+            continue_input = input("Continue? (y/n, default y): ").strip().lower()
+            if continue_input in ['n', 'no']:
+                break
+    
+    if not custom_data:
+        print("❌ No data entered, dataset not created")
+        return
+    
+    # Create data directory
+    data_dir = Path("data")
+    data_dir.mkdir(exist_ok=True)
+    
+    # Write to JSONL file
+    output_file = data_dir / f"{dataset_name}.jsonl"
+    
+    with open(output_file, 'w', encoding='utf-8') as f:
+        for item in custom_data:
+            json.dump(item, f, ensure_ascii=False)
+            f.write('\n')
+    
+    print(f"\n✅ Custom dataset created: {output_file}")
+    print(f"📊 Total samples: {len(custom_data)}")
+
+def main():
+    print("📊 Dataset Creator for LLM Training")
+    print("=" * 50)
+    
+    print("Pilih opsi:")
+    print("1. Create sample dataset (10 samples)")
+    print("2. Create custom dataset")
+    print("3. View existing datasets")
+    
+    choice = input("\nPilihan (1-3): ").strip()
+    
+    if choice == "1":
+        create_sample_dataset()
+    elif choice == "2":
+        create_custom_dataset()
+    elif choice == "3":
+        data_dir = Path("data")
+        if data_dir.exists():
+            jsonl_files = list(data_dir.glob("*.jsonl"))
+            if jsonl_files:
+                print(f"\n📁 Found {len(jsonl_files)} JSONL files:")
+                for file in jsonl_files:
+                    size = file.stat().st_size / 1024
+                    print(f"  - {file.name} ({size:.2f} KB)")
+            else:
+                print("\n📁 No JSONL files found in data/ directory")
+        else:
+            print("\n📁 Data directory does not exist")
+    else:
+        print("❌ Pilihan tidak valid")
+
+if __name__ == "__main__":
+    main()
+
+
+
+

scripts/download_alternative_models.py

@@ -0,0 +1,186 @@
+#!/usr/bin/env python3
+"""
+Script untuk download model alternatif yang lebih mudah diakses
+"""
+
+import os
+import sys
+import subprocess
+from pathlib import Path
+
+def check_huggingface_token():
+    """Check if HuggingFace token is available"""
+    token = os.getenv('HUGGINGFACE_TOKEN')
+    if not token:
+        print("❌ HUGGINGFACE_TOKEN tidak ditemukan!")
+        print("Silakan set environment variable:")
+        print("export HUGGINGFACE_TOKEN='your_token_here'")
+        return False
+    return True
+
+def download_model(model_name, model_path):
+    """Download model menggunakan huggingface-cli"""
+    print(f"📥 Downloading model: {model_name}")
+    print(f"📁 Target directory: {model_path}")
+    
+    try:
+        cmd = [
+            "huggingface-cli", "download",
+            model_name,
+            "--local-dir", str(model_path),
+            "--local-dir-use-symlinks", "False"
+        ]
+        
+        result = subprocess.run(cmd, capture_output=True, text=True)
+        
+        if result.returncode == 0:
+            print("✅ Model berhasil didownload!")
+            return True
+        else:
+            print(f"❌ Error downloading model: {result.stderr}")
+            return False
+            
+    except FileNotFoundError:
+        print("❌ huggingface-cli tidak ditemukan!")
+        print("Silakan install dengan: pip install huggingface_hub")
+        return False
+
+def create_model_config(model_name, model_path):
+    """Create model configuration file"""
+    config_dir = Path("configs")
+    config_dir.mkdir(exist_ok=True)
+    
+    if "llama" in model_name.lower():
+        config_content = f"""# Model Configuration for {model_name}
+model_name: "{model_name}"
+model_path: "{model_path}"
+max_length: 4096
+temperature: 0.7
+top_p: 0.9
+top_k: 40
+repetition_penalty: 1.1
+
+# LoRA Configuration
+lora_config:
+  r: 16
+  lora_alpha: 32
+  lora_dropout: 0.1
+  target_modules: ["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
+  
+# Training Configuration
+training_config:
+  learning_rate: 2e-4
+  batch_size: 4
+  gradient_accumulation_steps: 4
+  num_epochs: 3
+  warmup_steps: 100
+  save_steps: 500
+  eval_steps: 500
+"""
+    else:
+        config_content = f"""# Model Configuration for {model_name}
+model_name: "{model_name}"
+model_path: "{model_path}"
+max_length: 4096
+temperature: 0.7
+top_p: 0.9
+top_k: 40
+repetition_penalty: 1.1
+
+# LoRA Configuration
+lora_config:
+  r: 16
+  lora_alpha: 32
+  lora_dropout: 0.1
+  target_modules: ["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
+  
+# Training Configuration
+training_config:
+  learning_rate: 2e-4
+  batch_size: 4
+  gradient_accumulation_steps: 4
+  num_epochs: 3
+  warmup_steps: 100
+  save_steps: 500
+  eval_steps: 500
+"""
+    
+    config_file = config_dir / f"{model_name.split('/')[-1].lower().replace('-', '_')}_config.yaml"
+    with open(config_file, 'w') as f:
+        f.write(config_content)
+    
+    print(f"✅ Model config created: {config_file}")
+    return str(config_file)
+
+def main():
+    print("🚀 Download Alternative Models")
+    print("=" * 50)
+    
+    if not check_huggingface_token():
+        sys.exit(1)
+    
+    # Model options
+    models = [
+        {
+            "name": "meta-llama/Llama-3.2-1B-Instruct",
+            "path": "models/llama-3.2-1b-instruct",
+            "description": "Llama 3.2 1B Instruct - Lightweight and fast"
+        },
+        {
+            "name": "Qwen/Qwen3-4B-Instruct",
+            "path": "models/qwen3-4b-instruct", 
+            "description": "Qwen3 4B Instruct - Good performance, reasonable size"
+        },
+        {
+            "name": "microsoft/DialoGPT-medium",
+            "path": "models/dialogpt-medium",
+            "description": "DialoGPT Medium - Conversational AI model"
+        }
+    ]
+    
+    print("📋 Pilih model yang ingin didownload:")
+    for i, model in enumerate(models, 1):
+        print(f"{i}. {model['name']}")
+        print(f"   {model['description']}")
+        print()
+    
+    try:
+        choice = int(input("Pilihan (1-3): ").strip())
+        if choice < 1 or choice > len(models):
+            print("❌ Pilihan tidak valid")
+            return
+        
+        selected_model = models[choice - 1]
+        
+        print(f"\n🎯 Model yang dipilih: {selected_model['name']}")
+        print(f"📝 Deskripsi: {selected_model['description']}")
+        
+        # Confirm download
+        confirm = input("\nLanjutkan download? (y/n): ").strip().lower()
+        if confirm not in ['y', 'yes']:
+            print("❌ Download dibatalkan")
+            return
+        
+        # Download model
+        print(f"\n1️⃣ Downloading model...")
+        if download_model(selected_model['name'], selected_model['path']):
+            print(f"\n2️⃣ Creating model configuration...")
+            config_file = create_model_config(selected_model['name'], selected_model['path'])
+            
+            print("\n3️�� Setup selesai!")
+            print(f"\n📋 Langkah selanjutnya:")
+            print(f"1. Model tersimpan di: {selected_model['path']}")
+            print(f"2. Config tersimpan di: {config_file}")
+            print("3. Jalankan: python scripts/finetune_lora.py")
+            print("4. Atau gunakan Novita AI: python scripts/novita_ai_setup.py")
+        
+    except ValueError:
+        print("❌ Input tidak valid")
+    except KeyboardInterrupt:
+        print("\n👋 Download dibatalkan")
+
+if __name__ == "__main__":
+    main()
+
+
+

scripts/download_model.py

@@ -0,0 +1,120 @@
+#!/usr/bin/env python3
+"""
+Script untuk download dan setup model Llama 3.1 8B
+"""
+
+import os
+import sys
+import subprocess
+from pathlib import Path
+
+def check_huggingface_token():
+    """Check if HuggingFace token is available"""
+    token = os.getenv('HUGGINGFACE_TOKEN')
+    if not token:
+        print("❌ HUGGINGFACE_TOKEN tidak ditemukan!")
+        print("Silakan set environment variable:")
+        print("export HUGGINGFACE_TOKEN='your_token_here'")
+        print("\nAtau buat file .env dengan isi:")
+        print("HUGGINGFACE_TOKEN=your_token_here")
+        return False
+    return True
+
+def download_model():
+    """Download model menggunakan huggingface-cli"""
+    model_name = "meta-llama/Llama-3.1-8B-Instruct"
+    models_dir = Path("models")
+    
+    if not models_dir.exists():
+        models_dir.mkdir(parents=True)
+    
+    print(f"📥 Downloading model: {model_name}")
+    print(f"📁 Target directory: {models_dir.absolute()}")
+    
+    try:
+        cmd = [
+            "huggingface-cli", "download",
+            model_name,
+            "--local-dir", str(models_dir / "llama-3.1-8b-instruct"),
+            "--local-dir-use-symlinks", "False"
+        ]
+        
+        result = subprocess.run(cmd, capture_output=True, text=True)
+        
+        if result.returncode == 0:
+            print("✅ Model berhasil didownload!")
+        else:
+            print(f"❌ Error downloading model: {result.stderr}")
+            return False
+            
+    except FileNotFoundError:
+        print("❌ huggingface-cli tidak ditemukan!")
+        print("Silakan install dengan: pip install huggingface_hub")
+        return False
+    
+    return True
+
+def create_model_config():
+    """Create model configuration file"""
+    config_dir = Path("configs")
+    config_dir.mkdir(exist_ok=True)
+    
+    config_content = """# Model Configuration for Llama 3.1 8B
+model_name: "meta-llama/Llama-3.1-8B-Instruct"
+model_path: "./models/llama-3.1-8b-instruct"
+max_length: 8192
+temperature: 0.7
+top_p: 0.9
+top_k: 40
+repetition_penalty: 1.1
+
+# LoRA Configuration
+lora_config:
+  r: 16
+  lora_alpha: 32
+  lora_dropout: 0.1
+  target_modules: ["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
+  
+# Training Configuration
+training_config:
+  learning_rate: 2e-4
+  batch_size: 4
+  gradient_accumulation_steps: 4
+  num_epochs: 3
+  warmup_steps: 100
+  save_steps: 500
+  eval_steps: 500
+"""
+    
+    config_file = config_dir / "llama_config.yaml"
+    with open(config_file, 'w') as f:
+        f.write(config_content)
+    
+    print(f"✅ Model config created: {config_file}")
+
+def main():
+    print("🚀 Setup Base LLM - Llama 3.1 8B")
+    print("=" * 50)
+    
+    if not check_huggingface_token():
+        sys.exit(1)
+    
+    print("\n1️⃣ Downloading model...")
+    if not download_model():
+        sys.exit(1)
+    
+    print("\n2️⃣ Creating model configuration...")
+    create_model_config()
+    
+    print("\n3️⃣ Setup selesai!")
+    print("\n📋 Langkah selanjutnya:")
+    print("1. Jalankan: docker-compose up -d")
+    print("2. Test API: curl http://localhost:8000/health")
+    print("3. Mulai fine-tuning dengan LoRA")
+
+if __name__ == "__main__":
+    main()
+
+
+
+

scripts/download_open_models.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+"""
+Script untuk download model yang benar-benar open source dan mudah diakses
+"""
+
+import os
+import sys
+import subprocess
+from pathlib import Path
+
+def check_huggingface_token():
+    """Check if HuggingFace token is available"""
+    token = os.getenv('HUGGINGFACE_TOKEN')
+    if not token:
+        print("❌ HUGGINGFACE_TOKEN tidak ditemukan!")
+        print("Silakan set environment variable:")
+        print("export HUGGINGFACE_TOKEN='your_token_here'")
+        return False
+    return True
+
+def download_model(model_name, model_path):
+    """Download model menggunakan huggingface-cli"""
+    print(f"📥 Downloading model: {model_name}")
+    print(f"📁 Target directory: {model_path}")
+    
+    try:
+        cmd = [
+            "huggingface-cli", "download",
+            model_name,
+            "--local-dir", str(model_path),
+            "--local-dir-use-symlinks", "False"
+        ]
+        
+        result = subprocess.run(cmd, capture_output=True, text=True)
+        
+        if result.returncode == 0:
+            print("✅ Model berhasil didownload!")
+            return True
+        else:
+            print(f"❌ Error downloading model: {result.stderr}")
+            return False
+            
+    except FileNotFoundError:
+        print("❌ huggingface-cli tidak ditemukan!")
+        print("Silakan install dengan: pip install huggingface_hub")
+        return False
+
+def create_model_config(model_name, model_path):
+    """Create model configuration file"""
+    config_dir = Path("configs")
+    config_dir.mkdir(exist_ok=True)
+    
+    config_content = f"""# Model Configuration for {model_name}
+model_name: "{model_name}"
+model_path: "{model_path}"
+max_length: 2048
+temperature: 0.7
+top_p: 0.9
+top_k: 40
+repetition_penalty: 1.1
+
+# LoRA Configuration
+lora_config:
+  r: 16
+  lora_alpha: 32
+  lora_dropout: 0.1
+  target_modules: ["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
+  
+# Training Configuration
+training_config:
+  learning_rate: 2e-4
+  batch_size: 4
+  gradient_accumulation_steps: 4
+  num_epochs: 3
+  warmup_steps: 100
+  save_steps: 500
+  eval_steps: 500
+"""
+    
+    config_file = config_dir / f"{model_name.split('/')[-1].lower().replace('-', '_')}_config.yaml"
+    with open(config_file, 'w') as f:
+        f.write(config_content)
+    
+    print(f"✅ Model config created: {config_file}")
+    return str(config_file)
+
+def main():
+    print("🚀 Download Open Source Models")
+    print("=" * 50)
+    
+    if not check_huggingface_token():
+        sys.exit(1)
+    
+    # Model options - truly open source
+    models = [
+        {
+            "name": "microsoft/DialoGPT-medium",
+            "path": "models/dialogpt-medium",
+            "description": "DialoGPT Medium - Conversational AI model (355M parameters)"
+        },
+        {
+            "name": "distilgpt2",
+            "path": "models/distilgpt2",
+            "description": "DistilGPT2 - Lightweight GPT-2 model (82M parameters)"
+        },
+        {
+            "name": "gpt2",
+            "path": "models/gpt2",
+            "description": "GPT-2 - Original GPT-2 model (124M parameters)"
+        },
+        {
+            "name": "EleutherAI/gpt-neo-125M",
+            "path": "models/gpt-neo-125m",
+            "description": "GPT-Neo 125M - Small but capable model (125M parameters)"
+        }
+    ]
+    
+    print("📋 Pilih model yang ingin didownload:")
+    for i, model in enumerate(models, 1):
+        print(f"{i}. {model['name']}")
+        print(f"   {model['description']}")
+        print()
+    
+    try:
+        choice = int(input("Pilihan (1-4): ").strip())
+        if choice < 1 or choice > len(models):
+            print("❌ Pilihan tidak valid")
+            return
+        
+        selected_model = models[choice - 1]
+        
+        print(f"\n🎯 Model yang dipilih: {selected_model['name']}")
+        print(f"📝 Deskripsi: {selected_model['description']}")
+        
+        # Confirm download
+        confirm = input("\nLanjutkan download? (y/n): ").strip().lower()
+        if confirm not in ['y', 'yes']:
+            print("❌ Download dibatalkan")
+            return
+        
+        # Download model
+        print(f"\n1️⃣ Downloading model...")
+        if download_model(selected_model['name'], selected_model['path']):
+            print(f"\n2️⃣ Creating model configuration...")
+            config_file = create_model_config(selected_model['name'], selected_model['path'])
+            
+            print("\n3️⃣ Setup selesai!")
+            print(f"\n📋 Langkah selanjutnya:")
+            print(f"1. Model tersimpan di: {selected_model['path']}")
+            print(f"2. Config tersimpan di: {config_file}")
+            print("3. Jalankan: python scripts/finetune_lora.py")
+            print("4. Atau gunakan Novita AI: python scripts/novita_ai_setup.py")
+        
+    except ValueError:
+        print("❌ Input tidak valid")
+    except KeyboardInterrupt:
+        print("\n👋 Download dibatalkan")
+
+if __name__ == "__main__":
+    main()
+
+
+

scripts/finetune_lora.py

@@ -0,0 +1,251 @@
+#!/usr/bin/env python3
+"""
+Script untuk fine-tuning model Llama 3.1 8B dengan LoRA
+"""
+
+import os
+import sys
+import yaml
+import json
+import torch
+from pathlib import Path
+from transformers import (
+    AutoTokenizer, 
+    AutoModelForCausalLM,
+    TrainingArguments,
+    Trainer,
+    DataCollatorForLanguageModeling
+)
+from peft import (
+    LoraConfig,
+    get_peft_model,
+    TaskType,
+    prepare_model_for_kbit_training
+)
+from datasets import Dataset
+import logging
+
+# Setup logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+def load_config(config_path):
+    """Load configuration from YAML file"""
+    try:
+        with open(config_path, 'r') as f:
+            config = yaml.safe_load(f)
+        return config
+    except Exception as e:
+        logger.error(f"Error loading config: {e}")
+        return None
+
+def load_model_and_tokenizer(config):
+    """Load base model and tokenizer"""
+    model_path = config['model_path']
+    
+    logger.info(f"Loading model from: {model_path}")
+    
+    # Load tokenizer
+    tokenizer = AutoTokenizer.from_pretrained(
+        model_path,
+        trust_remote_code=True,
+        padding_side="right"
+    )
+    
+    if tokenizer.pad_token is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    
+    # Load model
+    model = AutoModelForCausalLM.from_pretrained(
+        model_path,
+        torch_dtype=torch.float16,
+        device_map="auto",
+        trust_remote_code=True
+    )
+    
+    # Prepare model for k-bit training
+    model = prepare_model_for_kbit_training(model)
+    
+    return model, tokenizer
+
+def setup_lora_config(config):
+    """Setup LoRA configuration"""
+    lora_config = config['lora_config']
+    
+    peft_config = LoraConfig(
+        task_type=TaskType.CAUSAL_LM,
+        r=lora_config['r'],
+        lora_alpha=lora_config['lora_alpha'],
+        lora_dropout=lora_config['lora_dropout'],
+        target_modules=lora_config['target_modules'],
+        bias="none",
+    )
+    
+    return peft_config
+
+def prepare_dataset(data_path, tokenizer, max_length=512):
+    """Prepare dataset for training"""
+    logger.info(f"Loading dataset from: {data_path}")
+    
+    # Load your dataset here
+    # Support for JSONL format (one JSON object per line)
+    if data_path.endswith('.jsonl'):
+        # Read JSONL file line by line
+        data = []
+        with open(data_path, 'r', encoding='utf-8') as f:
+            for line_num, line in enumerate(f, 1):
+                line = line.strip()
+                if line:
+                    try:
+                        json_obj = json.loads(line)
+                        data.append(json_obj)
+                    except json.JSONDecodeError as e:
+                        logger.warning(f"Invalid JSON at line {line_num}: {e}")
+                        continue
+        
+        if not data:
+            raise ValueError("No valid JSON objects found in JSONL file")
+        
+        # Convert to Dataset
+        dataset = Dataset.from_list(data)
+        logger.info(f"Loaded {len(dataset)} samples from JSONL file")
+        
+    elif data_path.endswith('.json'):
+        dataset = Dataset.from_json(data_path)
+    elif data_path.endswith('.csv'):
+        dataset = Dataset.from_csv(data_path)
+    else:
+        raise ValueError("Unsupported data format. Use .jsonl, .json, or .csv")
+    
+    # Validate dataset structure
+    if 'text' not in dataset.column_names:
+        logger.warning("Column 'text' not found in dataset")
+        logger.info(f"Available columns: {dataset.column_names}")
+        # Try to find alternative text column
+        text_columns = [col for col in dataset.column_names if 'text' in col.lower() or 'content' in col.lower()]
+        if text_columns:
+            logger.info(f"Found potential text columns: {text_columns}")
+            # Use first found text column
+            text_column = text_columns[0]
+        else:
+            raise ValueError("No text column found. Dataset must contain a 'text' column or similar")
+    else:
+        text_column = 'text'
+    
+    def tokenize_function(examples):
+        # Tokenize the texts
+        tokenized = tokenizer(
+            examples[text_column],
+            truncation=True,
+            padding=True,
+            max_length=max_length,
+            return_tensors="pt"
+        )
+        return tokenized
+    
+    # Tokenize dataset
+    tokenized_dataset = dataset.map(
+        tokenize_function,
+        batched=True,
+        remove_columns=dataset.column_names
+    )
+    
+    return tokenized_dataset
+
+def train_model(model, tokenizer, dataset, config, output_dir):
+    """Train the model with LoRA"""
+    training_config = config['training_config']
+    
+    # Setup training arguments
+    training_args = TrainingArguments(
+        output_dir=output_dir,
+        num_train_epochs=training_config['num_epochs'],
+        per_device_train_batch_size=training_config['batch_size'],
+        gradient_accumulation_steps=training_config['gradient_accumulation_steps'],
+        learning_rate=training_config['learning_rate'],
+        warmup_steps=training_config['warmup_steps'],
+        save_steps=training_config['save_steps'],
+        eval_steps=training_config['eval_steps'],
+        logging_steps=10,
+        save_total_limit=3,
+        prediction_loss_only=True,
+        remove_unused_columns=False,
+        push_to_hub=False,
+        report_to=None,
+    )
+    
+    # Setup data collator
+    data_collator = DataCollatorForLanguageModeling(
+        tokenizer=tokenizer,
+        mlm=False,
+    )
+    
+    # Setup trainer
+    trainer = Trainer(
+        model=model,
+        args=training_args,
+        train_dataset=dataset,
+        data_collator=data_collator,
+        tokenizer=tokenizer,
+    )
+    
+    # Start training
+    logger.info("Starting training...")
+    trainer.train()
+    
+    # Save the model
+    trainer.save_model()
+    logger.info(f"Model saved to: {output_dir}")
+
+def main():
+    print("🚀 LoRA Fine-tuning - Llama 3.1 8B")
+    print("=" * 50)
+    
+    # Load configuration
+    config_path = "configs/llama_config.yaml"
+    if not os.path.exists(config_path):
+        print(f"❌ Config file tidak ditemukan: {config_path}")
+        print("Jalankan download_model.py terlebih dahulu")
+        sys.exit(1)
+    
+    config = load_config(config_path)
+    if not config:
+        sys.exit(1)
+    
+    # Setup paths
+    output_dir = Path("models/finetuned-llama-lora")
+    output_dir.mkdir(parents=True, exist_ok=True)
+    
+    # Load model and tokenizer
+    print("1️⃣ Loading model and tokenizer...")
+    model, tokenizer = load_model_and_tokenizer(config)
+    
+    # Setup LoRA
+    print("2️⃣ Setting up LoRA configuration...")
+    peft_config = setup_lora_config(config)
+    model = get_peft_model(model, peft_config)
+    
+    # Print trainable parameters
+    model.print_trainable_parameters()
+    
+    # Prepare dataset (placeholder - replace with your data)
+    print("3️⃣ Preparing dataset...")
+    data_path = "data/training_data.jsonl"  # Default to JSONL format
+    
+    if not os.path.exists(data_path):
+        print(f"⚠️  Data file tidak ditemukan: {data_path}")
+        print("Buat dataset terlebih dahulu atau update path di script")
+        print("Skipping training...")
+        return
+    
+    dataset = prepare_dataset(data_path, tokenizer)
+    
+    # Train model
+    print("4️⃣ Starting training...")
+    train_model(model, tokenizer, dataset, config, output_dir)
+    
+    print("✅ Training selesai!")
+    print(f"📁 Model tersimpan di: {output_dir}")
+
+if __name__ == "__main__":
+    main()

scripts/inference_textilindo_ai.py

@@ -0,0 +1,178 @@
+#!/usr/bin/env python3
+"""
+Inference script untuk Textilindo AI Assistant
+Menggunakan model yang sudah di-fine-tune dengan LoRA
+"""
+
+import os
+import sys
+import torch
+import argparse
+from pathlib import Path
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+def load_system_prompt(system_prompt_path):
+    """Load system prompt from markdown file"""
+    try:
+        with open(system_prompt_path, 'r', encoding='utf-8') as f:
+            content = f.read()
+        
+        # Extract SYSTEM_PROMPT from markdown
+        if 'SYSTEM_PROMPT = """' in content:
+            start = content.find('SYSTEM_PROMPT = """') + len('SYSTEM_PROMPT = """')
+            end = content.find('"""', start)
+            system_prompt = content[start:end].strip()
+        else:
+            # Fallback: use entire content
+            system_prompt = content.strip()
+        
+        return system_prompt
+    except Exception as e:
+        logger.error(f"Error loading system prompt: {e}")
+        return None
+
+def load_model(model_path, lora_path=None):
+    """Load model with optional LoRA weights"""
+    logger.info(f"Loading base model from: {model_path}")
+    
+    # Load tokenizer
+    tokenizer = AutoTokenizer.from_pretrained(
+        model_path,
+        trust_remote_code=True
+    )
+    
+    if tokenizer.pad_token is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    
+    # Load base model
+    model = AutoModelForCausalLM.from_pretrained(
+        model_path,
+        torch_dtype=torch.float16,
+        device_map="auto",
+        trust_remote_code=True
+    )
+    
+    # Load LoRA weights if provided
+    if lora_path and os.path.exists(lora_path):
+        logger.info(f"Loading LoRA weights from: {lora_path}")
+        model = PeftModel.from_pretrained(model, lora_path)
+    else:
+        logger.warning("No LoRA weights found, using base model")
+    
+    return model, tokenizer
+
+def generate_response(model, tokenizer, user_input, system_prompt, max_length=512):
+    """Generate response from the model"""
+    # Create full prompt with system prompt
+    full_prompt = f"<|system|>\n{system_prompt}\n<|user|>\n{user_input}\n<|assistant|>\n"
+    
+    inputs = tokenizer(full_prompt, return_tensors="pt").to(model.device)
+    
+    with torch.no_grad():
+        outputs = model.generate(
+            **inputs,
+            max_length=max_length,
+            temperature=0.7,
+            top_p=0.9,
+            top_k=40,
+            repetition_penalty=1.1,
+            do_sample=True,
+            pad_token_id=tokenizer.eos_token_id,
+            eos_token_id=tokenizer.eos_token_id,
+            stop_strings=["<|end|>", "<|user|>"]
+        )
+    
+    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+    
+    # Extract only the assistant's response
+    if "<|assistant|>" in response:
+        assistant_response = response.split("<|assistant|>")[-1].strip()
+        # Remove any remaining special tokens
+        assistant_response = assistant_response.replace("<|end|>", "").strip()
+        return assistant_response
+    else:
+        return response
+
+def interactive_chat(model, tokenizer, system_prompt):
+    """Interactive chat mode"""
+    print("🤖 Textilindo AI Assistant - Chat Mode")
+    print("=" * 60)
+    print("Type 'quit' to exit")
+    print("-" * 60)
+    
+    while True:
+        try:
+            user_input = input("\n👤 Customer: ").strip()
+            
+            if user_input.lower() in ['quit', 'exit', 'q']:
+                print("👋 Terima kasih! Sampai jumpa!")
+                break
+            
+            if not user_input:
+                continue
+            
+            print("\n🤖 Textilindo AI: ", end="", flush=True)
+            response = generate_response(model, tokenizer, user_input, system_prompt)
+            print(response)
+                
+        except KeyboardInterrupt:
+            print("\n👋 Terima kasih! Sampai jumpa!")
+            break
+        except Exception as e:
+            logger.error(f"Error generating response: {e}")
+            print(f"❌ Error: {e}")
+
+def main():
+    parser = argparse.ArgumentParser(description='Textilindo AI Assistant Inference')
+    parser.add_argument('--model_path', type=str, default='./models/llama-3.1-8b-instruct',
+                        help='Path to base model')
+    parser.add_argument('--lora_path', type=str, default=None,
+                        help='Path to LoRA weights')
+    parser.add_argument('--system_prompt', type=str, default='configs/system_prompt.md',
+                        help='Path to system prompt file')
+    parser.add_argument('--prompt', type=str, default=None,
+                        help='Single prompt to process')
+    
+    args = parser.parse_args()
+    
+    print("🤖 Textilindo AI Assistant - Inference")
+    print("=" * 60)
+    
+    # Load system prompt
+    system_prompt = load_system_prompt(args.system_prompt)
+    if not system_prompt:
+        print(f"❌ System prompt tidak ditemukan: {args.system_prompt}")
+        sys.exit(1)
+    
+    # Check if model exists
+    if not os.path.exists(args.model_path):
+        print(f"❌ Base model tidak ditemukan: {args.model_path}")
+        print("Jalankan setup_textilindo_training.py terlebih dahulu")
+        sys.exit(1)
+    
+    try:
+        # Load model
+        print("1️⃣ Loading model...")
+        model, tokenizer = load_model(args.model_path, args.lora_path)
+        print("✅ Model loaded successfully!")
+        
+        if args.prompt:
+            # Single prompt mode
+            print(f"\n📝 Processing prompt: {args.prompt}")
+            response = generate_response(model, tokenizer, args.prompt, system_prompt)
+            print(f"\n🤖 Response: {response}")
+        else:
+            # Interactive mode
+            interactive_chat(model, tokenizer, system_prompt)
+            
+    except Exception as e:
+        logger.error(f"Error: {e}")
+        print(f"❌ Error loading model: {e}")
+
+if __name__ == "__main__":
+    main()

scripts/local_training_setup.py

@@ -0,0 +1,273 @@
+#!/usr/bin/env python3
+"""
+Script untuk setup training lokal dengan model yang lebih kecil
+"""
+
+import os
+import sys
+import subprocess
+from pathlib import Path
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+def check_system_requirements():
+    """Check system requirements untuk training lokal"""
+    print("🔍 Checking System Requirements...")
+    print("=" * 50)
+    
+    # Check Python version
+    python_version = sys.version_info
+    print(f"🐍 Python: {python_version.major}.{python_version.minor}.{python_version.micro}")
+    
+    if python_version < (3, 8):
+        print("❌ Python 3.8+ required")
+        return False
+    else:
+        print("✅ Python version OK")
+    
+    # Check available memory
+    try:
+        import psutil
+        memory = psutil.virtual_memory()
+        memory_gb = memory.total / (1024**3)
+        print(f"💾 RAM: {memory_gb:.1f} GB")
+        
+        if memory_gb < 8:
+            print("⚠️  Warning: Less than 8GB RAM may cause issues")
+        else:
+            print("✅ RAM sufficient")
+    except ImportError:
+        print("⚠️  psutil not available, cannot check memory")
+    
+    # Check disk space
+    try:
+        disk = psutil.disk_usage('.')
+        disk_gb = disk.free / (1024**3)
+        print(f"💿 Free Disk: {disk_gb:.1f} GB")
+        
+        if disk_gb < 10:
+            print("⚠️  Warning: Less than 10GB free space")
+        else:
+            print("✅ Disk space sufficient")
+    except:
+        print("⚠️  Cannot check disk space")
+    
+    # Check CUDA (optional)
+    try:
+        import torch
+        if torch.cuda.is_available():
+            gpu_count = torch.cuda.device_count()
+            print(f"🎮 CUDA GPUs: {gpu_count}")
+            for i in range(gpu_count):
+                gpu_name = torch.cuda.get_device_name(i)
+                gpu_memory = torch.cuda.get_device_properties(i).total_memory / (1024**3)
+                print(f"   GPU {i}: {gpu_name} ({gpu_memory:.1f} GB)")
+            print("✅ CUDA available - Fast training possible")
+        else:
+            print("⚠️  CUDA not available - Training will be slower (CPU only)")
+    except ImportError:
+        print("⚠️  PyTorch not available")
+    
+    return True
+
+def download_small_model():
+    """Download model yang cocok untuk training lokal"""
+    print("\n📥 Downloading Small Model for Local Training...")
+    print("=" * 50)
+    
+    # Model options yang cocok untuk training lokal
+    small_models = [
+        {
+            "name": "distilgpt2",
+            "path": "models/distilgpt2",
+            "size_mb": 82,
+            "description": "DistilGPT2 - Very lightweight (82M parameters)"
+        },
+        {
+            "name": "microsoft/DialoGPT-small",
+            "path": "models/dialogpt-small", 
+            "size_mb": 117,
+            "description": "DialoGPT Small - Conversational (117M parameters)"
+        },
+        {
+            "name": "EleutherAI/gpt-neo-125M",
+            "path": "models/gpt-neo-125m",
+            "size_mb": 125,
+            "description": "GPT-Neo 125M - Good balance (125M parameters)"
+        },
+        {
+            "name": "gpt2",
+            "path": "models/gpt2",
+            "size_mb": 124,
+            "description": "GPT-2 - Original but small (124M parameters)"
+        }
+    ]
+    
+    print("📋 Available small models:")
+    for i, model in enumerate(small_models, 1):
+        print(f"{i}. {model['name']}")
+        print(f"   {model['description']}")
+        print(f"   Size: ~{model['size_mb']} MB")
+        print()
+    
+    try:
+        choice = int(input("Pilih model (1-4): ").strip())
+        if choice < 1 or choice > len(small_models):
+            print("❌ Pilihan tidak valid, menggunakan default: distilgpt2")
+            choice = 1
+        
+        selected_model = small_models[choice - 1]
+        print(f"\n🎯 Selected: {selected_model['name']}")
+        
+        # Download model
+        print(f"\n📥 Downloading {selected_model['name']}...")
+        if download_model_with_transformers(selected_model['name'], selected_model['path']):
+            print(f"✅ Model downloaded successfully!")
+            return selected_model
+        else:
+            print("❌ Download failed")
+            return None
+            
+    except (ValueError, KeyboardInterrupt):
+        print("\n❌ Download cancelled")
+        return None
+
+def download_model_with_transformers(model_name, model_path):
+    """Download model menggunakan transformers library"""
+    try:
+        from transformers import AutoTokenizer, AutoModelForCausalLM
+        
+        print(f"Downloading tokenizer...")
+        tokenizer = AutoTokenizer.from_pretrained(model_name)
+        tokenizer.save_pretrained(model_path)
+        
+        print(f"Downloading model...")
+        model = AutoModelForCausalLM.from_pretrained(model_name)
+        model.save_pretrained(model_path)
+        
+        return True
+        
+    except Exception as e:
+        logger.error(f"Error downloading model: {e}")
+        return False
+
+def create_local_training_config(model_info):
+    """Create configuration untuk training lokal"""
+    config_dir = Path("configs")
+    config_dir.mkdir(exist_ok=True)
+    
+    config_content = f"""# Local Training Configuration for {model_info['name']}
+model_name: "{model_info['name']}"
+model_path: "{model_info['path']}"
+max_length: 512
+temperature: 0.7
+top_p: 0.9
+top_k: 40
+repetition_penalty: 1.1
+
+# LoRA Configuration (for memory efficiency)
+lora_config:
+  r: 8  # Reduced for smaller models
+  lora_alpha: 16
+  lora_dropout: 0.1
+  target_modules: ["q_proj", "v_proj", "k_proj", "o_proj"]
+  
+# Training Configuration (optimized for local training)
+training_config:
+  learning_rate: 1e-4  # Lower learning rate for stability
+  batch_size: 2  # Smaller batch size for memory
+  gradient_accumulation_steps: 8  # Accumulate gradients
+  num_epochs: 3
+  warmup_steps: 50
+  save_steps: 100
+  eval_steps: 100
+  max_grad_norm: 1.0
+  weight_decay: 0.01
+  
+# Hardware Configuration
+hardware_config:
+  device: "auto"  # Will use GPU if available
+  mixed_precision: true  # Use mixed precision for memory efficiency
+  gradient_checkpointing: true  # Save memory during training
+"""
+    
+    config_file = config_dir / f"local_training_{model_info['name'].split('/')[-1].lower().replace('-', '_')}.yaml"
+    with open(config_file, 'w') as f:
+        f.write(config_content)
+    
+    print(f"✅ Local training config created: {config_file}")
+    return str(config_file)
+
+def setup_local_training_environment():
+    """Setup environment untuk training lokal"""
+    print("\n🔧 Setting up Local Training Environment...")
+    print("=" * 50)
+    
+    # Install required packages
+    packages = [
+        "torch",
+        "transformers", 
+        "datasets",
+        "accelerate",
+        "peft",
+        "bitsandbytes",
+        "scipy",
+        "scikit-learn"
+    ]
+    
+    print("📦 Installing required packages...")
+    for package in packages:
+        try:
+            subprocess.run([sys.executable, "-m", "pip", "install", package], 
+                         check=True, capture_output=True)
+            print(f"✅ {package} installed")
+        except subprocess.CalledProcessError:
+            print(f"⚠️  Failed to install {package}")
+    
+    print("\n✅ Local training environment setup complete!")
+
+def main():
+    print("🚀 Local Training Setup")
+    print("=" * 50)
+    
+    # Check system requirements
+    if not check_system_requirements():
+        print("❌ System requirements not met")
+        return
+    
+    # Setup training environment
+    setup_local_training_environment()
+    
+    # Download small model
+    model_info = download_small_model()
+    if not model_info:
+        print("❌ Model download failed")
+        return
+    
+    # Create training config
+    config_file = create_local_training_config(model_info)
+    
+    print(f"\n🎉 Local Training Setup Complete!")
+    print("=" * 50)
+    print(f"📁 Model: {model_info['path']}")
+    print(f"⚙️  Config: {config_file}")
+    print(f"📊 Dataset: data/lora_dataset_20250829_113330.jsonl")
+    
+    print(f"\n📋 Next steps:")
+    print("1. Review configuration: cat configs/local_training_*.yaml")
+    print("2. Start training: python scripts/finetune_lora.py")
+    print("3. Monitor training: tail -f logs/training.log")
+    
+    print(f"\n💡 Tips for local training:")
+    print("- Use smaller batch sizes if you run out of memory")
+    print("- Enable gradient checkpointing for memory efficiency")
+    print("- Monitor GPU memory usage with nvidia-smi")
+    print("- Consider using mixed precision training")
+
+if __name__ == "__main__":
+    main()
+
+
+

scripts/novita_ai_setup.py

@@ -0,0 +1,256 @@
+#!/usr/bin/env python3
+"""
+Script untuk setup dan menggunakan Novita AI
+"""
+
+import os
+import sys
+import requests
+import json
+from pathlib import Path
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+class NovitaAIClient:
+    def __init__(self, api_key):
+        self.api_key = api_key
+        self.base_url = "https://api.novita.ai"
+        self.headers = {
+            "Authorization": f"Bearer {api_key}",
+            "Content-Type": "application/json"
+        }
+    
+    def test_connection(self):
+        """Test koneksi ke Novita AI API"""
+        try:
+            response = requests.get(
+                f"{self.base_url}/v1/models",
+                headers=self.headers
+            )
+            if response.status_code == 200:
+                logger.info("✅ Koneksi ke Novita AI berhasil!")
+                return True
+            else:
+                logger.error(f"❌ Error: {response.status_code} - {response.text}")
+                return False
+        except Exception as e:
+            logger.error(f"❌ Error koneksi: {e}")
+            return False
+    
+    def get_available_models(self):
+        """Dapatkan daftar model yang tersedia"""
+        try:
+            response = requests.get(
+                f"{self.base_url}/v1/models",
+                headers=self.headers
+            )
+            if response.status_code == 200:
+                models = response.json()
+                logger.info("📋 Model yang tersedia:")
+                for model in models.get('data', []):
+                    logger.info(f"  - {model.get('id', 'Unknown')}: {model.get('name', 'Unknown')}")
+                return models
+            else:
+                logger.error(f"❌ Error: {response.status_code}")
+                return None
+        except Exception as e:
+            logger.error(f"❌ Error: {e}")
+            return None
+    
+    def create_fine_tuning_job(self, model_name, training_file, validation_file=None):
+        """Buat fine-tuning job"""
+        try:
+            payload = {
+                "model": model_name,
+                "training_file": training_file,
+                "validation_file": validation_file,
+                "hyperparameters": {
+                    "n_epochs": 3,
+                    "batch_size": 4,
+                    "learning_rate_multiplier": 1.0
+                }
+            }
+            
+            response = requests.post(
+                f"{self.base_url}/v1/fine_tuning/jobs",
+                headers=self.headers,
+                json=payload
+            )
+            
+            if response.status_code == 200:
+                job = response.json()
+                logger.info(f"✅ Fine-tuning job created: {job.get('id')}")
+                return job
+            else:
+                logger.error(f"❌ Error: {response.status_code} - {response.text}")
+                return None
+                
+        except Exception as e:
+            logger.error(f"❌ Error: {e}")
+            return None
+    
+    def list_fine_tuning_jobs(self):
+        """List semua fine-tuning jobs"""
+        try:
+            response = requests.get(
+                f"{self.base_url}/v1/fine_tuning/jobs",
+                headers=self.headers
+            )
+            
+            if response.status_code == 200:
+                jobs = response.json()
+                logger.info("📋 Fine-tuning jobs:")
+                for job in jobs.get('data', []):
+                    status = job.get('status', 'unknown')
+                    model = job.get('model', 'unknown')
+                    job_id = job.get('id', 'unknown')
+                    logger.info(f"  - {job_id}: {model} ({status})")
+                return jobs
+            else:
+                logger.error(f"❌ Error: {response.status_code}")
+                return None
+                
+        except Exception as e:
+            logger.error(f"❌ Error: {e}")
+            return None
+    
+    def get_fine_tuning_job(self, job_id):
+        """Dapatkan detail fine-tuning job"""
+        try:
+            response = requests.get(
+                f"{self.base_url}/v1/fine_tuning/jobs/{job_id}",
+                headers=self.headers
+            )
+            
+            if response.status_code == 200:
+                job = response.json()
+                logger.info(f"📋 Job {job_id}:")
+                logger.info(f"  Status: {job.get('status')}")
+                logger.info(f"  Model: {job.get('model')}")
+                logger.info(f"  Created: {job.get('created_at')}")
+                return job
+            else:
+                logger.error(f"❌ Error: {response.status_code}")
+                return None
+                
+        except Exception as e:
+            logger.error(f"❌ Error: {e}")
+            return None
+
+def setup_novita_environment():
+    """Setup environment untuk Novita AI"""
+    print("🚀 Setup Novita AI Environment")
+    print("=" * 40)
+    
+    # Check API key
+    api_key = os.getenv('NOVITA_API_KEY')
+    if not api_key:
+        print("⚠️  NOVITA_API_KEY tidak ditemukan")
+        api_key = input("Masukkan Novita AI API key: ").strip()
+        if api_key:
+            os.environ['NOVITA_API_KEY'] = api_key
+        else:
+            print("❌ API key diperlukan untuk melanjutkan")
+            return None
+    
+    # Test connection
+    client = NovitaAIClient(api_key)
+    if not client.test_connection():
+        print("❌ Gagal koneksi ke Novita AI")
+        return None
+    
+    return client
+
+def create_sample_dataset():
+    """Buat sample dataset untuk fine-tuning"""
+    data_dir = Path("data")
+    data_dir.mkdir(exist_ok=True)
+    
+    # Sample data untuk fine-tuning
+    sample_data = [
+        {
+            "messages": [
+                {"role": "system", "content": "Anda adalah asisten AI yang membantu dengan pertanyaan dalam bahasa Indonesia."},
+                {"role": "user", "content": "Apa itu machine learning?"},
+                {"role": "assistant", "content": "Machine learning adalah cabang dari artificial intelligence yang memungkinkan komputer belajar dari data tanpa diprogram secara eksplisit."}
+            ]
+        },
+        {
+            "messages": [
+                {"role": "system", "content": "Anda adalah asisten AI yang membantu dengan pertanyaan dalam bahasa Indonesia."},
+                {"role": "user", "content": "Jelaskan tentang deep learning"},
+                {"role": "assistant", "content": "Deep learning adalah subset dari machine learning yang menggunakan neural network dengan banyak layer untuk memproses data kompleks."}
+            ]
+        },
+        {
+            "messages": [
+                {"role": "system", "content": "Anda adalah asisten AI yang membantu dengan pertanyaan dalam bahasa Indonesia."},
+                {"role": "user", "content": "Bagaimana cara kerja neural network?"},
+                {"role": "assistant", "content": "Neural network bekerja dengan menerima input, memproses melalui hidden layers, dan menghasilkan output berdasarkan weights yang telah dilatih."}
+            ]
+        }
+    ]
+    
+    # Save sebagai JSONL
+    output_file = data_dir / "training_data.jsonl"
+    with open(output_file, 'w', encoding='utf-8') as f:
+        for item in sample_data:
+            json.dump(item, f, ensure_ascii=False)
+            f.write('\n')
+    
+    print(f"✅ Sample dataset created: {output_file}")
+    return str(output_file)
+
+def main():
+    print("🤖 Novita AI Setup & Fine-tuning")
+    print("=" * 50)
+    
+    # Setup environment
+    client = setup_novita_environment()
+    if not client:
+        return
+    
+    # Get available models
+    print("\n1️⃣ Getting available models...")
+    models = client.get_available_models()
+    
+    # Create sample dataset
+    print("\n2️⃣ Creating sample dataset...")
+    training_file = create_sample_dataset()
+    
+    # Show menu
+    while True:
+        print("\n📋 Menu:")
+        print("1. List fine-tuning jobs")
+        print("2. Create fine-tuning job")
+        print("3. Check job status")
+        print("4. Exit")
+        
+        choice = input("\nPilihan (1-4): ").strip()
+        
+        if choice == "1":
+            client.list_fine_tuning_jobs()
+        elif choice == "2":
+            if models and models.get('data'):
+                model_id = input("Masukkan model ID: ").strip()
+                job = client.create_fine_tuning_job(model_id, training_file)
+                if job:
+                    print(f"✅ Job created: {job.get('id')}")
+            else:
+                print("❌ Tidak ada model tersedia")
+        elif choice == "3":
+            job_id = input("Masukkan job ID: ").strip()
+            client.get_fine_tuning_job(job_id)
+        elif choice == "4":
+            print("👋 Goodbye!")
+            break
+        else:
+            print("❌ Pilihan tidak valid")
+
+if __name__ == "__main__":
+    main()
+
+
+

scripts/novita_ai_setup_v2.py

@@ -0,0 +1,376 @@
+#!/usr/bin/env python3
+"""
+Script untuk setup dan menggunakan Novita AI (Updated Version)
+"""
+
+import os
+import sys
+import requests
+import json
+from pathlib import Path
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+class NovitaAIClient:
+    def __init__(self, api_key):
+        self.api_key = api_key
+        # Use correct Novita AI endpoint
+        self.possible_endpoints = [
+            "https://api.novita.ai/openai",
+            "https://api.novita.ai",
+            "https://api.novita.com/openai"
+        ]
+        self.base_url = None
+        self.headers = {
+            "Authorization": f"Bearer {api_key}",
+            "Content-Type": "application/json"
+        }
+    
+    def find_working_endpoint(self):
+        """Find working API endpoint"""
+        for endpoint in self.possible_endpoints:
+            try:
+                logger.info(f"🔍 Testing endpoint: {endpoint}")
+                response = requests.get(
+                    f"{endpoint}/v1/models",
+                    headers=self.headers,
+                    timeout=10
+                )
+                if response.status_code == 200:
+                    self.base_url = endpoint
+                    logger.info(f"✅ Working endpoint found: {endpoint}")
+                    return True
+                else:
+                    logger.info(f"⚠️  Endpoint {endpoint} returned {response.status_code}")
+            except Exception as e:
+                logger.info(f"❌ Endpoint {endpoint} failed: {e}")
+                continue
+        
+        return False
+    
+    def test_connection(self):
+        """Test koneksi ke Novita AI API"""
+        if not self.find_working_endpoint():
+            logger.error("❌ Tidak ada endpoint yang berfungsi")
+            return False
+        
+        try:
+            # Use OpenAI-compatible paths
+            test_paths = [
+                "/models",
+                "/v1/models",
+                "/chat/completions"
+            ]
+            
+            for path in test_paths:
+                try:
+                    response = requests.get(
+                        f"{self.base_url}{path}",
+                        headers=self.headers,
+                        timeout=10
+                    )
+                    if response.status_code == 200:
+                        logger.info(f"✅ Koneksi ke Novita AI berhasil! Endpoint: {self.base_url}{path}")
+                        return True
+                    elif response.status_code == 401:
+                        logger.error("❌ Unauthorized - API key mungkin salah")
+                        return False
+                    elif response.status_code == 404:
+                        logger.info(f"⚠️  Path {path} tidak ditemukan, mencoba yang lain...")
+                        continue
+                    else:
+                        logger.info(f"⚠️  Endpoint {path} returned {response.status_code}")
+                except Exception as e:
+                    logger.info(f"⚠️  Path {path} failed: {e}")
+                    continue
+            
+            logger.error("❌ Tidak ada endpoint yang berfungsi")
+            return False
+            
+        except Exception as e:
+            logger.error(f"❌ Error koneksi: {e}")
+            return False
+    
+    def get_available_models(self):
+        """Dapatkan daftar model yang tersedia"""
+        if not self.base_url:
+            logger.error("❌ Base URL belum diset")
+            return None
+            
+        try:
+            # Use OpenAI-compatible model endpoints
+            model_paths = [
+                "/models",
+                "/v1/models"
+            ]
+            
+            for path in model_paths:
+                try:
+                    response = requests.get(
+                        f"{self.base_url}{path}",
+                        headers=self.headers,
+                        timeout=10
+                    )
+                    if response.status_code == 200:
+                        models = response.json()
+                        logger.info("📋 Model yang tersedia:")
+                        if isinstance(models, dict) and 'data' in models:
+                            for model in models['data']:
+                                logger.info(f"  - {model.get('id', 'Unknown')}: {model.get('name', 'Unknown')}")
+                        elif isinstance(models, list):
+                            for model in models:
+                                logger.info(f"  - {model.get('id', 'Unknown')}: {model.get('name', 'Unknown')}")
+                        else:
+                            logger.info(f"  Response format: {type(models)}")
+                            logger.info(f"  Content: {models}")
+                        return models
+                    else:
+                        logger.info(f"⚠️  Path {path} returned {response.status_code}")
+                except Exception as e:
+                    logger.info(f"⚠️  Path {path} failed: {e}")
+                    continue
+            
+            logger.error("❌ Tidak bisa mendapatkan daftar model")
+            return None
+            
+        except Exception as e:
+            logger.error(f"❌ Error: {e}")
+            return None
+    
+    def create_fine_tuning_job(self, model_name, training_file, validation_file=None):
+        """Buat fine-tuning job"""
+        if not self.base_url:
+            logger.error("❌ Base URL belum diset")
+            return None
+            
+        try:
+            payload = {
+                "model": model_name,
+                "training_file": training_file,
+                "validation_file": validation_file,
+                "hyperparameters": {
+                    "n_epochs": 3,
+                    "batch_size": 4,
+                    "learning_rate_multiplier": 1.0
+                }
+            }
+            
+            # Use OpenAI-compatible fine-tuning endpoints
+            ft_paths = [
+                "/fine_tuning/jobs",
+                "/v1/fine_tuning/jobs"
+            ]
+            
+            for path in ft_paths:
+                try:
+                    response = requests.post(
+                        f"{self.base_url}{path}",
+                        headers=self.headers,
+                        json=payload,
+                        timeout=30
+                    )
+                    
+                    if response.status_code == 200:
+                        job = response.json()
+                        logger.info(f"✅ Fine-tuning job created: {job.get('id')}")
+                        return job
+                    elif response.status_code == 404:
+                        logger.info(f"⚠️  Path {path} tidak ditemukan, mencoba yang lain...")
+                        continue
+                    else:
+                        logger.error(f"❌ Error: {response.status_code} - {response.text}")
+                        continue
+                        
+                except Exception as e:
+                    logger.info(f"⚠️  Path {path} failed: {e}")
+                    continue
+            
+            logger.error("❌ Tidak bisa membuat fine-tuning job")
+            return None
+                
+        except Exception as e:
+            logger.error(f"❌ Error: {e}")
+            return None
+    
+    def list_fine_tuning_jobs(self):
+        """List semua fine-tuning jobs"""
+        if not self.base_url:
+            logger.error("❌ Base URL belum diset")
+            return None
+            
+        try:
+            # Use OpenAI-compatible job listing endpoints
+            job_paths = [
+                "/fine_tuning/jobs",
+                "/v1/fine_tuning/jobs"
+            ]
+            
+            for path in job_paths:
+                try:
+                    response = requests.get(
+                        f"{self.base_url}{path}",
+                        headers=self.headers,
+                        timeout=10
+                    )
+                    
+                    if response.status_code == 200:
+                        jobs = response.json()
+                        logger.info("📋 Fine-tuning jobs:")
+                        if isinstance(jobs, dict) and 'data' in jobs:
+                            for job in jobs['data']:
+                                status = job.get('status', 'unknown')
+                                model = job.get('model', 'unknown')
+                                job_id = job.get('id', 'unknown')
+                                logger.info(f"  - {job_id}: {model} ({status})")
+                        elif isinstance(jobs, list):
+                            for job in jobs:
+                                status = job.get('status', 'unknown')
+                                model = job.get('model', 'unknown')
+                                job_id = job.get('id', 'unknown')
+                                logger.info(f"  - {job_id}: {model} ({status})")
+                        else:
+                            logger.info(f"  Response format: {type(jobs)}")
+                            logger.info(f"  Content: {jobs}")
+                        return jobs
+                    elif response.status_code == 404:
+                        logger.info(f"⚠️  Path {path} tidak ditemukan, mencoba yang lain...")
+                        continue
+                    else:
+                        logger.error(f"❌ Error: {response.status_code}")
+                        continue
+                        
+                except Exception as e:
+                    logger.info(f"⚠️  Path {path} failed: {e}")
+                    continue
+            
+            logger.error("❌ Tidak bisa mendapatkan daftar jobs")
+            return None
+                
+        except Exception as e:
+            logger.error(f"❌ Error: {e}")
+            return None
+
+def setup_novita_environment():
+    """Setup environment untuk Novita AI"""
+    print("🚀 Setup Novita AI Environment")
+    print("=" * 40)
+    
+    # Check API key
+    api_key = os.getenv('NOVITA_API_KEY')
+    if not api_key:
+        print("⚠️  NOVITA_API_KEY tidak ditemukan")
+        api_key = input("Masukkan Novita AI API key: ").strip()
+        if api_key:
+            os.environ['NOVITA_API_KEY'] = api_key
+        else:
+            print("❌ API key diperlukan untuk melanjutkan")
+            return None
+    
+    # Test connection
+    client = NovitaAIClient(api_key)
+    if not client.test_connection():
+        print("❌ Gagal koneksi ke Novita AI")
+        print("💡 Tips:")
+        print("- Pastikan API key benar")
+        print("- Cek koneksi internet")
+        print("- Cek dokumentasi Novita AI untuk endpoint yang benar")
+        return None
+    
+    return client
+
+def create_sample_dataset():
+    """Gunakan dataset yang sudah ada atau buat yang baru jika tidak ada"""
+    data_dir = Path("data")
+    data_dir.mkdir(exist_ok=True)
+    
+    # Cek apakah dataset sudah ada
+    existing_dataset = data_dir / "lora_dataset_20250829_113330.jsonl"
+    if existing_dataset.exists():
+        print(f"✅ Dataset sudah ada: {existing_dataset}")
+        print(f"📊 File size: {existing_dataset.stat().st_size / 1024:.2f} KB")
+        return str(existing_dataset)
+    
+    # Jika tidak ada, buat sample dataset
+    print("⚠️  Dataset tidak ditemukan, membuat sample dataset...")
+    sample_data = [
+        {
+            "messages": [
+                {"role": "system", "content": "Anda adalah asisten AI yang membantu dengan pertanyaan dalam bahasa Indonesia."},
+                {"role": "user", "content": "Apa itu machine learning?"},
+                {"role": "assistant", "content": "Machine learning adalah cabang dari artificial intelligence yang memungkinkan komputer belajar dari data tanpa diprogram secara eksplisit."}
+            ]
+        },
+        {
+            "messages": [
+                {"role": "system", "content": "Anda adalah asisten AI yang membantu dengan pertanyaan dalam bahasa Indonesia."},
+                {"role": "user", "content": "Jelaskan tentang deep learning"},
+                {"role": "assistant", "content": "Deep learning adalah subset dari machine learning yang menggunakan neural network dengan banyak layer untuk memproses data kompleks."}
+            ]
+        }
+    ]
+    
+    # Save sebagai JSONL
+    output_file = data_dir / "training_data.jsonl"
+    with open(output_file, 'w', encoding='utf-8') as f:
+        for item in sample_data:
+            json.dump(item, f, ensure_ascii=False)
+            f.write('\n')
+    
+    print(f"✅ Sample dataset created: {output_file}")
+    return str(output_file)
+
+def main():
+    print("🤖 Novita AI Setup & Fine-tuning (Updated)")
+    print("=" * 50)
+    
+    # Setup environment
+    client = setup_novita_environment()
+    if not client:
+        return
+    
+    # Get available models
+    print("\n1️⃣ Getting available models...")
+    models = client.get_available_models()
+    
+    # Create sample dataset
+    print("\n2️⃣ Creating sample dataset...")
+    training_file = create_sample_dataset()
+    
+    # Show menu
+    while True:
+        print("\n📋 Menu:")
+        print("1. List fine-tuning jobs")
+        print("2. Create fine-tuning job")
+        print("3. Check job status")
+        print("4. Test API endpoints")
+        print("5. Exit")
+        
+        choice = input("\nPilihan (1-5): ").strip()
+        
+        if choice == "1":
+            client.list_fine_tuning_jobs()
+        elif choice == "2":
+            if models:
+                model_id = input("Masukkan model ID: ").strip()
+                job = client.create_fine_tuning_job(model_id, training_file)
+                if job:
+                    print(f"✅ Job created: {job.get('id')}")
+            else:
+                print("❌ Tidak ada model tersedia")
+        elif choice == "3":
+            job_id = input("Masukkan job ID: ").strip()
+            # This would need to be implemented based on actual API
+            print("⚠️  Check job status belum diimplementasikan")
+        elif choice == "4":
+            print("🔍 Testing API endpoints...")
+            client.test_connection()
+        elif choice == "5":
+            print("👋 Goodbye!")
+            break
+        else:
+            print("❌ Pilihan tidak valid")
+
+if __name__ == "__main__":
+    main()

scripts/run_novita_finetuning.py

@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""
+Script sederhana untuk menjalankan fine-tuning Novita AI
+"""
+
+import os
+import sys
+from pathlib import Path
+
+# Import NovitaAIClient dari script yang sudah ada
+sys.path.append('scripts')
+from novita_ai_setup_v2 import NovitaAIClient, create_sample_dataset
+
+def main():
+    print("🚀 Novita AI Fine-tuning - Auto Run")
+    print("=" * 50)
+    
+    # Check environment variables
+    api_key = os.getenv('NOVITA_API_KEY')
+    if not api_key:
+        print("❌ NOVITA_API_KEY tidak ditemukan")
+        print("Silakan set: export NOVITA_API_KEY='your_key'")
+        return
+    
+    base_url = os.getenv('NOVITA_BASE_URL', 'https://api.novita.ai/openai')
+    print(f"🔑 API Key: {api_key[:10]}...{api_key[-10:]}")
+    print(f"🌐 Base URL: {base_url}")
+    
+    # Create client
+    client = NovitaAIClient(api_key)
+    client.base_url = base_url
+    
+    # Test connection
+    print("\n1️⃣ Testing connection...")
+    if not client.test_connection():
+        print("❌ Koneksi gagal")
+        return
+    
+    # Get available models
+    print("\n2️⃣ Getting available models...")
+    models = client.get_available_models()
+    
+    if not models:
+        print("❌ Tidak bisa mendapatkan daftar model")
+        return
+    
+    # Select model automatically (Llama 3.2 1B Instruct if available)
+    selected_model = None
+    preferred_models = [
+        "meta-llama/llama-3.2-1b-instruct",
+        "meta-llama/llama-3.2-3b-instruct",
+        "qwen/qwen3-4b-fp8",
+        "qwen/qwen3-8b-fp8"
+    ]
+    
+    print("\n🎯 Selecting model...")
+    for preferred in preferred_models:
+        if isinstance(models, dict) and 'data' in models:
+            for model in models['data']:
+                if model.get('id') == preferred:
+                    selected_model = preferred
+                    print(f"✅ Selected: {preferred}")
+                    break
+        elif isinstance(models, list):
+            for model in models:
+                if model.get('id') == preferred:
+                    selected_model = preferred
+                    print(f"✅ Selected: {preferred}")
+                    break
+        
+        if selected_model:
+            break
+    
+    if not selected_model:
+        # Fallback to first available model
+        if isinstance(models, dict) and 'data' in models and models['data']:
+            selected_model = models['data'][0].get('id')
+        elif isinstance(models, list) and models:
+            selected_model = models[0].get('id')
+        
+        if selected_model:
+            print(f"⚠️  Fallback to: {selected_model}")
+        else:
+            print("❌ Tidak ada model yang tersedia")
+            return
+    
+    # Create dataset
+    print("\n3️⃣ Preparing dataset...")
+    training_file = create_sample_dataset()
+    
+    # Create fine-tuning job
+    print(f"\n4️⃣ Creating fine-tuning job...")
+    print(f"   Model: {selected_model}")
+    print(f"   Training file: {training_file}")
+    
+    job = client.create_fine_tuning_job(selected_model, training_file)
+    
+    if job:
+        print(f"\n✅ Fine-tuning job created successfully!")
+        print(f"   Job ID: {job.get('id')}")
+        print(f"   Status: {job.get('status', 'unknown')}")
+        print(f"   Model: {job.get('model', 'unknown')}")
+        
+        print(f"\n📋 Next steps:")
+        print(f"1. Monitor job status")
+        print(f"2. Check logs for progress")
+        print(f"3. Download fine-tuned model when complete")
+        
+    else:
+        print("\n❌ Failed to create fine-tuning job")
+        print("💡 Check the error messages above")
+
+if __name__ == "__main__":
+    main()
+
+
+

scripts/setup_textilindo_training.py

@@ -0,0 +1,175 @@
+#!/usr/bin/env python3
+"""
+Setup script untuk Textilindo AI Assistant training
+Download model dan prepare environment
+"""
+
+import os
+import sys
+import yaml
+import torch
+from pathlib import Path
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+def load_config(config_path):
+    """Load configuration from YAML file"""
+    try:
+        with open(config_path, 'r') as f:
+            config = yaml.safe_load(f)
+        return config
+    except Exception as e:
+        logger.error(f"Error loading config: {e}")
+        return None
+
+def download_model(config):
+    """Download base model"""
+    model_name = config['model_name']
+    model_path = config['model_path']
+    
+    logger.info(f"Downloading model: {model_name}")
+    logger.info(f"Target path: {model_path}")
+    
+    # Create models directory
+    Path(model_path).mkdir(parents=True, exist_ok=True)
+    
+    try:
+        # Download tokenizer
+        logger.info("Downloading tokenizer...")
+        tokenizer = AutoTokenizer.from_pretrained(
+            model_name,
+            trust_remote_code=True,
+            cache_dir=model_path
+        )
+        
+        # Download model with memory optimization
+        logger.info("Downloading model...")
+        model = AutoModelForCausalLM.from_pretrained(
+            model_name,
+            torch_dtype=torch.float16,
+            trust_remote_code=True,
+            cache_dir=model_path,
+            low_cpu_mem_usage=True,
+            load_in_8bit=True  # Use 8-bit quantization for memory efficiency
+        )
+        
+        # Save to local path
+        logger.info(f"Saving model to: {model_path}")
+        tokenizer.save_pretrained(model_path)
+        model.save_pretrained(model_path)
+        
+        logger.info("✅ Model downloaded successfully!")
+        return True
+        
+    except Exception as e:
+        logger.error(f"Error downloading model: {e}")
+        return False
+
+def check_requirements():
+    """Check if all requirements are met"""
+    print("🔍 Checking requirements...")
+    
+    # Check Python version
+    if sys.version_info < (3, 8):
+        print("❌ Python 3.8+ required")
+        return False
+    
+    # Check PyTorch
+    try:
+        import torch
+        print(f"✅ PyTorch {torch.__version__}")
+    except ImportError:
+        print("❌ PyTorch not installed")
+        return False
+    
+    # Check CUDA availability
+    if torch.cuda.is_available():
+        print(f"✅ CUDA available: {torch.cuda.get_device_name(0)}")
+        print(f"   GPU Memory: {torch.cuda.get_device_properties(0).total_memory / 1024**3:.1f} GB")
+    else:
+        print("⚠️  CUDA not available - training will be slower on CPU")
+    
+    # Check required packages
+    required_packages = [
+        'transformers',
+        'peft',
+        'datasets',
+        'accelerate',
+        'bitsandbytes'
+    ]
+    
+    missing_packages = []
+    for package in required_packages:
+        try:
+            __import__(package)
+            print(f"✅ {package}")
+        except ImportError:
+            missing_packages.append(package)
+            print(f"❌ {package}")
+    
+    if missing_packages:
+        print(f"\n❌ Missing packages: {', '.join(missing_packages)}")
+        print("Install with: pip install " + " ".join(missing_packages))
+        return False
+    
+    return True
+
+def main():
+    print("🚀 Textilindo AI Assistant - Setup")
+    print("=" * 50)
+    
+    # Check requirements
+    if not check_requirements():
+        print("\n❌ Requirements not met. Please install missing packages.")
+        sys.exit(1)
+    
+    # Load configuration
+    config_path = "configs/training_config.yaml"
+    if not os.path.exists(config_path):
+        print(f"❌ Config file tidak ditemukan: {config_path}")
+        sys.exit(1)
+    
+    config = load_config(config_path)
+    if not config:
+        sys.exit(1)
+    
+    # Check if model already exists
+    model_path = config['model_path']
+    if os.path.exists(model_path) and os.path.exists(os.path.join(model_path, "config.json")):
+        print(f"✅ Model already exists: {model_path}")
+        print("Skipping download...")
+    else:
+        # Download model
+        print("1️⃣ Downloading base model...")
+        if not download_model(config):
+            print("❌ Failed to download model")
+            sys.exit(1)
+    
+    # Check dataset
+    dataset_path = config['dataset_path']
+    if not os.path.exists(dataset_path):
+        print(f"❌ Dataset tidak ditemukan: {dataset_path}")
+        print("Please ensure your dataset is in the correct location")
+        sys.exit(1)
+    else:
+        print(f"✅ Dataset found: {dataset_path}")
+    
+    # Check system prompt
+    system_prompt_path = "configs/system_prompt.md"
+    if not os.path.exists(system_prompt_path):
+        print(f"❌ System prompt tidak ditemukan: {system_prompt_path}")
+        sys.exit(1)
+    else:
+        print(f"✅ System prompt found: {system_prompt_path}")
+    
+    print("\n✅ Setup completed successfully!")
+    print("\n📋 Next steps:")
+    print("1. Run training: python scripts/train_textilindo_ai.py")
+    print("2. Test model: python scripts/test_textilindo_ai.py")
+    print("3. Test with LoRA: python scripts/test_textilindo_ai.py --lora_path models/textilindo-ai-lora-YYYYMMDD_HHMMSS")
+
+if __name__ == "__main__":
+    main()

scripts/test_model.py

@@ -0,0 +1,201 @@
+#!/usr/bin/env python3
+"""
+Script untuk testing model yang sudah di-fine-tune
+"""
+
+import os
+import sys
+import yaml
+import torch
+from pathlib import Path
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+def load_finetuned_model(model_path, lora_weights_path):
+    """Load fine-tuned model with LoRA weights"""
+    logger.info(f"Loading base model from: {model_path}")
+    
+    # Load base model
+    model = AutoModelForCausalLM.from_pretrained(
+        model_path,
+        torch_dtype=torch.float16,
+        device_map="auto",
+        trust_remote_code=True
+    )
+    
+    # Load LoRA weights
+    logger.info(f"Loading LoRA weights from: {lora_weights_path}")
+    model = PeftModel.from_pretrained(model, lora_weights_path)
+    
+    # Load tokenizer
+    tokenizer = AutoTokenizer.from_pretrained(
+        model_path,
+        trust_remote_code=True
+    )
+    
+    if tokenizer.pad_token is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    
+    return model, tokenizer
+
+def generate_response(model, tokenizer, prompt, max_length=512):
+    """Generate response from the model"""
+    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+    
+    with torch.no_grad():
+        outputs = model.generate(
+            **inputs,
+            max_length=max_length,
+            temperature=0.7,
+            top_p=0.9,
+            top_k=40,
+            repetition_penalty=1.1,
+            do_sample=True,
+            pad_token_id=tokenizer.eos_token_id
+        )
+    
+    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+    return response
+
+def interactive_test(model, tokenizer):
+    """Interactive testing mode"""
+    print("🤖 Interactive Testing Mode")
+    print("Type 'quit' to exit")
+    print("-" * 50)
+    
+    while True:
+        try:
+            user_input = input("\n👤 You: ").strip()
+            
+            if user_input.lower() in ['quit', 'exit', 'q']:
+                print("👋 Goodbye!")
+                break
+            
+            if not user_input:
+                continue
+            
+            print("\n🤖 Assistant: ", end="")
+            response = generate_response(model, tokenizer, user_input)
+            
+            # Extract only the generated part (remove input)
+            if user_input in response:
+                generated_part = response.split(user_input)[-1].strip()
+                print(generated_part)
+            else:
+                print(response)
+                
+        except KeyboardInterrupt:
+            print("\n👋 Goodbye!")
+            break
+        except Exception as e:
+            logger.error(f"Error generating response: {e}")
+            print(f"❌ Error: {e}")
+
+def batch_test(model, tokenizer, test_cases):
+    """Batch testing with predefined test cases"""
+    print("🧪 Batch Testing Mode")
+    print("=" * 50)
+    
+    for i, test_case in enumerate(test_cases, 1):
+        print(f"\n📝 Test Case {i}: {test_case['prompt']}")
+        print("-" * 40)
+        
+        try:
+            response = generate_response(model, tokenizer, test_case['prompt'])
+            print(f"🤖 Response: {response}")
+            
+            if 'expected' in test_case:
+                print(f"🎯 Expected: {test_case['expected']}")
+                
+        except Exception as e:
+            logger.error(f"Error in test case {i}: {e}")
+            print(f"❌ Error: {e}")
+
+def main():
+    print("🧪 Model Testing - Fine-tuned Llama 3.1 8B")
+    print("=" * 50)
+    
+    # Check if model exists
+    base_model_path = "models/llama-3.1-8b-instruct"
+    lora_weights_path = "models/finetuned-llama-lora"
+    
+    if not os.path.exists(base_model_path):
+        print(f"❌ Base model tidak ditemukan: {base_model_path}")
+        print("Jalankan download_model.py terlebih dahulu")
+        sys.exit(1)
+    
+    if not os.path.exists(lora_weights_path):
+        print(f"⚠️  LoRA weights tidak ditemukan: {lora_weights_path}")
+        print("Model akan menggunakan base model tanpa fine-tuning")
+        lora_weights_path = None
+    
+    try:
+        # Load model
+        print("1️⃣ Loading model...")
+        if lora_weights_path:
+            model, tokenizer = load_finetuned_model(base_model_path, lora_weights_path)
+        else:
+            from transformers import AutoTokenizer, AutoModelForCausalLM
+            model = AutoModelForCausalLM.from_pretrained(
+                base_model_path,
+                torch_dtype=torch.float16,
+                device_map="auto",
+                trust_remote_code=True
+            )
+            tokenizer = AutoTokenizer.from_pretrained(
+                base_model_path,
+                trust_remote_code=True
+            )
+        
+        print("✅ Model loaded successfully!")
+        
+        # Test cases
+        test_cases = [
+            {
+                "prompt": "Apa itu machine learning?",
+                "expected": "Penjelasan tentang machine learning"
+            },
+            {
+                "prompt": "Jelaskan tentang deep learning dalam bahasa Indonesia",
+                "expected": "Penjelasan tentang deep learning"
+            },
+            {
+                "prompt": "Buat puisi tentang teknologi",
+                "expected": "Puisi tentang teknologi"
+            }
+        ]
+        
+        # Choose testing mode
+        print("\n2️⃣ Pilih mode testing:")
+        print("1. Interactive mode (chat)")
+        print("2. Batch testing")
+        print("3. Custom prompt")
+        
+        choice = input("\nPilihan (1-3): ").strip()
+        
+        if choice == "1":
+            interactive_test(model, tokenizer)
+        elif choice == "2":
+            batch_test(model, tokenizer, test_cases)
+        elif choice == "3":
+            custom_prompt = input("Masukkan prompt custom: ").strip()
+            if custom_prompt:
+                response = generate_response(model, tokenizer, custom_prompt)
+                print(f"\n🤖 Response: {response}")
+        else:
+            print("❌ Pilihan tidak valid")
+            
+    except Exception as e:
+        logger.error(f"Error: {e}")
+        print(f"❌ Error loading model: {e}")
+
+if __name__ == "__main__":
+    main()
+
+
+
+

scripts/test_novita_connection.py

@@ -0,0 +1,158 @@
+#!/usr/bin/env python3
+"""
+Simple script untuk test koneksi Novita AI
+"""
+
+import os
+import requests
+import json
+
+def test_novita_connection():
+    """Test koneksi ke Novita AI dengan berbagai cara"""
+    
+    api_key = os.getenv('NOVITA_API_KEY')
+    if not api_key:
+        print("❌ NOVITA_API_KEY tidak ditemukan")
+        return
+    
+    print(f"🔑 API Key: {api_key[:10]}...{api_key[-10:]}")
+    print("🔍 Testing koneksi ke Novita AI...")
+    
+    # Test different possible endpoints
+    endpoints_to_test = [
+        "https://api.novita.ai",
+        "https://api.novita.com",
+        "https://novita.ai/api",
+        "https://novita.com/api",
+        "https://api.novita.ai/v1",
+        "https://api.novita.com/v1",
+        "https://novita.ai/api/v1",
+        "https://novita.com/api/v1"
+    ]
+    
+    headers = {
+        "Authorization": f"Bearer {api_key}",
+        "Content-Type": "application/json"
+    }
+    
+    working_endpoints = []
+    
+    for endpoint in endpoints_to_test:
+        print(f"\n🔍 Testing: {endpoint}")
+        
+        # Test basic connectivity
+        try:
+            # Test GET request
+            response = requests.get(f"{endpoint}/models", headers=headers, timeout=10)
+            print(f"  GET /models: {response.status_code}")
+            
+            if response.status_code == 200:
+                print(f"  ✅ Success! Response: {response.text[:200]}...")
+                working_endpoints.append(endpoint)
+            elif response.status_code == 401:
+                print(f"  ⚠️  Unauthorized - API key mungkin salah")
+            elif response.status_code == 404:
+                print(f"  ⚠️  Not Found - Endpoint tidak ada")
+            else:
+                print(f"  ⚠️  Status: {response.status_code}")
+                
+        except requests.exceptions.ConnectionError as e:
+            print(f"  ❌ Connection Error: {e}")
+        except requests.exceptions.Timeout as e:
+            print(f"  ⏰ Timeout: {e}")
+        except Exception as e:
+            print(f"  ❌ Error: {e}")
+        
+        # Test POST request
+        try:
+            test_data = {"test": "connection"}
+            response = requests.post(f"{endpoint}/test", headers=headers, json=test_data, timeout=10)
+            print(f"  POST /test: {response.status_code}")
+        except Exception as e:
+            print(f"  ❌ POST Error: {e}")
+    
+    print(f"\n📊 Summary:")
+    if working_endpoints:
+        print(f"✅ Working endpoints: {len(working_endpoints)}")
+        for endpoint in working_endpoints:
+            print(f"  - {endpoint}")
+    else:
+        print("❌ No working endpoints found")
+        print("\n💡 Suggestions:")
+        print("1. Check if the API key is correct")
+        print("2. Check Novita AI documentation for correct endpoints")
+        print("3. Try using a different API key")
+        print("4. Check if there are any IP restrictions")
+    
+    return working_endpoints
+
+def test_openai_compatible():
+    """Test if Novita AI is OpenAI compatible"""
+    print("\n🤖 Testing OpenAI compatibility...")
+    
+    api_key = os.getenv('NOVITA_API_KEY')
+    if not api_key:
+        print("❌ NOVITA_API_KEY tidak ditemukan")
+        return
+    
+    # Try OpenAI-compatible endpoints
+    openai_endpoints = [
+        "https://api.novita.ai/v1",
+        "https://api.novita.com/v1",
+        "https://novita.ai/api/v1",
+        "https://novita.com/api/v1"
+    ]
+    
+    headers = {
+        "Authorization": f"Bearer {api_key}",
+        "Content-Type": "application/json"
+    }
+    
+    for endpoint in openai_endpoints:
+        print(f"\n🔍 Testing OpenAI endpoint: {endpoint}")
+        
+        try:
+            # Test models endpoint
+            response = requests.get(f"{endpoint}/models", headers=headers, timeout=10)
+            print(f"  GET /models: {response.status_code}")
+            
+            if response.status_code == 200:
+                print(f"  ✅ Success!")
+                try:
+                    models = response.json()
+                    print(f"  📋 Models: {json.dumps(models, indent=2)[:300]}...")
+                except:
+                    print(f"  📋 Response: {response.text[:200]}...")
+            elif response.status_code == 401:
+                print(f"  ⚠️  Unauthorized")
+            elif response.status_code == 404:
+                print(f"  ⚠️  Not Found")
+            else:
+                print(f"  ⚠️  Status: {response.status_code}")
+                
+        except Exception as e:
+            print(f"  ❌ Error: {e}")
+
+def main():
+    print("🔍 Novita AI Connection Tester")
+    print("=" * 40)
+    
+    # Test basic connection
+    working_endpoints = test_novita_connection()
+    
+    # Test OpenAI compatibility
+    test_openai_compatible()
+    
+    print(f"\n🎯 Next Steps:")
+    if working_endpoints:
+        print("✅ Koneksi berhasil! Anda bisa melanjutkan dengan fine-tuning")
+        print("💡 Gunakan endpoint yang berfungsi untuk setup selanjutnya")
+    else:
+        print("❌ Koneksi gagal. Cek dokumentasi Novita AI")
+        print("💡 Atau gunakan alternatif lain seperti local models")
+
+if __name__ == "__main__":
+    main()
+
+
+

scripts/test_textilindo_ai.py

@@ -0,0 +1,235 @@
+#!/usr/bin/env python3
+"""
+Script untuk testing Textilindo AI Assistant yang sudah di-fine-tune
+"""
+
+import os
+import sys
+import yaml
+import torch
+import argparse
+from pathlib import Path
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+def load_system_prompt(system_prompt_path):
+    """Load system prompt from markdown file"""
+    try:
+        with open(system_prompt_path, 'r', encoding='utf-8') as f:
+            content = f.read()
+        
+        # Extract SYSTEM_PROMPT from markdown
+        if 'SYSTEM_PROMPT = """' in content:
+            start = content.find('SYSTEM_PROMPT = """') + len('SYSTEM_PROMPT = """')
+            end = content.find('"""', start)
+            system_prompt = content[start:end].strip()
+        else:
+            # Fallback: use entire content
+            system_prompt = content.strip()
+        
+        return system_prompt
+    except Exception as e:
+        logger.error(f"Error loading system prompt: {e}")
+        return None
+
+def load_finetuned_model(model_path, lora_weights_path, system_prompt):
+    """Load fine-tuned model with LoRA weights"""
+    logger.info(f"Loading base model from: {model_path}")
+    
+    # Load base model
+    model = AutoModelForCausalLM.from_pretrained(
+        model_path,
+        torch_dtype=torch.float16,
+        device_map="auto",
+        trust_remote_code=True
+    )
+    
+    # Load LoRA weights if available
+    if lora_weights_path and os.path.exists(lora_weights_path):
+        logger.info(f"Loading LoRA weights from: {lora_weights_path}")
+        model = PeftModel.from_pretrained(model, lora_weights_path)
+    else:
+        logger.warning("No LoRA weights found, using base model")
+    
+    # Load tokenizer
+    tokenizer = AutoTokenizer.from_pretrained(
+        model_path,
+        trust_remote_code=True
+    )
+    
+    if tokenizer.pad_token is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    
+    return model, tokenizer
+
+def generate_response(model, tokenizer, user_input, system_prompt, max_length=512):
+    """Generate response from the model"""
+    # Create full prompt with system prompt
+    full_prompt = f"<|system|>\n{system_prompt}\n<|user|>\n{user_input}\n<|assistant|>\n"
+    
+    inputs = tokenizer(full_prompt, return_tensors="pt").to(model.device)
+    
+    with torch.no_grad():
+        outputs = model.generate(
+            **inputs,
+            max_length=max_length,
+            temperature=0.7,
+            top_p=0.9,
+            top_k=40,
+            repetition_penalty=1.1,
+            do_sample=True,
+            pad_token_id=tokenizer.eos_token_id,
+            eos_token_id=tokenizer.eos_token_id,
+            stop_strings=["<|end|>", "<|user|>"]
+        )
+    
+    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+    
+    # Extract only the assistant's response
+    if "<|assistant|>" in response:
+        assistant_response = response.split("<|assistant|>")[-1].strip()
+        # Remove any remaining special tokens
+        assistant_response = assistant_response.replace("<|end|>", "").strip()
+        return assistant_response
+    else:
+        return response
+
+def interactive_test(model, tokenizer, system_prompt):
+    """Interactive testing mode"""
+    print("🤖 Textilindo AI Assistant - Interactive Mode")
+    print("=" * 60)
+    print("Type 'quit' to exit")
+    print("-" * 60)
+    
+    while True:
+        try:
+            user_input = input("\n👤 Customer: ").strip()
+            
+            if user_input.lower() in ['quit', 'exit', 'q']:
+                print("👋 Terima kasih! Sampai jumpa!")
+                break
+            
+            if not user_input:
+                continue
+            
+            print("\n🤖 Textilindo AI: ", end="", flush=True)
+            response = generate_response(model, tokenizer, user_input, system_prompt)
+            print(response)
+                
+        except KeyboardInterrupt:
+            print("\n👋 Terima kasih! Sampai jumpa!")
+            break
+        except Exception as e:
+            logger.error(f"Error generating response: {e}")
+            print(f"❌ Error: {e}")
+
+def batch_test(model, tokenizer, system_prompt, test_cases):
+    """Batch testing with predefined test cases"""
+    print("🧪 Textilindo AI Assistant - Batch Testing")
+    print("=" * 60)
+    
+    for i, test_case in enumerate(test_cases, 1):
+        print(f"\n📝 Test Case {i}: {test_case['prompt']}")
+        print("-" * 40)
+        
+        try:
+            response = generate_response(model, tokenizer, test_case['prompt'], system_prompt)
+            print(f"🤖 Response: {response}")
+            
+            if 'expected' in test_case:
+                print(f"🎯 Expected: {test_case['expected']}")
+                
+        except Exception as e:
+            logger.error(f"Error in test case {i}: {e}")
+            print(f"❌ Error: {e}")
+
+def main():
+    parser = argparse.ArgumentParser(description='Test Textilindo AI Assistant')
+    parser.add_argument('--model_path', type=str, default='./models/llama-3.1-8b-instruct',
+                        help='Path to base model')
+    parser.add_argument('--lora_path', type=str, default=None,
+                        help='Path to LoRA weights')
+    parser.add_argument('--system_prompt', type=str, default='configs/system_prompt.md',
+                        help='Path to system prompt file')
+    
+    args = parser.parse_args()
+    
+    print("🧪 Textilindo AI Assistant Testing")
+    print("=" * 60)
+    
+    # Load system prompt
+    system_prompt = load_system_prompt(args.system_prompt)
+    if not system_prompt:
+        print(f"❌ System prompt tidak ditemukan: {args.system_prompt}")
+        sys.exit(1)
+    
+    # Check if model exists
+    if not os.path.exists(args.model_path):
+        print(f"❌ Base model tidak ditemukan: {args.model_path}")
+        print("Jalankan download_model.py terlebih dahulu")
+        sys.exit(1)
+    
+    try:
+        # Load model
+        print("1️⃣ Loading model...")
+        model, tokenizer = load_finetuned_model(args.model_path, args.lora_path, system_prompt)
+        print("✅ Model loaded successfully!")
+        
+        # Test cases specific to Textilindo
+        test_cases = [
+            {
+                "prompt": "dimana lokasi textilindo?",
+                "expected": "Textilindo berkantor pusat di Jl. Raya Prancis No.39, Kosambi Tim., Kec. Kosambi, Kabupaten Tangerang, Banten 15213"
+            },
+            {
+                "prompt": "Jam berapa textilindo beroperasional?",
+                "expected": "Jam operasional Senin-Jumat 08:00-17:00, Sabtu 08:00-12:00."
+            },
+            {
+                "prompt": "Berapa ketentuan pembelian?",
+                "expected": "Minimal order 1 roll per jenis kain"
+            },
+            {
+                "prompt": "bagimana dengan pembayarannya?",
+                "expected": "Pembayaran dapat dilakukan via transfer bank atau cash on delivery"
+            },
+            {
+                "prompt": "apa ada gratis ongkir?",
+                "expected": "Gratis ongkir untuk order minimal 5 roll."
+            },
+            {
+                "prompt": "Apa bisa dikirimkan sample? apa gratis?",
+                "expected": "hallo kak untuk sampel kita bisa kirimkan gratis ya kak 😊"
+            }
+        ]
+        
+        # Choose testing mode
+        print("\n2️⃣ Pilih mode testing:")
+        print("1. Interactive mode (chat)")
+        print("2. Batch testing")
+        print("3. Custom prompt")
+        
+        choice = input("\nPilihan (1-3): ").strip()
+        
+        if choice == "1":
+            interactive_test(model, tokenizer, system_prompt)
+        elif choice == "2":
+            batch_test(model, tokenizer, system_prompt, test_cases)
+        elif choice == "3":
+            custom_prompt = input("Masukkan prompt custom: ").strip()
+            if custom_prompt:
+                response = generate_response(model, tokenizer, custom_prompt, system_prompt)
+                print(f"\n🤖 Response: {response}")
+        else:
+            print("❌ Pilihan tidak valid")
+            
+    except Exception as e:
+        logger.error(f"Error: {e}")
+        print(f"❌ Error loading model: {e}")
+
+if __name__ == "__main__":
+    main()

scripts/train_textilindo_ai.py

@@ -0,0 +1,282 @@
+#!/usr/bin/env python3
+"""
+Script untuk fine-tuning Llama 3.2 1B dengan LoRA untuk Textilindo AI Assistant
+Menggunakan system prompt dan dataset khusus Textilindo
+"""
+
+import os
+import sys
+import yaml
+import json
+import torch
+from pathlib import Path
+from transformers import (
+    AutoTokenizer, 
+    AutoModelForCausalLM,
+    TrainingArguments,
+    Trainer,
+    DataCollatorForLanguageModeling
+)
+from peft import (
+    LoraConfig,
+    get_peft_model,
+    TaskType,
+    prepare_model_for_kbit_training
+)
+from datasets import Dataset
+import logging
+from datetime import datetime
+
+# Setup logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+def load_config(config_path):
+    """Load configuration from YAML file"""
+    try:
+        with open(config_path, 'r') as f:
+            config = yaml.safe_load(f)
+        return config
+    except Exception as e:
+        logger.error(f"Error loading config: {e}")
+        return None
+
+def load_system_prompt(system_prompt_path):
+    """Load system prompt from markdown file"""
+    try:
+        with open(system_prompt_path, 'r', encoding='utf-8') as f:
+            content = f.read()
+        
+        # Extract SYSTEM_PROMPT from markdown
+        if 'SYSTEM_PROMPT = """' in content:
+            start = content.find('SYSTEM_PROMPT = """') + len('SYSTEM_PROMPT = """')
+            end = content.find('"""', start)
+            system_prompt = content[start:end].strip()
+        else:
+            # Fallback: use entire content
+            system_prompt = content.strip()
+        
+        return system_prompt
+    except Exception as e:
+        logger.error(f"Error loading system prompt: {e}")
+        return None
+
+def load_model_and_tokenizer(config):
+    """Load base model and tokenizer"""
+    model_path = config['model_path']
+    
+    logger.info(f"Loading model from: {model_path}")
+    
+    # Load tokenizer
+    tokenizer = AutoTokenizer.from_pretrained(
+        model_path,
+        trust_remote_code=True,
+        padding_side="right"
+    )
+    
+    if tokenizer.pad_token is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    
+    # Load model with memory optimization
+    model = AutoModelForCausalLM.from_pretrained(
+        model_path,
+        torch_dtype=torch.float16,
+        device_map="auto",
+        trust_remote_code=True,
+        low_cpu_mem_usage=True,
+        load_in_8bit=True  # Use 8-bit quantization for memory efficiency
+    )
+    
+    # Prepare model for k-bit training
+    model = prepare_model_for_kbit_training(model)
+    
+    return model, tokenizer
+
+def setup_lora_config(config):
+    """Setup LoRA configuration"""
+    lora_config = config['lora_config']
+    
+    peft_config = LoraConfig(
+        task_type=TaskType.CAUSAL_LM,
+        r=lora_config['r'],
+        lora_alpha=lora_config['lora_alpha'],
+        lora_dropout=lora_config['lora_dropout'],
+        target_modules=lora_config['target_modules'],
+        bias="none",
+    )
+    
+    return peft_config
+
+def prepare_textilindo_dataset(data_path, tokenizer, system_prompt, max_length=2048):
+    """Prepare Textilindo dataset for training with system prompt"""
+    logger.info(f"Loading dataset from: {data_path}")
+    
+    # Load JSONL dataset
+    data = []
+    with open(data_path, 'r', encoding='utf-8') as f:
+        for line_num, line in enumerate(f, 1):
+            line = line.strip()
+            if line:
+                try:
+                    json_obj = json.loads(line)
+                    data.append(json_obj)
+                except json.JSONDecodeError as e:
+                    logger.warning(f"Invalid JSON at line {line_num}: {e}")
+                    continue
+    
+    if not data:
+        raise ValueError("No valid JSON objects found in JSONL file")
+    
+    logger.info(f"Loaded {len(data)} samples from JSONL file")
+    
+    # Convert to training format with system prompt
+    training_data = []
+    for item in data:
+        # Extract instruction and output
+        instruction = item.get('instruction', '')
+        output = item.get('output', '')
+        
+        if not instruction or not output:
+            continue
+        
+        # Create training text with system prompt
+        training_text = f"<|system|>\n{system_prompt}\n<|user|>\n{instruction}\n<|assistant|>\n{output}<|end|>"
+        
+        training_data.append({
+            'text': training_text,
+            'instruction': instruction,
+            'output': output
+        })
+    
+    # Convert to Dataset
+    dataset = Dataset.from_list(training_data)
+    logger.info(f"Prepared {len(dataset)} training samples")
+    
+    def tokenize_function(examples):
+        # Tokenize the texts
+        tokenized = tokenizer(
+            examples['text'],
+            truncation=True,
+            padding=True,
+            max_length=max_length,
+            return_tensors="pt"
+        )
+        return tokenized
+    
+    # Tokenize dataset
+    tokenized_dataset = dataset.map(
+        tokenize_function,
+        batched=True,
+        remove_columns=dataset.column_names
+    )
+    
+    return tokenized_dataset
+
+def train_model(model, tokenizer, dataset, config, output_dir):
+    """Train the model with LoRA"""
+    training_config = config['training_config']
+    
+    # Setup training arguments
+    training_args = TrainingArguments(
+        output_dir=output_dir,
+        num_train_epochs=training_config['num_epochs'],
+        per_device_train_batch_size=training_config['batch_size'],
+        gradient_accumulation_steps=training_config['gradient_accumulation_steps'],
+        learning_rate=training_config['learning_rate'],
+        warmup_steps=training_config['warmup_steps'],
+        save_steps=training_config['save_steps'],
+        eval_steps=training_config['eval_steps'],
+        logging_steps=training_config.get('logging_steps', 10),
+        save_total_limit=training_config.get('save_total_limit', 3),
+        prediction_loss_only=training_config.get('prediction_loss_only', True),
+        remove_unused_columns=training_config.get('remove_unused_columns', False),
+        push_to_hub=training_config.get('push_to_hub', False),
+        report_to=training_config.get('report_to', None),
+        fp16=True,  # Enable mixed precision training
+        dataloader_pin_memory=False,  # Reduce memory usage
+    )
+    
+    # Setup data collator
+    data_collator = DataCollatorForLanguageModeling(
+        tokenizer=tokenizer,
+        mlm=False,
+    )
+    
+    # Setup trainer
+    trainer = Trainer(
+        model=model,
+        args=training_args,
+        train_dataset=dataset,
+        data_collator=data_collator,
+        tokenizer=tokenizer,
+    )
+    
+    # Start training
+    logger.info("Starting training...")
+    trainer.train()
+    
+    # Save the model
+    trainer.save_model()
+    logger.info(f"Model saved to: {output_dir}")
+
+def main():
+    print("🚀 Textilindo AI Assistant - LoRA Fine-tuning")
+    print("=" * 60)
+    
+    # Load configuration
+    config_path = "configs/training_config.yaml"
+    if not os.path.exists(config_path):
+        print(f"❌ Config file tidak ditemukan: {config_path}")
+        sys.exit(1)
+    
+    config = load_config(config_path)
+    if not config:
+        sys.exit(1)
+    
+    # Load system prompt
+    system_prompt_path = "configs/system_prompt.md"
+    if not os.path.exists(system_prompt_path):
+        print(f"❌ System prompt tidak ditemukan: {system_prompt_path}")
+        sys.exit(1)
+    
+    system_prompt = load_system_prompt(system_prompt_path)
+    if not system_prompt:
+        sys.exit(1)
+    
+    # Setup paths
+    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+    output_dir = Path(f"models/textilindo-ai-lora-{timestamp}")
+    output_dir.mkdir(parents=True, exist_ok=True)
+    
+    # Check if dataset exists
+    data_path = config['dataset_path']
+    if not os.path.exists(data_path):
+        print(f"❌ Dataset tidak ditemukan: {data_path}")
+        sys.exit(1)
+    
+    # Load model and tokenizer
+    print("1️⃣ Loading model and tokenizer...")
+    model, tokenizer = load_model_and_tokenizer(config)
+    
+    # Setup LoRA
+    print("2️⃣ Setting up LoRA configuration...")
+    peft_config = setup_lora_config(config)
+    model = get_peft_model(model, peft_config)
+    
+    # Print trainable parameters
+    model.print_trainable_parameters()
+    
+    # Prepare dataset
+    print("3️⃣ Preparing Textilindo dataset...")
+    dataset = prepare_textilindo_dataset(data_path, tokenizer, system_prompt, config['max_length'])
+    
+    # Train model
+    print("4️⃣ Starting training...")
+    train_model(model, tokenizer, dataset, config, output_dir)
+    
+    print("✅ Training selesai!")
+    print(f"📁 Model tersimpan di: {output_dir}")
+    print(f"🔧 Untuk testing, jalankan: python scripts/test_textilindo_ai.py --model_path {output_dir}")
+
+if __name__ == "__main__":
+    main()

scripts/train_textilindo_ai_optimized.py

@@ -0,0 +1,296 @@
+#!/usr/bin/env python3
+"""
+Memory-optimized training script untuk Textilindo AI Assistant
+Optimized untuk Llama 3.1 8B dengan LoRA pada laptop
+"""
+
+import os
+import sys
+import yaml
+import json
+import torch
+from pathlib import Path
+from transformers import (
+    AutoTokenizer, 
+    AutoModelForCausalLM,
+    TrainingArguments,
+    Trainer,
+    DataCollatorForLanguageModeling,
+    BitsAndBytesConfig
+)
+from peft import (
+    LoraConfig,
+    get_peft_model,
+    TaskType,
+    prepare_model_for_kbit_training
+)
+from datasets import Dataset
+import logging
+from datetime import datetime
+import gc
+
+# Setup logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+def load_config(config_path):
+    """Load configuration from YAML file"""
+    try:
+        with open(config_path, 'r') as f:
+            config = yaml.safe_load(f)
+        return config
+    except Exception as e:
+        logger.error(f"Error loading config: {e}")
+        return None
+
+def load_system_prompt(system_prompt_path):
+    """Load system prompt from markdown file"""
+    try:
+        with open(system_prompt_path, 'r', encoding='utf-8') as f:
+            content = f.read()
+        
+        # Extract SYSTEM_PROMPT from markdown
+        if 'SYSTEM_PROMPT = """' in content:
+            start = content.find('SYSTEM_PROMPT = """') + len('SYSTEM_PROMPT = """')
+            end = content.find('"""', start)
+            system_prompt = content[start:end].strip()
+        else:
+            # Fallback: use entire content
+            system_prompt = content.strip()
+        
+        return system_prompt
+    except Exception as e:
+        logger.error(f"Error loading system prompt: {e}")
+        return None
+
+def load_model_and_tokenizer(config):
+    """Load base model and tokenizer with memory optimization"""
+    model_path = config['model_path']
+    
+    logger.info(f"Loading model from: {model_path}")
+    
+    # Configure quantization for memory efficiency
+    bnb_config = BitsAndBytesConfig(
+        load_in_4bit=True,
+        bnb_4bit_use_double_quant=True,
+        bnb_4bit_quant_type="nf4",
+        bnb_4bit_compute_dtype=torch.float16
+    )
+    
+    # Load tokenizer
+    tokenizer = AutoTokenizer.from_pretrained(
+        model_path,
+        trust_remote_code=True,
+        padding_side="right"
+    )
+    
+    if tokenizer.pad_token is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    
+    # Load model with 4-bit quantization
+    model = AutoModelForCausalLM.from_pretrained(
+        model_path,
+        quantization_config=bnb_config,
+        device_map="auto",
+        trust_remote_code=True,
+        low_cpu_mem_usage=True
+    )
+    
+    # Prepare model for k-bit training
+    model = prepare_model_for_kbit_training(model)
+    
+    return model, tokenizer
+
+def setup_lora_config(config):
+    """Setup LoRA configuration optimized for 8B model"""
+    lora_config = config['lora_config']
+    
+    peft_config = LoraConfig(
+        task_type=TaskType.CAUSAL_LM,
+        r=lora_config['r'],
+        lora_alpha=lora_config['lora_alpha'],
+        lora_dropout=lora_config['lora_dropout'],
+        target_modules=lora_config['target_modules'],
+        bias="none",
+    )
+    
+    return peft_config
+
+def prepare_textilindo_dataset(data_path, tokenizer, system_prompt, max_length=1024):
+    """Prepare Textilindo dataset for training with system prompt"""
+    logger.info(f"Loading dataset from: {data_path}")
+    
+    # Load JSONL dataset
+    data = []
+    with open(data_path, 'r', encoding='utf-8') as f:
+        for line_num, line in enumerate(f, 1):
+            line = line.strip()
+            if line:
+                try:
+                    json_obj = json.loads(line)
+                    data.append(json_obj)
+                except json.JSONDecodeError as e:
+                    logger.warning(f"Invalid JSON at line {line_num}: {e}")
+                    continue
+    
+    if not data:
+        raise ValueError("No valid JSON objects found in JSONL file")
+    
+    logger.info(f"Loaded {len(data)} samples from JSONL file")
+    
+    # Convert to training format with system prompt
+    training_data = []
+    for item in data:
+        # Extract instruction and output
+        instruction = item.get('instruction', '')
+        output = item.get('output', '')
+        
+        if not instruction or not output:
+            continue
+        
+        # Create training text with system prompt (shorter for memory efficiency)
+        training_text = f"<|system|>\n{system_prompt[:500]}...\n<|user|>\n{instruction}\n<|assistant|>\n{output}<|end|>"
+        
+        training_data.append({
+            'text': training_text,
+            'instruction': instruction,
+            'output': output
+        })
+    
+    # Convert to Dataset
+    dataset = Dataset.from_list(training_data)
+    logger.info(f"Prepared {len(dataset)} training samples")
+    
+    def tokenize_function(examples):
+        # Tokenize the texts
+        tokenized = tokenizer(
+            examples['text'],
+            truncation=True,
+            padding=True,
+            max_length=max_length,
+            return_tensors="pt"
+        )
+        return tokenized
+    
+    # Tokenize dataset
+    tokenized_dataset = dataset.map(
+        tokenize_function,
+        batched=True,
+        remove_columns=dataset.column_names
+    )
+    
+    return tokenized_dataset
+
+def train_model(model, tokenizer, dataset, config, output_dir):
+    """Train the model with LoRA - memory optimized"""
+    training_config = config['training_config']
+    
+    # Setup training arguments optimized for memory
+    training_args = TrainingArguments(
+        output_dir=output_dir,
+        num_train_epochs=training_config['num_epochs'],
+        per_device_train_batch_size=training_config['batch_size'],
+        gradient_accumulation_steps=training_config['gradient_accumulation_steps'],
+        learning_rate=training_config['learning_rate'],
+        warmup_steps=training_config['warmup_steps'],
+        save_steps=training_config['save_steps'],
+        eval_steps=training_config['eval_steps'],
+        logging_steps=training_config.get('logging_steps', 5),
+        save_total_limit=training_config.get('save_total_limit', 2),
+        prediction_loss_only=training_config.get('prediction_loss_only', True),
+        remove_unused_columns=training_config.get('remove_unused_columns', False),
+        push_to_hub=training_config.get('push_to_hub', False),
+        report_to=training_config.get('report_to', None),
+        fp16=True,  # Enable mixed precision training
+        dataloader_pin_memory=False,  # Reduce memory usage
+        dataloader_num_workers=0,  # Reduce memory usage
+        gradient_checkpointing=True,  # Enable gradient checkpointing for memory
+        optim="adamw_torch",  # Use AdamW optimizer
+        max_grad_norm=1.0,  # Gradient clipping
+    )
+    
+    # Setup data collator
+    data_collator = DataCollatorForLanguageModeling(
+        tokenizer=tokenizer,
+        mlm=False,
+    )
+    
+    # Setup trainer
+    trainer = Trainer(
+        model=model,
+        args=training_args,
+        train_dataset=dataset,
+        data_collator=data_collator,
+        tokenizer=tokenizer,
+    )
+    
+    # Start training
+    logger.info("Starting training...")
+    trainer.train()
+    
+    # Save the model
+    trainer.save_model()
+    logger.info(f"Model saved to: {output_dir}")
+
+def main():
+    print("🚀 Textilindo AI Assistant - Memory Optimized LoRA Training")
+    print("=" * 70)
+    
+    # Load configuration
+    config_path = "configs/training_config.yaml"
+    if not os.path.exists(config_path):
+        print(f"❌ Config file tidak ditemukan: {config_path}")
+        sys.exit(1)
+    
+    config = load_config(config_path)
+    if not config:
+        sys.exit(1)
+    
+    # Load system prompt
+    system_prompt_path = "configs/system_prompt.md"
+    if not os.path.exists(system_prompt_path):
+        print(f"❌ System prompt tidak ditemukan: {system_prompt_path}")
+        sys.exit(1)
+    
+    system_prompt = load_system_prompt(system_prompt_path)
+    if not system_prompt:
+        sys.exit(1)
+    
+    # Setup paths
+    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+    output_dir = Path(f"models/textilindo-ai-lora-8b-{timestamp}")
+    output_dir.mkdir(parents=True, exist_ok=True)
+    
+    # Check if dataset exists
+    data_path = config['dataset_path']
+    if not os.path.exists(data_path):
+        print(f"❌ Dataset tidak ditemukan: {data_path}")
+        sys.exit(1)
+    
+    # Load model and tokenizer
+    print("1️⃣ Loading model and tokenizer (4-bit quantized)...")
+    model, tokenizer = load_model_and_tokenizer(config)
+    
+    # Setup LoRA
+    print("2️⃣ Setting up LoRA configuration...")
+    peft_config = setup_lora_config(config)
+    model = get_peft_model(model, peft_config)
+    
+    # Print trainable parameters
+    model.print_trainable_parameters()
+    
+    # Prepare dataset
+    print("3️⃣ Preparing Textilindo dataset...")
+    dataset = prepare_textilindo_dataset(data_path, tokenizer, system_prompt, config['max_length'])
+    
+    # Train model
+    print("4️⃣ Starting training...")
+    print("⚠️  This may take several hours. Monitor GPU memory usage.")
+    train_model(model, tokenizer, dataset, config, output_dir)
+    
+    print("✅ Training selesai!")
+    print(f"📁 Model tersimpan di: {output_dir}")
+    print(f"🔧 Untuk testing, jalankan: python scripts/test_textilindo_ai.py --model_path {output_dir}")
+
+if __name__ == "__main__":
+    main()

scripts/train_with_monitoring.py

@@ -0,0 +1,228 @@
+#!/usr/bin/env python3
+"""
+Script untuk training dengan monitoring GPU dan logging yang lengkap
+"""
+
+import os
+import sys
+import time
+import json
+import psutil
+import GPUtil
+from pathlib import Path
+from datetime import datetime
+import logging
+from finetune_lora import main as finetune_main
+
+def setup_logging():
+    """Setup logging dengan format yang lengkap"""
+    log_dir = Path("logs")
+    log_dir.mkdir(exist_ok=True)
+    
+    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+    log_file = log_dir / f"training_{timestamp}.log"
+    
+    # Setup logging format
+    logging.basicConfig(
+        level=logging.INFO,
+        format='%(asctime)s - %(levelname)s - %(message)s',
+        handlers=[
+            logging.FileHandler(log_file, encoding='utf-8'),
+            logging.StreamHandler(sys.stdout)
+        ]
+    )
+    
+    return logging.getLogger(__name__)
+
+def get_system_info():
+    """Get system information"""
+    info = {
+        "timestamp": datetime.now().isoformat(),
+        "cpu_count": psutil.cpu_count(),
+        "memory_total_gb": round(psutil.virtual_memory().total / (1024**3), 2),
+        "memory_available_gb": round(psutil.virtual_memory().available / (1024**3), 2),
+        "disk_usage": {}
+    }
+    
+    # Disk usage
+    for partition in psutil.disk_partitions():
+        try:
+            usage = psutil.disk_usage(partition.mountpoint)
+            info["disk_usage"][partition.mountpoint] = {
+                "total_gb": round(usage.total / (1024**3), 2),
+                "used_gb": round(usage.used / (1024**3), 2),
+                "free_gb": round(usage.free / (1024**3), 2),
+                "percent": usage.percent
+            }
+        except PermissionError:
+            continue
+    
+    return info
+
+def get_gpu_info():
+    """Get GPU information"""
+    try:
+        gpus = GPUtil.getGPUs()
+        gpu_info = []
+        
+        for gpu in gpus:
+            gpu_info.append({
+                "id": gpu.id,
+                "name": gpu.name,
+                "memory_total_mb": gpu.memoryTotal,
+                "memory_used_mb": gpu.memoryUsed,
+                "memory_free_mb": gpu.memoryFree,
+                "memory_utilization_percent": gpu.memoryUtil * 100,
+                "gpu_utilization_percent": gpu.load * 100,
+                "temperature_celsius": gpu.temperature
+            })
+        
+        return gpu_info
+    except Exception as e:
+        logging.warning(f"Could not get GPU info: {e}")
+        return []
+
+def monitor_resources(logger, interval=30):
+    """Monitor system resources during training"""
+    logger.info("🔍 Starting resource monitoring...")
+    
+    start_time = time.time()
+    monitoring_data = []
+    
+    try:
+        while True:
+            # Get current resource usage
+            current_time = time.time()
+            elapsed_time = current_time - start_time
+            
+            # System info
+            system_info = get_system_info()
+            system_info["elapsed_time_seconds"] = elapsed_time
+            
+            # GPU info
+            gpu_info = get_gpu_info()
+            
+            # Memory usage
+            memory = psutil.virtual_memory()
+            system_info["memory_used_gb"] = round(memory.used / (1024**3), 2)
+            system_info["memory_percent"] = memory.percent
+            
+            # CPU usage
+            system_info["cpu_percent"] = psutil.cpu_percent(interval=1)
+            
+            # Combine all info
+            monitoring_entry = {
+                "timestamp": datetime.now().isoformat(),
+                "elapsed_time_seconds": elapsed_time,
+                "system": system_info,
+                "gpu": gpu_info
+            }
+            
+            monitoring_data.append(monitoring_entry)
+            
+            # Log summary
+            logger.info(f"⏱️  Elapsed: {elapsed_time/60:.1f}min | "
+                       f"CPU: {system_info['cpu_percent']:.1f}% | "
+                       f"RAM: {system_info['memory_percent']:.1f}%")
+            
+            if gpu_info:
+                for gpu in gpu_info:
+                    logger.info(f"🎮 GPU {gpu['id']}: "
+                               f"Util: {gpu['gpu_utilization_percent']:.1f}% | "
+                               f"Memory: {gpu['memory_utilization_percent']:.1f}% | "
+                               f"Temp: {gpu['temperature_celsius']:.1f}°C")
+            
+            # Save monitoring data periodically
+            if len(monitoring_data) % 10 == 0:  # Every 10 entries
+                monitoring_file = Path("logs") / f"monitoring_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
+                with open(monitoring_file, 'w') as f:
+                    json.dump(monitoring_data, f, indent=2)
+                logger.info(f"💾 Monitoring data saved: {monitoring_file}")
+            
+            time.sleep(interval)
+            
+    except KeyboardInterrupt:
+        logger.info("⏹️  Resource monitoring stopped by user")
+    
+    return monitoring_data
+
+def main():
+    """Main function untuk training dengan monitoring"""
+    print("🚀 Training dengan Monitoring - Llama 3.1 8B LoRA")
+    print("=" * 60)
+    
+    # Setup logging
+    logger = setup_logging()
+    
+    # Log system information
+    logger.info("🖥️  System Information:")
+    system_info = get_system_info()
+    for key, value in system_info.items():
+        if key != "disk_usage":
+            logger.info(f"  {key}: {value}")
+    
+    # Log GPU information
+    gpu_info = get_gpu_info()
+    if gpu_info:
+        logger.info("🎮 GPU Information:")
+        for gpu in gpu_info:
+            logger.info(f"  GPU {gpu['id']}: {gpu['name']}")
+            logger.info(f"    Memory: {gpu['memory_total_mb']}MB total")
+            logger.info(f"    Temperature: {gpu['temperature_celsius']}°C")
+    else:
+        logger.warning("⚠️  No GPU detected. Training will be very slow on CPU!")
+    
+    # Check prerequisites
+    logger.info("🔍 Checking prerequisites...")
+    
+    # Check if model exists
+    model_path = Path("models/llama-3.1-8b-instruct")
+    if not model_path.exists():
+        logger.error("❌ Base model not found. Please run download_model.py first!")
+        return
+    
+    # Check if dataset exists
+    data_path = Path("data/training_data.jsonl")
+    if not data_path.exists():
+        logger.error("❌ Training dataset not found. Please run create_sample_dataset.py first!")
+        return
+    
+    # Check if config exists
+    config_path = Path("configs/llama_config.yaml")
+    if not config_path.exists():
+        logger.error("❌ Model configuration not found. Please run download_model.py first!")
+        return
+    
+    logger.info("✅ All prerequisites met!")
+    
+    # Start resource monitoring in background
+    import threading
+    monitoring_thread = threading.Thread(
+        target=monitor_resources, 
+        args=(logger, 30),  # Monitor every 30 seconds
+        daemon=True
+    )
+    monitoring_thread.start()
+    
+    # Start training
+    logger.info("🚀 Starting LoRA fine-tuning...")
+    try:
+        finetune_main()
+        logger.info("✅ Training completed successfully!")
+    except Exception as e:
+        logger.error(f"❌ Training failed: {e}")
+        raise
+    finally:
+        logger.info("📊 Training session ended")
+        
+        # Save final monitoring data
+        monitoring_file = Path("logs") / f"final_monitoring_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
+        # Note: In a real implementation, you'd want to capture the monitoring data
+        logger.info(f"💾 Final monitoring data saved: {monitoring_file}")
+
+if __name__ == "__main__":
+    main()
+
+
+
+

test_deployment.py

@@ -0,0 +1,266 @@
+#!/usr/bin/env python3
+"""
+Test script for Textilindo AI Assistant deployment
+Run this to verify your setup before deploying to Hugging Face Spaces
+"""
+
+import os
+import sys
+import json
+import requests
+from pathlib import Path
+
+def test_file_structure():
+    """Test if all required files exist"""
+    print("🔍 Testing file structure...")
+    
+    required_files = [
+        "app.py",
+        "Dockerfile", 
+        "requirements.txt",
+        "configs/system_prompt.md",
+        "configs/training_config.yaml",
+        "templates/chat.html"
+    ]
+    
+    required_dirs = [
+        "data",
+        "configs",
+        "templates"
+    ]
+    
+    missing_files = []
+    missing_dirs = []
+    
+    for file in required_files:
+        if not Path(file).exists():
+            missing_files.append(file)
+    
+    for dir in required_dirs:
+        if not Path(dir).exists():
+            missing_dirs.append(dir)
+    
+    if missing_files:
+        print(f"❌ Missing files: {missing_files}")
+        return False
+    
+    if missing_dirs:
+        print(f"❌ Missing directories: {missing_dirs}")
+        return False
+    
+    print("✅ All required files and directories exist")
+    return True
+
+def test_data_files():
+    """Test if data files exist and are valid"""
+    print("🔍 Testing data files...")
+    
+    data_dir = Path("data")
+    if not data_dir.exists():
+        print("❌ Data directory not found")
+        return False
+    
+    jsonl_files = list(data_dir.glob("*.jsonl"))
+    if not jsonl_files:
+        print("❌ No JSONL files found in data directory")
+        return False
+    
+    print(f"✅ Found {len(jsonl_files)} JSONL files:")
+    for file in jsonl_files:
+        print(f"  - {file.name}")
+    
+    # Test one JSONL file
+    test_file = jsonl_files[0]
+    try:
+        with open(test_file, 'r', encoding='utf-8') as f:
+            lines = f.readlines()
+        
+        if not lines:
+            print(f"❌ {test_file.name} is empty")
+            return False
+        
+        # Test first line
+        first_line = lines[0].strip()
+        if first_line:
+            json.loads(first_line)
+            print(f"✅ {test_file.name} contains valid JSON")
+        
+        print(f"✅ {test_file.name} has {len(lines)} lines")
+        return True
+        
+    except json.JSONDecodeError as e:
+        print(f"❌ {test_file.name} contains invalid JSON: {e}")
+        return False
+    except Exception as e:
+        print(f"❌ Error reading {test_file.name}: {e}")
+        return False
+
+def test_config_files():
+    """Test if configuration files are valid"""
+    print("🔍 Testing configuration files...")
+    
+    # Test system prompt
+    prompt_file = Path("configs/system_prompt.md")
+    if not prompt_file.exists():
+        print("❌ System prompt file not found")
+        return False
+    
+    try:
+        with open(prompt_file, 'r', encoding='utf-8') as f:
+            content = f.read()
+        
+        if 'SYSTEM_PROMPT' not in content:
+            print("⚠️  SYSTEM_PROMPT not found in system_prompt.md")
+        else:
+            print("✅ System prompt file is valid")
+        
+    except Exception as e:
+        print(f"❌ Error reading system prompt: {e}")
+        return False
+    
+    # Test training config
+    config_file = Path("configs/training_config.yaml")
+    if not config_file.exists():
+        print("❌ Training config file not found")
+        return False
+    
+    try:
+        import yaml
+        with open(config_file, 'r') as f:
+            config = yaml.safe_load(f)
+        
+        required_fields = ['model_name', 'model_path', 'dataset_path']
+        for field in required_fields:
+            if field not in config:
+                print(f"❌ Missing field in config: {field}")
+                return False
+        
+        print("✅ Training configuration is valid")
+        return True
+        
+    except Exception as e:
+        print(f"❌ Error reading training config: {e}")
+        return False
+
+def test_app_import():
+    """Test if the app can be imported"""
+    print("🔍 Testing app import...")
+    
+    try:
+        # Add current directory to path
+        sys.path.insert(0, '.')
+        
+        # Try to import the app
+        import app
+        print("✅ App module imported successfully")
+        
+        # Test if FastAPI app exists
+        if hasattr(app, 'app'):
+            print("✅ FastAPI app found")
+        else:
+            print("❌ FastAPI app not found")
+            return False
+        
+        return True
+        
+    except ImportError as e:
+        print(f"❌ Error importing app: {e}")
+        return False
+    except Exception as e:
+        print(f"❌ Unexpected error: {e}")
+        return False
+
+def test_environment():
+    """Test environment variables"""
+    print("🔍 Testing environment...")
+    
+    # Check if HUGGINGFACE_API_KEY is set
+    api_key = os.getenv('HUGGINGFACE_API_KEY')
+    if api_key:
+        print("✅ HUGGINGFACE_API_KEY is set")
+    else:
+        print("⚠️  HUGGINGFACE_API_KEY not set (will use mock responses)")
+    
+    # Check Python version
+    python_version = sys.version_info
+    if python_version >= (3, 8):
+        print(f"✅ Python version {python_version.major}.{python_version.minor} is compatible")
+    else:
+        print(f"❌ Python version {python_version.major}.{python_version.minor} is too old (need 3.8+)")
+        return False
+    
+    return True
+
+def test_dependencies():
+    """Test if required dependencies can be imported"""
+    print("🔍 Testing dependencies...")
+    
+    required_modules = [
+        'fastapi',
+        'uvicorn',
+        'pydantic',
+        'requests',
+        'huggingface_hub'
+    ]
+    
+    missing_modules = []
+    
+    for module in required_modules:
+        try:
+            __import__(module)
+            print(f"✅ {module}")
+        except ImportError:
+            missing_modules.append(module)
+            print(f"❌ {module}")
+    
+    if missing_modules:
+        print(f"❌ Missing modules: {missing_modules}")
+        print("Install with: pip install " + " ".join(missing_modules))
+        return False
+    
+    return True
+
+def main():
+    """Run all tests"""
+    print("🧪 Textilindo AI Assistant - Deployment Test")
+    print("=" * 50)
+    
+    tests = [
+        test_file_structure,
+        test_data_files,
+        test_config_files,
+        test_environment,
+        test_dependencies,
+        test_app_import
+    ]
+    
+    passed = 0
+    total = len(tests)
+    
+    for test in tests:
+        try:
+            if test():
+                passed += 1
+            print()
+        except Exception as e:
+            print(f"❌ Test failed with error: {e}")
+            print()
+    
+    print("=" * 50)
+    print(f"📊 Test Results: {passed}/{total} tests passed")
+    
+    if passed == total:
+        print("🎉 All tests passed! Your setup is ready for deployment.")
+        print("\n📋 Next steps:")
+        print("1. Run: ./deploy_to_hf_space.sh")
+        print("2. Or manually deploy to Hugging Face Spaces")
+        print("3. Set environment variables in your space settings")
+        print("4. Test your deployed application")
+    else:
+        print("❌ Some tests failed. Please fix the issues above before deploying.")
+        return 1
+    
+    return 0
+
+if __name__ == "__main__":
+    sys.exit(main())