Fresh deploy: all latest files

.gitignore

@@ -0,0 +1,195 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+archive/
+deploy.sh
+# C extensions
+*.so
+test*
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
+.pdm.toml
+.pdm-python
+.pdm-build/
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Abstra
+# Abstra is an AI-powered process automation framework.
+# Ignore directories containing user credentials, local state, and settings.
+# Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  you could uncomment the following to ignore the enitre vscode folder
+# .vscode/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Cursor
+#  Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to
+#  exclude from AI features like autocomplete and code analysis. Recommended for sensitive data
+#  refer to https://docs.cursor.com/context/ignore-files
+.cursorignore
+.cursorindexingignore

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Jing Bi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,93 @@
+---
+title: Perceptual Copilot
+emoji: 👁️
+colorFrom: yellow
+colorTo: purple
+sdk: gradio
+sdk_version: 5.33.1
+app_file: main.py
+pinned: false
+license: mit
+---
+
+## ✨ What is Perceptual Copilot?
+
+Perceptual Copilot is a prototype that demonstrates the integration of OpenAI agents with visual tools to process real-time video streams. This experimental platform showcases both the promising potential and current limitations of equipping agents with vision capabilities to understand and interact with live visual data. 
+
+
+### Architecture Overview
+
+
+
+```
+┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+│      Webcam     │───▶│      Memory     │◀──▶│      Gradio     │
+└─────────────────┘    └─────────────────┘    └─────────────────┘
+                                │
+                                ▼
+                       ┌─────────────────┐    ┌─────────────────┐
+                       │      Agent      │◀──▶│      Tools      │
+                       └─────────────────┘    └─────────────────┘
+```
+
+### Available Tools
+
+| Tool | Description | Output |
+|------|-------------|---------|
+| `caption` | Generate detailed image descriptions | Rich visual descriptions |
+| `ocr` | Extract text from images | Extracted text content |
+| `localize` | Detect and locate objects | Bounding boxes with labels |
+| `qa` | Answer questions about images | Contextual answers |
+| `time` | Get current timestamp | Current date and time |
+| _More tools coming soon..._ | Additional capabilities in development | Various outputs |
+
+## 🚀 Quick Start
+
+### Prerequisites
+
+- Webcam access
+
+### Installation
+
+1. **Install dependencies**
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+2. **Set up environment variables**
+   ```bash
+   export HF_TOKEN="your_huggingface_token"
+   export API_KEY="your_openai_api_key"
+   export END_LANG="your_llm_endpoint"
+   export END_TASK="your_task_endpoint"
+   export MODEL_AGENT="your_agent_model"
+   export MODEL_MLLM="your_multimodal_model"
+   export MODEL_LOC="your_localization_model"
+   ```
+
+3. **Launch the application**
+   ```bash
+   python main.py
+   ```
+
+## 💡 Usage Examples
+
+### Basic Interaction
+- **User**: "What do you see?"
+- **Assistant**: *Generates detailed caption of current view*
+
+### OCR Functionality
+- **User**: "Read the text in this document"
+- **Assistant**: *Extracts and returns all visible text*
+
+### Object Detection
+- **User**: "What objects are in front of me?"
+- **Assistant**: *Identifies and localizes objects with bounding boxes*
+
+
+## Acknowledgments
+
+- Built with [Gradio](https://gradio.app/) for the interactive web interface
+- Uses [Supervision](https://supervision.roboflow.com/) for frame annotation
+- WebRTC integration via [FastRTC](https://github.com/gradio-app/gradio)
+

app/agent.py

@@ -0,0 +1,28 @@
+
+from agents import Agent
+from app.memory import Memory
+from openai import AsyncOpenAI
+from app.config import env
+from agents import set_default_openai_client, set_default_openai_api, set_tracing_disabled
+from app.tool import caption, ocr, localize, qa, time, video_caption, video_qa
+
+def build_agent():
+    client = AsyncOpenAI(base_url=env.end_lang,api_key=env.api_key)
+    set_default_openai_client(client=client, use_for_tracing=False)
+    set_default_openai_api("chat_completions")
+    set_tracing_disabled(disabled=True)
+    chat_agent = Agent[Memory](
+        name="Assistant",
+        # tools=[caption, ocr, qa, time, localize, video_caption, video_qa],
+        tools=[caption, ocr, qa, time, video_caption, video_qa],
+        model=env.model_agent,
+        instructions=(
+            "Your name is Perceptual Copilot. As a helpful assistant, your functions include answering questions about images, "
+            "Optical Character Recognition (OCR), image caption generation, object localization "
+            "within images, and video caption generation and Q&A. For video-related tools, you "
+            "will need to determine the appropriate time window to analyze from the past."
+        ),
+    )
+
+    return chat_agent
+

app/config.py

@@ -0,0 +1,40 @@
+import os
+import logging
+from openai import OpenAI
+
+
+
+
+try:
+    from dotenv import load_dotenv
+    load_dotenv()
+except ImportError:
+    pass
+
+
+class Envs:
+    def __init__(self):
+        self.hf_token = os.getenv("HF_TOKEN")
+        self.api_key = os.getenv("API_KEY")
+        self.end_task = os.getenv("END_TASK")
+        self.end_lang = os.getenv("END_LANG")
+        self.model_agent = os.getenv("MODEL_AGENT")
+        self.model_mllm = os.getenv("MODEL_MLLM")
+        self.model_loc = os.getenv("MODEL_LOC")
+        
+        # Only initialize OpenAI client if we have the required env vars
+        if self.end_lang and self.api_key:
+            self.client = OpenAI(base_url=self.end_lang, api_key=self.api_key)
+        else:
+            self.client = None
+            print("WARNING: OpenAI client not initialized due to missing environment variables")
+            
+        self.debug = os.getenv("DEBUG", "1").lower() in ("true", "1", "yes")
+        self.fps = int(os.getenv("FPS", "1"))
+    
+
+env = Envs()
+
+logger = logging.getLogger('copilot')
+logger.setLevel(logging.DEBUG if env.debug else logging.INFO)
+logger.addHandler(logging.StreamHandler())

app/memory.py

@@ -0,0 +1,300 @@
+import asyncio
+from dataclasses import dataclass, field
+from agents import Runner, RunHooks
+import threading
+from typing import Any, Dict, Optional, List
+import traceback
+import time
+from datetime import datetime
+import numpy as np
+import gradio as gr
+
+from .config import logger, env
+@dataclass
+class RunnerStep:
+    """Log entry for a single Runner step"""
+    timestamp: str
+    step_type: str
+    agent_name: str
+    turn_number: int
+    details: Dict[str, Any] = field(default_factory=dict)
+    duration_ms: Optional[float] = None
+
+    def __str__(self) -> str:
+        return f"[{self.timestamp}][T{self.turn_number}][{self.step_type}]: {self.details}"
+
+@dataclass
+class Message:
+    role: str
+    content: str
+    mode: str
+    metadata: Dict[str, Any] = field(default_factory=dict)
+
+    @classmethod
+    def user(cls, content: str) -> "Message":
+        return cls("user", content, '')
+    
+    @classmethod
+    def system(cls, content: str) -> "Message":
+        return cls("system", content, '')
+
+    @classmethod
+    def tool(cls, content: str, **kwargs) -> "Message":
+        return cls("assistant", content, 'tool', kwargs)
+    
+    @classmethod
+    def assistant(cls, content: str, mode='') -> "Message":
+        return cls("assistant", content, mode)
+    
+    @classmethod
+    def tts(cls, content: str) -> "Message":
+        return cls("assistant", content, 'tts')
+    
+    def to_dict(self) -> Dict[str, Any]:
+        result = {"role": self.role, "content": self.content}
+        if self.mode == "tool":
+            metadata = self.metadata.copy()
+            if title := metadata.get("title"):
+                metadata["title"] = title.title()
+            result["metadata"] = metadata
+        return result
+
+
+@dataclass
+class Snapshot:
+    sender: str
+    data: Any
+    status: str = 'done'
+    
+    @property
+    def gr(self):
+        if isinstance(self.data, np.ndarray):
+            return gr.Image(self.data)
+        return self.data
+
+
+class RunnerLoggerHooks(RunHooks):
+    """Custom hooks to log every step of the Runner"""
+    
+    def __init__(self, memory_instance):
+        super().__init__()
+        self.memory = memory_instance
+        self.current_turn = 0
+        self.turn_start_time = None
+    
+    async def on_agent_start(self, context, agent):
+        self.current_turn += 1
+        self.turn_start_time = time.time()
+        self.memory.snapshots.append(Snapshot(
+            sender='agent',
+            data='Deciding tools',
+            status='pending'
+        ))
+        step = RunnerStep(
+            timestamp=datetime.now().isoformat(),
+            step_type="turn_start",
+            agent_name=agent.name,
+            turn_number=self.current_turn,
+            details={"message": f"Starting turn {self.current_turn} with agent {agent.name}"}
+        )
+        self.memory.log_runner_step(step)
+    
+    async def on_agent_end(self, context, agent, result):
+        if self.turn_start_time:
+            duration = (time.time() - self.turn_start_time) * 1000
+        else:
+            duration = None
+            
+        step = RunnerStep(
+            timestamp=datetime.now().isoformat(),
+            step_type="agent_call",
+            agent_name=agent.name,
+            turn_number=self.current_turn,
+            details={"message": f"Agent {agent.name} completed", "result_type": type(result).__name__},
+            duration_ms=duration
+        )
+        self.memory.log_runner_step(step)
+    
+    async def on_tool_start(self, context, agent, tool_call):
+        tool_name = getattr(tool_call, 'name', 'unknown')
+        tool_args = None
+        for attr in ['arguments', 'args', 'function', 'parameters']:
+            if hasattr(tool_call, attr):
+                tool_args = getattr(tool_call, attr)
+                break
+        self.memory.snapshots.append(Snapshot(
+            sender='agent',
+            status='pending',
+            data=f'Calling **{tool_name}**'
+        ))
+        step = RunnerStep(
+            timestamp=datetime.now().isoformat(),
+            step_type="tool_call",
+            agent_name=agent.name,
+            turn_number=self.current_turn,
+            details={
+                "tool_name": tool_name,
+                "tool_args": tool_args,
+                "message": f"Calling tool {tool_name}"
+            }
+        )
+        self.memory.log_runner_step(step)
+    
+    async def on_tool_end(self, context, agent, tool_call, result):
+        # Handle different tool_call object attributes safely
+        tool_name = getattr(tool_call, 'name', 'unknown')
+        
+        step = RunnerStep(
+            timestamp=datetime.now().isoformat(),
+            step_type="tool_result",
+            agent_name=agent.name,
+            turn_number=self.current_turn,
+            details={
+                "tool_name": tool_name,
+                "result_length": len(str(result)) if result else 0,
+                "message": f"Tool {tool_name} completed"
+            }
+        )
+        self.memory.log_runner_step(step)
+
+
+class Chat:
+    def __init__(self):
+        self.history = [] 
+
+    def append(self, message: Message):
+        self.history.append(message)
+
+    def clear_pending_tools(self) -> None:
+        self.history = [
+            message
+            for message in self.history
+            if not (
+                message.mode == "tool"
+                and message.metadata.get("status") == "pending"
+            )
+        ]
+    
+    @property
+    def messages(self):
+        return [i.to_dict() for i in self.history]
+
+
+class Memory:
+    def __init__(self, agent, limit: int = 200) -> None:
+        self.limit: int = limit
+        self.frames: list[Any] = []  
+        self.snapshots: list[Any] = []      
+        self.inputs: list[Any] = [] 
+        self.chat = Chat()
+
+        self.runner_steps: List[RunnerStep] = []
+        self.step_limit: int = 1000  # Keep last 1000 steps
+        self.logger_hooks: Optional[RunnerLoggerHooks] = None
+
+        self._chat_q: asyncio.Queue[Any] = asyncio.Queue()
+        self._input_q: asyncio.Queue[Any] = asyncio.Queue()
+        self._loop: Optional[asyncio.AbstractEventLoop] = None
+        self.is_waiting: bool = False
+        self.is_running: bool = False
+        self._last_frame_time: float = 0
+        self.setup(agent)
+
+    def log_runner_step(self, step: RunnerStep) -> None:
+        """Log a runner step and maintain the step history limit"""
+        self.runner_steps.append(step)
+        logger.debug(f"[ 🛠️ ]{step}")
+        while len(self.runner_steps) > self.step_limit:
+            self.runner_steps.pop(0)
+
+    def enqueue(self, data: Any) -> None:
+        current_time = time.time()
+        if  current_time-self._last_frame_time > 1.0 / env.fps:
+            self._last_frame_time = current_time
+            self.frames.append(data)
+            while len(self.frames) > self.limit:
+                self.frames.pop(0)
+        return self.snapshots.pop(0) if self.snapshots else None
+    
+    def receive(self, text: str) -> None:
+        self.chat.append(Message.user(text))
+        self._loop.call_soon_threadsafe(self._chat_q.put_nowait, text)
+
+
+    def setup(self, agent) -> None:
+        """Bind *agent* and spawn the background monitor threads."""
+        self.v_agent = agent
+        self.logger_hooks = RunnerLoggerHooks(self)
+        def _runner() -> None:
+            self._loop = asyncio.new_event_loop()
+            asyncio.set_event_loop(self._loop)
+            try:
+                self._loop.create_task(self._monitor_chat())
+                self._loop.run_forever()
+            finally:
+                self._loop.close()
+
+        threading.Thread(target=_runner, daemon=True).start()
+    
+    async def _monitor_chat(self) -> None:
+        """Process incoming chat messages, respecting the waiting gate."""
+        while True:
+            text = await self._chat_q.get()
+            logger.debug(f"Processing: {text}")
+            start_step = RunnerStep(
+                timestamp=datetime.now().isoformat(),
+                step_type="processing_start",
+                agent_name=getattr(self.v_agent, 'name', 'unknown'),
+                turn_number=0,
+                details={"user_input": text}
+            )
+            self.log_runner_step(start_step)
+            
+            try:
+                self.is_running = True
+                result = await Runner.run(
+                    starting_agent=self.v_agent,
+                    input=text,
+                    context=self,
+                    hooks=self.logger_hooks  # Add our custom hooks here
+                )
+                
+                self.is_running = False
+                
+                # Log successful completion
+                success_step = RunnerStep(
+                    timestamp=datetime.now().isoformat(),
+                    step_type="final_output",
+                    agent_name=getattr(self.v_agent, 'name', 'unknown'),
+                    turn_number=self.logger_hooks.current_turn if self.logger_hooks else 0,
+                    details={
+                        "output_type": type(result.final_output).__name__,
+                        "output_preview": str(result.final_output)[:100] + "..." if len(str(result.final_output)) > 100 else str(result.final_output)
+                    }
+                )
+                self.log_runner_step(success_step)
+                
+            except Exception as exc:  # noqa: BLE001
+                self.is_running = False
+                full_traceback = traceback.format_exc()
+                logger.debug(f"Error in _monitor_chat: {exc}\n{full_traceback}")
+                self.chat.clear_pending_tools()
+                
+                # Log the error
+                error_step = RunnerStep(
+                    timestamp=datetime.now().isoformat(),
+                    step_type="error",
+                    agent_name=getattr(self.v_agent, 'name', 'unknown'),
+                    turn_number=self.logger_hooks.current_turn if self.logger_hooks else 0,
+                    details={
+                        "error_type": type(exc).__name__,
+                        "error_message": str(exc),
+                        "traceback": full_traceback
+                    }
+                )
+                self.log_runner_step(error_step)
+                continue
+            final = result.final_output.split('</think>', 1)[-1]
+            self.chat.clear_pending_tools()
+            self.chat.append(Message.assistant(final))
+            await asyncio.sleep(0)

app/tool.py

@@ -0,0 +1,184 @@
+import datetime
+import json
+import cv2
+import httpx
+from app.config import env
+from app.utils import image_w_box, encode_image
+from agents import RunContextWrapper, function_tool
+from app.memory import Memory,Snapshot
+
+
+
+
+def task(name, image):
+    resp = httpx.post(f"{env.end_task}",
+        data={"name": name},
+        files={"file": ("frame.jpg", image.tobytes(), "image/jpeg")},
+        timeout=10,
+        headers={"Authorization": env.api_key},
+    )
+    resp.raise_for_status()
+    return resp.json()['result']
+
+def completion(messages, model):
+    response = env.client.chat.completions.create(
+        model=model,
+        messages=messages
+    )
+    return response.choices[0].message.content
+
+
+def completion_image(images, prompt, model):
+    messages = [
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": prompt},
+                {"type": "image_url", "image_url": {"url": f"data:{mime};base64,{b64}"}},
+            ],
+        }
+        for b64, mime in map(encode_image, images)
+    ]
+    return completion(messages, model=model)
+
+# ------------------------ Function Tools ------------------------
+@function_tool
+def caption(wrapper: RunContextWrapper[Memory]) -> str:  
+    """
+    Generate a descriptive caption for the most recent frame, record it as a snapshot, and return it.
+    Returns:
+        str:
+            The generated caption for the current view (i.e., the latest frame).
+    """
+    mem = wrapper.context
+    prompt = "Describe the image with rich details but in a concise manner."
+    result = completion_image([mem.frames[-1]], prompt, env.model_mllm)
+    mem.snapshots.append(Snapshot(sender='caption', data=result))
+    return result
+
+@function_tool
+def ocr(wrapper: RunContextWrapper[Memory]) -> str:  
+    """
+    Perform OCR on the most recent frame, record it as a snapshot, and return the extracted text.
+    Returns:
+        str:
+            The extracted text from the current view (i.e., the latest frame).
+    """
+    mem = wrapper.context
+    prompt = "Extract all text from image/payslip without miss anything."
+    result = completion_image([mem.frames[-1]], prompt, env.model_mllm)
+    mem.snapshots.append(Snapshot(sender='ocr', data=result))
+    return result
+
+@function_tool
+def qa(wrapper: RunContextWrapper[Memory], question: str) -> str:  
+    """
+    Answer a question based on the most recent frame, record it as a snapshot, and return the answer.
+
+    Args:
+        question (str): The question to be answered.
+    Returns:
+        str:
+            The answer to the question based on the current view (i.e., the latest frame).
+    """
+    mem = wrapper.context
+    prompt = f"Answer the question based on the image. Question: {question}"
+    result = completion_image([mem.frames[-1]], prompt, env.model_mllm)
+    mem.snapshots.append(Snapshot(sender='qa', data=result))
+    return result
+
+
+@function_tool
+def localize(wrapper: RunContextWrapper[Memory]) -> str:
+    """
+    Localize all objects in the most recent frame
+    Returns:
+        str:
+            The localization result for the current view (i.e., the latest frame).
+            the format is {name:list of bboxes}
+    """
+    mem = wrapper.context
+    frame = mem.frames[-1]
+    _, img = cv2.imencode('.jpg', frame)
+    objxbox = task(env.model_loc, img)
+    mem.snapshots.append(Snapshot(sender='localize', data=image_w_box(frame, objxbox)))
+    return json.dumps(objxbox, indent=2)
+
+
+@function_tool
+def time(wrapper: RunContextWrapper[Memory]) -> str:  
+    """
+    Get the current time, record it as a snapshot, and return the time.
+    Returns:
+        str:
+            The current time.
+    """
+    mem = wrapper.context
+    result = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+    mem.snapshots.append(Snapshot(sender='time', data=result))
+    return result
+
+def sample_frames(mem: Memory, n: int) -> list:
+    """
+    Sample frames from the past n seconds of video.
+    
+    Args:
+        mem (Memory): The memory context containing frames.
+        n (int): Number of seconds to look back for video frames.
+    Returns:
+        list: Sampled frames from the video sequence.
+    """
+    if len(mem.frames) == 0:
+        return []
+    
+    available_frames = min(n * env.fps, len(mem.frames))
+    recent_frames = mem.frames[-available_frames:]
+    sampled_frames = recent_frames[::env.fps // 2]
+    
+    return sampled_frames
+
+@function_tool
+def video_caption(wrapper: RunContextWrapper[Memory], n=2) -> str:
+    """
+    Generate a descriptive caption for a video sequence from the past n seconds of frames.
+    The n is a required parameter that specifies how many seconds of video frames to consider.
+    
+    Args:
+        n (int): Number of seconds to look back for video frames.
+    Returns:
+        str:
+            The generated caption for the video sequence from the past n seconds.
+    """
+    mem = wrapper.context
+    sampled_frames = sample_frames(mem, n)
+    
+    if len(sampled_frames) == 0:
+        return "No frames available for video caption."
+    
+    prompt = "Describe this video sequence focusing on any changes or actions that occur over time."
+    result = completion_image(sampled_frames, prompt, env.model_mllm)
+    mem.snapshots.append(Snapshot(sender='video caption', data=result))
+    return result
+
+@function_tool
+def video_qa(wrapper: RunContextWrapper[Memory], question: str, n=2) -> str:
+    """
+    Answer a question based on a video sequence from the past n seconds of frames.
+    
+    Args:
+        question (str): The question to be answered.
+        n (int): Number of seconds to look back for video frames.
+    Returns:
+        str:
+            The answer to the question based on the video sequence from the past n seconds.
+    """
+    mem = wrapper.context
+    sampled_frames = sample_frames(mem, n)
+    
+    if len(sampled_frames) == 0:
+        return "No frames available for video Q&A."
+    
+    prompt = f"Answer the question based on this video sequence. Question: {question}"
+    result = completion_image(sampled_frames, prompt, env.model_mllm)
+    mem.snapshots.append(Snapshot(sender='video qa', data=result))
+    return result

app/utils.py

@@ -0,0 +1,72 @@
+from PIL import Image
+import base64
+import supervision as sv
+import numpy as np
+import cv2
+colors = sv.ColorPalette.from_hex(
+    [
+        "#a1c9f4",
+        "#ffb482",
+        "#8de5a1",
+        "#ff9f9b",
+        "#d0bbff",
+        "#debb9b",
+        "#fab0e4",
+        "#cfcfcf",
+        "#fffea3",
+        "#b9f2f0",
+        "#a1c9f4",
+        "#ffb482",
+        "#8de5a1",
+        "#ff9f9b",
+        "#d0bbff",
+        "#debb9b",
+        "#fab0e4",
+        "#cfcfcf",
+        "#fffea3",
+        "#b9f2f0",
+    ]
+)
+
+def image_w_box(image,objxbox):
+
+    box_annotator = sv.BoxCornerAnnotator(thickness=10, corner_length=30, color=colors)
+    label_annotator = sv.LabelAnnotator(color=colors)
+    mask_annotator = sv.MaskAnnotator(opacity=0.2, color=colors)
+
+    xyxys = np.array([v for boxes in objxbox.values() for v in boxes])
+    unique_labels = sorted(objxbox.keys())
+    class_id_map = dict(enumerate(unique_labels))
+    labels = [l for l, boxes in objxbox.items() for _ in boxes]
+    class_id = [list(class_id_map.values()).index(label) for label in labels]
+
+    masks = np.zeros((len(xyxys), image.shape[0], image.shape[1]), dtype=bool)
+    for i, (x1, y1, x2, y2) in enumerate(xyxys):
+        masks[i, int(y1):int(y2), int(x1):int(x2)] = labels[i]
+
+    if len(xyxys) == 0:
+        return image
+    detections = sv.Detections(
+        xyxy=xyxys,
+        mask=masks,
+        class_id=np.array(class_id),
+    )
+    # Convert RGB to BGR for annotation
+    image_bgr = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR)
+    # After annotation, convert back to RGB
+    annotated_image = box_annotator.annotate(scene=image_bgr.copy(), detections=detections)
+    annotated_image = label_annotator.annotate(scene=annotated_image, detections=detections, labels=labels)
+    annotated_image = mask_annotator.annotate(scene=annotated_image, detections=detections)
+
+    return cv2.cvtColor(annotated_image, cv2.COLOR_BGR2RGB)
+
+
+def encode_image(img) -> tuple[str, str]:
+    arr = np.array(img.convert("RGB")) if isinstance(img, Image.Image) else img
+    if not isinstance(arr, np.ndarray):
+        raise ValueError("Unsupported image type")
+    ok, buf = cv2.imencode('.jpg', cv2.cvtColor(arr, cv2.COLOR_RGB2BGR))
+    if not ok:
+        raise ValueError("Encoding failed")
+    b64 = base64.b64encode(buf).decode('utf-8')
+    return b64, "image/jpeg"

main.py

@@ -0,0 +1,196 @@
+from pathlib import Path
+import os
+import cv2
+import gradio as gr
+from fastrtc import WebRTC
+from app.config import env
+from fastrtc import AdditionalOutputs
+from app.memory import Memory,Message
+from fastrtc import get_cloudflare_turn_credentials
+from app.agent import build_agent
+from fastrtc import get_current_context
+session_memories = {}
+
+def get_session_memory(session_id: str = None) -> Memory:
+    if session_id not in session_memories:
+        session_memories[session_id] = Memory(build_agent())
+        welcome_message = "👋 Now I can see. Feel free to ask me about anything!"
+        session_memories[session_id].chat.append(Message.assistant(welcome_message))
+    return session_memories[session_id]
+
+def video_handler(frame):
+    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
+    rtcid = get_current_context().webrtc_id
+    mem = get_session_memory(rtcid)
+    if (s := mem.enqueue(frame)):
+        if mem.chat.history[-1].metadata.get('status') == 'pending':
+            mem.chat.history[-1] = Message.tool(s.gr, title=s.sender, status=s.status)
+        else:
+            mem.chat.append(Message.tool(s.gr, title=s.sender, status=s.status))
+    return frame, AdditionalOutputs(mem.chat.messages, rtcid)
+
+def chat_handler(text, webrtc_state):
+    if webrtc_state is None:
+        return "", [{"role": "assistant", "content": "Please start your camera first to begin the conversation."}], webrtc_state
+    
+    mem = get_session_memory(webrtc_state)
+    if not mem.is_running:
+        mem.receive(text.strip())
+    return "", mem.chat.messages, webrtc_state
+
+
+
+
+
+if __name__ == "__main__":
+    print("🚀 Starting Perceptual Copilot...")
+    print(f"HF Spaces: {os.getenv('SPACE_ID') is not None}")
+    print(f"Environment check - API_KEY: {'✓' if env.api_key else '✗'}")
+    print(f"Environment check - END_LANG: {'✓' if env.end_lang else '✗'}")
+    print(f"Environment check - OpenAI Client: {'✓' if env.client else '✗'}")
+    
+    
+
+    with gr.Blocks(
+        title="🤖 Perceptual Copilot - AI Vision Assistant", 
+        theme=gr.themes.Soft(
+            primary_hue="blue",
+            secondary_hue="orange", 
+            neutral_hue="slate",
+            font=("system-ui", "sans-serif")
+        ),
+        css=Path("styles.css").read_text(),
+    ) as demo:
+        # Header section with project context
+        gr.Markdown(
+            """
+        <div class="ultra-sleek-header">
+            <h1 class="hero-title">
+                <span class="title-primary">Perceptual</span>
+                <span class="title-accent">Copilot</span>
+            </h1>
+            <p class="hero-subtitle">
+                <span class="status-dot"></span>
+                A visual agent system that integrates OpenAI agents with visual tools to process video streams.
+            </p>
+            <div class="hero-actions">
+                <a class="hero-link hero-link-primary" href="https://proj-robot.jing.vision/" target="_blank" rel="noopener noreferrer">See It in Action</a>
+            </div>
+            <div class="feature-pills">
+                <span class="pill">Embodied agent core</span>
+                <span class="pill">Real-time streaming</span>
+                <span class="pill">Large vision language model</span>
+                <span class="pill">Reasoning</span>
+            </div>
+        </div>
+        """,
+            elem_classes="ultra-sleek-header",
+        )
+        
+        state = gr.State(value=None)
+        
+        # Main interface with improved layout
+        with gr.Row(equal_height=True):
+            with gr.Column(scale=1, elem_classes="video-container"):
+                video = WebRTC(
+                    label="🎥 Camera Stream",
+                    rtc_configuration=get_cloudflare_turn_credentials(hf_token=env.hf_token),
+                    track_constraints={
+                        "width": {"exact": 600}, 
+                        "height": {"exact": 600}, 
+                        "aspectRatio": {"exact": 1}},
+                    mode="send",
+                    modality="video",
+                    mirror_webcam=True,
+                    width=600,
+                    height=600,
+                    full_screen=False,
+                )
+            
+            with gr.Column(scale=1, elem_classes="chat-container"):
+                gr.Markdown(
+                    """
+                    <div class="chat-header">
+                        <h3>💬 Chat</h3>
+                    </div>
+                    """
+                )
+                chatbot = gr.Chatbot(
+                    type="messages", 
+                    height=450,
+                    label="🤖 AI Assistant",
+                    placeholder="Chat history will appear here...",
+                    show_label=False,
+                )
+                
+                with gr.Row(elem_classes="items-center"):
+                    textbox = gr.Textbox(
+                        placeholder="💭 Type a question and press Enter to send.",
+                        lines=1,
+                        show_label=False,
+                    )
+                gr.Markdown(
+                    """
+                    <div class="latency-note latency-note-inline">
+                        We self-host our models, so responses may take slightly longer.
+                    </div>
+                    """
+                )
+        # Event handlers
+        video.stream(
+            fn=video_handler,
+            inputs=[video],
+            outputs=[video],
+            concurrency_limit=10,
+        )
+        video.on_additional_outputs(
+            fn=lambda messages, webrtc_id: (messages, webrtc_id),
+            outputs=[chatbot, state]
+        )
+        
+        # Chat handler for textbox
+        textbox.submit(
+            chat_handler,
+            inputs=[textbox, state],
+            outputs=[textbox, chatbot, state]
+        )
+        
+        # Enhanced instructions section
+        with gr.Column(elem_classes="instructions-container"):
+            gr.Markdown("""
+            <div class="info-grid">
+                <section class="info-card info-card-wide">
+                    <div class="info-label">The Core Layer</div>
+                    <p>
+                        Perceptual Copilot is the core perception and agent layer behind our embodied agent system, built for <a class="marker-link" href="https://proj-robot.jing.vision/" target="_blank" rel="noopener noreferrer">real-world robot deployment</a>.
+                        This is a lightweight live preview of that larger system, focused on the core agent and its visual capabilities.
+                    </p>
+                </section>
+                <section class="info-card">
+                    <div class="info-label">Quick Start</div>
+                    <ul>
+                        <li>Start the camera and grant browser access.</li>
+                        <li>Wait for the assistant’s welcome message.</li>
+                        <li>Ask questions about what is in view of the camera.</li>
+                    </ul>
+                </section>
+                <section class="info-card">
+                    <div class="info-label">Try Asking</div>
+                    <ul>
+                        <li>"What do you see in front of me?"</li>
+                        <li>"Read the text in this image."</li>
+                        <li>"What changed in the last few seconds?"</li>
+                    </ul>
+                </section>
+                <section class="info-card">
+                    <div class="info-label">In This Demo</div>
+                    <ul>
+                        <li>Captioning and open visual Q&amp;A</li>
+                        <li>OCR on live camera frames</li>
+                        <li>Short-window video reasoning</li>
+                    </ul>
+                </section>
+            </div>
+            """)
+    demo.queue(default_concurrency_limit=None)
+    demo.launch()

requirements.txt

@@ -0,0 +1,7 @@
+huggingface_hub
+python-dotenv
+supervision
+openai-agents
+fastrtc
+gradio
+pydantic

styles.css

@@ -0,0 +1,525 @@
+:root {
+    --page-bg: #ecebe5;
+    --page-bg-soft: #f6f5f1;
+    --surface: rgba(255, 255, 255, 0.84);
+    --surface-strong: rgba(255, 255, 255, 0.92);
+    --surface-border: rgba(16, 19, 26, 0.07);
+    --text: #10131a;
+    --muted: #5f6776;
+    --royal-blue: #3f63d8;
+    --royal-blue-deep: #2f4fbe;
+    --periwinkle: #7f8af8;
+    --shadow: 0 14px 34px rgba(16, 19, 26, 0.045);
+    --shadow-soft: 0 8px 22px rgba(16, 19, 26, 0.03);
+}
+.gradio-container {
+    background: linear-gradient(180deg, var(--page-bg-soft) 0%, var(--page-bg) 100%) !important;
+        color: var(--text);
+        font-family: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
+}
+
+.video-container,
+.chat-container,
+.instructions-container {
+    background: var(--surface);
+    border: 1px solid var(--surface-border);
+    border-radius: 22px;
+    padding: 22px;
+    box-shadow: var(--shadow);
+    backdrop-filter: blur(10px);
+}
+
+.chat-container textarea {
+    border: 1.5px solid rgba(65, 105, 225, 0.14) !important;
+        border-radius: 16px !important;
+        padding: 15px 16px !important;
+    font-size: 16px !important;
+    font-weight: 500 !important;
+    background: rgba(255, 255, 255, 0.98) !important;
+        box-shadow: inset 0 1px 0 rgba(255, 255, 255, 0.85), 0 6px 18px rgba(16, 19, 26, 0.03) !important;
+        transition: border-color 0.2s ease, box-shadow 0.2s ease, transform 0.2s ease !important;
+    min-height: 50px !important;
+    resize: none !important;
+}
+
+.chat-container textarea:focus {
+    border-color: var(--royal-blue) !important;
+        box-shadow: 0 0 0 4px rgba(65, 105, 225, 0.1) !important;
+    outline: none !important;
+    transform: translateY(-1px) !important;
+}
+
+.chat-container textarea::placeholder {
+    color: #6b7280 !important;
+        font-size: 15px !important;
+    font-weight: 400 !important;
+}
+
+.chat-container .gr-text-input {
+    border: none !important;
+    background: transparent !important;
+}
+
+.latency-note {
+    display: block;
+    margin: 0;
+    padding: 0;
+    border: 0;
+    background: transparent;
+    color: #7b8290;
+    font-size: 0.74rem;
+    line-height: 1.35;
+    letter-spacing: 0.01em;
+    white-space: normal;
+}
+
+.latency-note-inline {
+    width: 100%;
+    margin-top: 10px;
+    text-align: center;
+}
+.instructions-container {
+    margin: 20px 0;
+    background: rgba(255, 255, 255, 0.86);
+}
+
+.instructions-container a {
+    color: var(--royal-blue);
+    font-weight: 700;
+    text-decoration: none;
+}
+
+.instructions-container a:hover {
+    text-decoration: underline;
+}
+
+/* Ultra-sleek header styling with modern design */
+.ultra-sleek-header {
+    text-align: center;
+    padding: 20px 20px 5px 20px;
+    margin: 0;
+    background: transparent;
+    border: none;
+    position: relative;
+        overflow: hidden;
+}
+
+.ultra-sleek-header::before {
+    content: '';
+    position: absolute;
+    top: 0;
+    left: 0;
+    right: 0;
+    bottom: 0;
+    background: radial-gradient(ellipse at center top, rgba(99, 102, 241, 0.05) 0%, transparent 70%);
+    pointer-events: none;
+}
+
+
+.badge-icon {
+    font-size: 1rem;
+    animation: float 3s ease-in-out infinite;
+}
+
+@keyframes float {
+
+    0%,
+    100% {
+        transform: translateY(0px);
+    }
+
+    50% {
+        transform: translateY(-4px);
+    }
+}
+
+.hero-title {
+    margin: 8px 0 6px;
+    font-size: 4rem;
+    line-height: 1;
+        letter-spacing: -0.05em;
+    font-weight: 800;
+    color: var(--text);
+        text-shadow: 0 10px 28px rgba(114, 89, 255, 0.08);
+    }
+.title-primary {
+    background: linear-gradient(45deg,
+            #4169E1 0%,
+                #8A2BE2 50%,
+                #E91E63 100%);
+    -webkit-background-clip: text;
+    -webkit-text-fill-color: transparent;
+    background-clip: text;
+    position: relative;
+}
+
+.title-accent {
+    display: inline-block;
+        color: #132542;
+        margin-left: 10px;
+    }
+    
+    @supports ((-webkit-background-clip: text) or (background-clip: text)) {
+        .title-primary {
+            background: linear-gradient(90deg, #4f6ef7 0%, #7a43ee 46%, #e21c74 100%);
+            -webkit-background-clip: text;
+            -webkit-text-fill-color: transparent;
+            background-clip: text;
+        color: transparent;
+        }
+        }
+                                .title-primary::after {
+                                    content: "";
+                                    position: absolute;
+                                    left: 0;
+                                    right: 0;
+                                    bottom: -12px;
+                                    height: 6px;
+                                    border-radius: 999px;
+                                    background: linear-gradient(90deg, rgba(135, 139, 255, 0.9), rgba(221, 123, 255, 0.85));
+                                    box-shadow: 0 8px 22px rgba(165, 105, 255, 0.18);
+                                }
+
+.hero-subtitle {
+    margin: 14px auto 0;
+        max-width: 560px;
+        display: flex;
+        align-items: center;
+        justify-content: center;
+    gap: 10px;
+        color: var(--muted);
+        font-size: 1.12rem;
+        font-weight: 500;
+        line-height: 1.6;
+}
+
+.status-dot {
+    display: inline-block;
+    width: 8px;
+    height: 8px;
+    border-radius: 999px;
+        background: #18b77f;
+    position: relative;
+    flex-shrink: 0;
+}
+
+.status-dot::before {
+    content: "";
+    position: absolute;
+    inset: 0;
+        border-radius: inherit;
+        background: #18b77f;
+    animation: ping 2s cubic-bezier(0, 0, 0.2, 1) infinite;
+}
+
+@keyframes ping {
+    75%,
+        100% {
+            transform: scale(2);
+            opacity: 0;
+        }
+    }
+.hero-actions {
+    display: flex;
+    justify-content: center;
+    align-items: center;
+    gap: 12px;
+    margin-top: 18px;
+}
+
+.hero-link {
+    display: inline-flex;
+    align-items: center;
+    justify-content: center;
+    padding: 12px 20px;
+    border-radius: 999px;
+    text-decoration: none !important;
+    font-weight: 800;
+    letter-spacing: -0.01em;
+    border: 1px solid transparent;
+}
+
+.hero-link-primary {
+    position: relative;
+    overflow: hidden;
+    background: linear-gradient(135deg, var(--royal-blue), var(--royal-blue-deep) 62%, var(--periwinkle) 100%);
+    color: #ffffff !important;
+    box-shadow: 0 16px 34px rgba(65, 105, 225, 0.26);
+    animation: buttonPulse 2.8s ease-in-out infinite;
+}
+
+.hero-link-primary::after {
+    content: "";
+    position: absolute;
+    top: 0;
+    left: -45%;
+    width: 34%;
+    height: 100%;
+    background: linear-gradient(90deg, transparent, rgba(255, 255, 255, 0.28), transparent);
+    transform: skewX(-20deg);
+    animation: buttonSheen 3.2s ease-in-out infinite;
+}
+
+.hero-link-primary:hover {
+    transform: translateY(-1px);
+    box-shadow: 0 20px 38px rgba(65, 105, 225, 0.3);
+}
+
+@keyframes buttonPulse {
+
+    0%,
+        100% {
+            transform: translateY(0);
+            box-shadow: 0 16px 34px rgba(65, 105, 225, 0.26);
+        }
+    
+        50% {
+            transform: translateY(-1px);
+            box-shadow: 0 20px 40px rgba(65, 105, 225, 0.34);
+        }
+    }
+    
+    @keyframes buttonSheen {
+    
+        0%,
+        55% {
+            left: -45%;
+        }
+    
+        82%,
+        100% {
+            left: 125%;
+        }
+    }
+.feature-pills {
+    display: flex;
+        justify-content: center;
+        gap: 10px;
+            margin-top: 18px;
+        flex-wrap: wrap;
+    }
+    
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                .pill {
+    background: rgba(255, 255, 255, 0.66);
+        border: 1px solid rgba(16, 19, 26, 0.06);
+        border-radius: 999px;
+        padding: 7px 14px;
+        font-size: 0.82rem;
+        font-weight: 700;
+        color: #536071;
+        box-shadow: none;
+    }
+    
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                .info-grid {
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    display: grid;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    grid-template-columns: repeat(3, minmax(0, 1fr));
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    gap: 0;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    margin-top: 6px;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    border: 1px solid rgba(16, 19, 26, 0.09);
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    border-radius: 24px;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    overflow: hidden;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    background:
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        linear-gradient(90deg, rgba(16, 19, 26, 0.05) 1px, transparent 1px) 0 0 / 33.333% 100%,
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        linear-gradient(180deg, rgba(16, 19, 26, 0.04) 1px, transparent 1px) 0 0 / 100% 50%,
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        rgba(255, 255, 255, 0.94);
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    box-shadow: 0 20px 44px rgba(16, 19, 26, 0.035);
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                }
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                .info-card {
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    background: transparent;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    border: 0;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    border-right: 1px solid rgba(16, 19, 26, 0.09);
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    padding: 30px 36px 32px;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    box-shadow: none;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    border-radius: 0;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                }
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                .info-card-wide {
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    grid-column: span 3;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    border-right: 0;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    border-bottom: 1px solid rgba(16, 19, 26, 0.09);
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    background: linear-gradient(180deg, rgba(255, 255, 255, 0.92) 0%, rgba(249, 250, 253, 0.96) 100%);
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                }
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                .info-label {
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    margin-bottom: 12px;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    color: #4268f5;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    font-size: 0.74rem;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    font-weight: 800;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    letter-spacing: 0.18em;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    text-transform: uppercase;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                }
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                .info-card p,
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                .info-card li {
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    color: #666d7b;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    font-size: 1rem;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    line-height: 1.62;
+        }
+    
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                .info-card p {
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    margin: 0;
+        }
+    
+.info-card-wide p {
+    width: 100%;
+    max-width: none;
+    font-size: 1.18rem;
+    line-height: 1.72;
+    color: #4f5666;
+    text-wrap: pretty;
+}
+.info-card ul {
+    margin: 0;
+    padding: 0;
+    list-style: none;
+}
+.info-card li+li {
+    margin-top: 10px;
+}
+
+.info-card li {
+    position: relative;
+    padding-left: 16px;
+}
+
+.info-card li::before {
+    content: "";
+    position: absolute;
+    left: 0;
+    top: 0.7em;
+    width: 5px;
+    height: 5px;
+    border-radius: 999px;
+    background: rgba(66, 104, 245, 0.5);
+}
+
+.marker-link {
+    position: relative;
+    display: inline;
+    margin: 0;
+    padding: 0 0.02em 0.22em;
+    color: #3a5cf3 !important;
+    font-weight: 700;
+    text-decoration: none !important;
+    background-image: url("data:image/svg+xml,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%20220%2026%22%20preserveAspectRatio%3D%22none%22%3E%3Cpath%20d%3D%22M0%2013%20C%2018%203%2036%2023%2055%2013%20S%2092%2023%20110%2013%20S%20148%203%20165%2013%20S%20202%2023%20220%2013%22%20fill%3D%22none%22%20stroke%3D%22%232145de%22%20stroke-width%3D%224.2%22%20stroke-linecap%3D%22round%22%20stroke-linejoin%3D%22round%22%2F%3E%3Cpath%20d%3D%22M0%2016%20C%2020%208%2035%2024%2055%2016%20S%2090%208%20110%2016%20S%20145%2024%20165%2016%20S%20200%208%20220%2016%22%20fill%3D%22none%22%20stroke%3D%22%235070ff%22%20stroke-width%3D%222.6%22%20stroke-linecap%3D%22round%22%20stroke-linejoin%3D%22round%22%20stroke-opacity%3D%22.82%22%2F%3E%3C%2Fsvg%3E");
+    background-repeat: no-repeat;
+    background-position: 0 125%;
+    background-size: 100% 0.52em;
+    -webkit-box-decoration-break: clone;
+    box-decoration-break: clone;
+    transition: color 0.22s ease, background-size 0.22s ease, text-shadow 0.22s ease;
+}
+
+.marker-link:hover,
+.marker-link:focus-visible {
+    color: #133ad4 !important;
+    background-size: 100% 0.7em;
+    text-shadow: 0 0 16px rgba(80, 112, 255, 0.12);
+}
+
+.marker-link:hover::after,
+.marker-link:focus-visible::after {
+    opacity: 1;
+    transform: translateY(-10%);
+}
+
+.marker-link:focus-visible {
+    outline: none;
+}
+
+.info-card:nth-child(2),
+.info-card:nth-child(3) {
+    border-bottom: 0;
+}
+.info-card:nth-child(4) {
+    border-right: 0;
+}
+.items-center {
+    display: flex !important;
+    align-items: center !important;
+    gap: 8px !important;
+}
+.gradio-container footer,
+.gradio-container .api-docs,
+.gradio-container .built-with,
+.gradio-container .settings,
+.gradio-container .footer,
+.gradio-container div[data-testid="footer"],
+.gradio-container .gradio-footer {
+    display: none !important;
+}
+
+.gradio-container a[href*="gradio"] {
+    display: none !important;
+}
+
+.gradio-container .chatbot .message-wrap button[aria-label*="clear" i],
+.gradio-container .chatbot .message-wrap button[title*="clear" i],
+.chat-container button[aria-label*="clear" i],
+.chatbot button[aria-label*="clear" i],
+.message-wrap button[aria-label*="clear" i] {
+    display: none !important;
+}
+
+button:has(svg[data-testid*="clear"]),
+button:has(svg[data-testid*="delete"]),
+button:has(svg[data-testid*="trash"]) {
+    display: none !important;
+}
+@media (max-width: 768px) {
+    .ultra-sleek-header {
+        padding: 18px 14px 12px;
+    }
+
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                .latency-note {
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    white-space: normal;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                }
+    .hero-title {
+        font-size: 3rem;
+    
+
+        .hero-subtitle {
+            flex-direction: column;
+            gap: 8px;
+            padding: 0 12px;
+            font-size: 1.02rem;
+        }
+
+        .video-container,
+        .chat-container,
+        .instructions-container {
+            padding: 18px;
+            border-radius: 20px;
+        }
+
+        .info-grid {
+            grid-template-columns: 1fr;
+            background: rgba(255, 255, 255, 0.95);
+        }
+
+        .info-card-wide {
+            grid-column: span 1;
+        }
+
+        .info-card,
+        .info-card-wide {
+            border-right: 0;
+            border-bottom: 1px solid rgba(16, 19, 26, 0.08);
+            padding: 22px 20px 24px;
+    
+
+        .info-card:last-child {
+                border-bottom: 0;
+            }
+        }
+
+@media (max-width: 480px) {
+    .hero-title {
+        font-size: 2.5rem;
+        line-height: 1.08;
+    }
+
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                .title-accent {
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    display: block;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    margin: 4px 0 0;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                }
+        }