Jarvis
A personal multi-agent assistant that uses tools, remembers context and acts on the computer through voice and application control, built on the Agent Harness project.
- ownership
- Independent project
- when
- Jul 2026 - present
- org
- Independent
The film, as text: Jarvis. A personal multi-agent assistant that uses tools, remembers context and acts on the computer through voice and application control, built on the Agent Harness project. Voice to plan: A spoken request is transcribed and planned into steps. Markdown memory: Memories, preferences and tasks live in linked Markdown files, recalled across sessions. Skills and MCP tools: Skills and tools are retrieved by embedding search and run through MCP. Computer control: A retrieved tool acts on the computer: here, it opens a project folder. Model routing: Steps route between local and hosted models. Write back: The reply is spoken, and what happened is written back to memory. Independent project, Independent, Jul 2026 - present.
How it works
- Built on the Agent Harness: planning, execution loops, tool calling, skill retrieval and model routing come from the from-scratch harness rather than a prebuilt agent framework.
- Persistent Markdown memory: memories, preferences, tasks and past interactions are stored as linked files, with a web-based knowledge graph that visualises their relationships and recalls context across sessions.
- Skills and tools over MCP: tools are discovered, selected and executed through MCP, with embedding-based skill and tool retrieval on Qdrant and ChromaDB, alongside external APIs.
- Computer control: application actions and filesystem operations.
- Voice interaction: speech-to-text and text-to-speech.
- Model routing: local and hosted models, including Gemini, OpenAI, NVIDIA, OpenRouter and local models through Ollama.
- Full-stack application: a Python and FastAPI backend with a Next.js, React and TypeScript frontend.
Stack
- Agents
- MCP
- Persistent Markdown memory
- Knowledge graph
- Computer control
- Retrieval
- Embedding-based skill retrieval
- Qdrant
- ChromaDB
- Models
- Ollama
- Gemini
- OpenAI
- NVIDIA
- OpenRouter
- Voice
- Speech-to-text
- Text-to-speech
- Application
- Python
- FastAPI
- Next.js
- React
- TypeScript
- SQLite
- Docker