Agent Harness
An agent harness built from scratch to learn how production agents work: the loop, context, memory, tool calling, skill retrieval and model routing.
- ownership
- Independent project
- when
- Jul 2026 - present
- org
- Independent
The film, as text: Agent Harness. An agent harness built from scratch to learn how production agents work: the loop, context, memory, tool calling, skill retrieval and model routing. Context assembly: Each step sees the context it needs: the request, state, skills and prior results. Skill discovery: The right skill for a request is retrieved from the skill library. Tool selection: The harness selects a tool, runs it, and feeds the result back. The agentic loop: The loop continues until the task completes, or retries with what it learned. Built from scratch: Built from scratch: the loop, context, memory, MCP tools, skills and model routing. Independent project, Independent, Jul 2026 - present.
Problem.
Agent frameworks hide the loop, the context, the memory and the tool choice, which are the parts that decide whether an agent is reliable. Building the harness directly makes each of them explicit.
How it works
- Agent loop: planning, reasoning and execution loops with explicit state management, so every step sees the context it needs and decides whether to act again or return.
- Context engineering: structured context is assembled for each step from task state, memory and retrieved skills, so the model sees only what the step needs.
- Persistent memory: memory, task state and contextual recall carry across sessions.
- Tool calling and execution: a skill and tool system built on MCP, with tool discovery, selection and execution, and integration with external APIs.
- Skill discovery and retrieval: embedding-based retrieval over skills and tools with Qdrant and ChromaDB, scored for relevance, selects capabilities dynamically instead of loading a fixed tool set on every task.
- Model routing: configurable agent execution across local and hosted models, including Gemini, OpenAI, NVIDIA, OpenRouter and local models through Ollama.
Outcome
The agent engine behind Jarvis.
Stack
- Agent
- Planning and execution loops
- Context and state management
- Persistent memory
- MCP
- Retrieval
- Embedding-based skill retrieval
- Qdrant
- ChromaDB
- Models
- Model routing
- Ollama
- Gemini
- OpenAI
- NVIDIA
- OpenRouter
- Backend
- Python
- FastAPI
- SQLite
- Docker