Creating Local AI Agents with Foundry Local and Qwen

Companion skill for Lesson 17 – Creating Local AI Agents. Use it to help a learner build an agent that reasons, calls tools, and searches documentation entirely on their own machine — no cloud inference. Ground every recommendation in the lesson content and the runnable notebook.

Triggers

Activate this skill when a learner wants to:

• Run an agent fully on-device for privacy, cost, or offline reasons.
• Serve a model locally with Foundry Local and connect via the OpenAI-compatible endpoint.
• Use a Qwen function-calling model to drive reliable local tool calls.
• Add local RAG (Chroma) or a local MCP server.
• Design a hybrid local/cloud routing strategy.

Core mental model

An SLM trades breadth for privacy, cost, and offline operation. The winning strategy: let the SLM orchestrate and let tools do the heavy lifting. The model does not need to know the codebase — it needs to know when to call read_file and search_docs. That plays to an SLM's strength (bounded decisions like tool selection) and away from its weakness (broad knowledge, long multi-hop reasoning).

Why these specific pieces

• Foundry Local exposes an OpenAI-compatible HTTP endpoint, so cloud agent code transfers by changing only base_url (and using a local placeholder API key). It also auto-selects the best build (CPU/GPU/NPU) for the machine.
• Qwen models are trained for function calling and emit well-formed tool calls consistently — this is what turns a local chat model into a local agent.
• Chroma runs in-process and stores vectors on disk, so the whole RAG pipeline (embed → store → retrieve → reason) stays local.
• MCP is a transport, not a cloud service: an MCP server can run locally over stdio.

Setup essentials

foundry model run qwen2.5-7b-instruct
foundry service status

from foundry_local import FoundryLocalManager
from openai import OpenAI

manager = FoundryLocalManager("qwen2.5-7b-instruct")
client = OpenAI(base_url=manager.endpoint, api_key=manager.api_key)  # local placeholder

~8 GB RAM is a realistic minimum; a GPU/NPU helps but is not required.

Key patterns to reproduce

Point the learner at the notebook 17-local-agent-foundry-local.ipynb:

• Sandboxed tools: every file tool resolves paths and rejects anything outside a single project root — even locally, a tool runs with the user's permissions.
• Tool-calling loop: register tools with the OpenAI tools schema, execute requested tools locally, feed results back, repeat until a final answer.
• Local RAG: upsert docs into a Chroma collection; search_docs returns top-k chunks.
• Local MCP: connect to a local server over stdio; scope it to a project directory and validate its outputs.

Hybrid routing (local as one of the models)

Situation	Where it runs
Sensitive data / offline	Local SLM
Simple, bounded task	Local SLM (cheap, fast)
Hard multi-hop reasoning on non-sensitive data	Cloud model
Cloud outage	Local SLM (graceful degradation)

This mirrors the model-routing idea from Lesson 16, with the workstation as one of the routes. Prefer designs that fall back to local so the agent degrades in quality rather than failing outright.

Guardrails for the assistant

• Keep every file/tool operation scoped to a sandboxed project directory.
• Do not send code or data to the cloud when the learner's stated goal is privacy/offline — keep the whole pipeline local.
• Set realistic expectations for SLM quality; lean on tools and RAG rather than the model's memorised knowledge.
• Note that Lesson 17 has no Foundry Responses endpoint, so the cloud smoke-test action does not apply — validate by running the notebook locally.

---

Disclaimer: This document has been translated using AI translation service Co-op Translator. While we strive for accuracy, please be aware that automated translations may contain errors or inaccuracies. The original document in its native language should be considered the authoritative source. For critical information, professional human translation is recommended. We are not liable for any misunderstandings or misinterpretations arising from the use of this translation.