Use Copilot CLI with local LLMs

AI coding agents are more than sophisticated autocomplete tools. With access to a repository, a terminal, development utilities, and structured instructions, they can participate in the entire software development lifecycle: from exploring a problem and designing a solution to implementing changes and verifying the results.

However, giving an AI agent access to a codebase does not automatically make it a reliable engineering partner. Without clear constraints, access to real feedback, and a structured workflow, it can produce plausible but incorrect code, overlook project conventions, or gradually drift away from the original requirements.

A more robust approach combines four complementary elements: local model inference → project-specific instructions → tool integration through MCP → Spec-Driven Development (SDD).

Together, these elements provide a practical foundation for building AI-assisted development workflows that are more predictable, auditable, and cost-effective.

1. Running GitHub Copilot CLI with Local Language Models

GitHub Copilot CLI provides a terminal-based environment where an AI agent can interact with a development workspace. Instead of relying exclusively on a hosted model, it can also be configured to use a locally hosted language model through a compatible API endpoint.

This architecture separates the agent interface from the model infrastructure. Copilot CLI provides the interaction and orchestration layer, while the local inference server handles model requests.

Choosing an inference runtime

Two practical options are vLLM and llama.cpp.

vLLM is particularly suitable for systems with capable GPUs and sufficient memory. It provides an OpenAI-compatible API and supports model serving features such as structured tool calling. vLLM ist best "on the server side" serving multiple users. Yes, it get's faster (!) when talking to several users instead to a single user.
llama.cpp offers a more flexible option for consumer hardware, including CPU-based inference and GPU acceleration. Its support for quantized GGUF models makes it possible to run coding models with relatively modest hardware. That's the one Dev workstation setup.

The appropriate model depends on available memory, response-time requirements, context length, and coding capabilities. Smaller, quantized models can be useful for routine development tasks, while larger models may provide better results for complex reasoning and multi-step agent workflows.

Model size alone is not a reliable measure of usefulness. Don't think like Donald Trump "big is better" (and gold is the best). No, it isn't! A smaller coding-oriented model that responds quickly and handles tool calls correctly may be apparently more effective than a larger model that struggles with latency or structured output.

Don't lough at local models in the range of 9-14 B parameters! They are well-suited for common repetative tasks. And leave much room for the context window. Yes you can run a 27B or even 35B model on a GPU with 16GB VRAM with some tweaking - but then you end up just with 16k context window instead of 260k. And when it comes down to "agentic", the available context is what matters the most. Or you may be a millionair and add a RTX 6000 Pro to your workstation ;-)

Connecting Copilot CLI to the local endpoint

Copilot CLI supports a Bring Your Own Key config. For a local server, the essential environment variables are:

export COPILOT_PROVIDER_TYPE="openai"
export COPILOT_PROVIDER_BASE_URL="http://localhost:1234/v1"
export COPILOT_PROVIDER_API_KEY="dummy"
export COPILOT_MODEL="unsloth/Qwen3.6-35B-A3B-GGUF"
export COPILOT_OFFLINE="true"

The endpoint and model name must match your local inference server's configuration. In particular, COPILOT_MODEL must correspond to the model identifier exposed by the server. One of the most important configs here is indeed

COPILOT_OFFLINE="true"

otherwise Copilot CLI would still use cloud models.

With vLLM, tool calling requires additional configuration. Options such as --enable-auto-tool-choice and the appropriate --tool-call-parser allow the server to interpret structured tool requests rather than returning them as ordinary text.

Before debugging the agent itself, verify that the model server is available and exposes the expected model identifier. This helps distinguish connectivity and configuration problems from model-level issues.

Local inference offers greater control over data processing, model selection, and marginal inference costs. It can also make extensive experimentation more affordable, particularly when a workflow involves repeated planning, implementation, and verification.

There are trade-offs, however. Local models may have weaker reasoning capabilities, shorter practical context windows, or less reliable tool calling than leading hosted models. Running in fully offline mode can also disable GitHub-hosted integrations*.

* Which of course is only partially ("out of the box") true. You cann add GitHub MCP server and install the GitHub CLI (gh), provide your GitHub bearer token and here we go again!

2. Treat Instructions, Agents, and Skills as Engineering Configuration

A model needs more than access to source code. It needs to understand how that codebase is organized, which conventions it must follow, and what constraints govern its changes.

This becomes particularly important with local models, which may have less capacity to infer undocumented project conventions from limited context.

The solution is to make the project's expectations explicit and version-controlled.

Repository-wide instructions

A .github/copilot-instructions.md file can define the shared context that should guide the agent throughout its work.

Useful information includes:

Project architecture and directory structure.
Programming languages, frameworks, and coding conventions.
Build, test, lint, and validation commands.
Important data schemas and interface contracts.
Architectural constraints and prohibited changes.

Instructions should be specific and actionable. For example, "run the unit tests after modifying the authentication module" is more useful than "make sure the code is correct."

Keep this file focused. Because general instructions are repeatedly supplied as context, unnecessary detail consumes space that could otherwise be used for the actual task.

Path-specific instructions

Not every directory needs the same rules. Files under a UI component directory may follow different conventions than database migrations, API handlers, or infrastructure definitions.

Path-specific instructions can be placed under .github/instructions/ and associated with relevant files using an applyTo glob pattern.

This makes it possible to enforce rules where they belong without overwhelming every interaction with the entire project's conventions.

Agent definitions with `AGENTS.md`

AGENTS.md files can define responsibilities and boundaries for particular areas of a repository.

For example, an agent responsible for database migrations might be instructed to create migrations and tests but never modify application deployment configuration. A frontend-focused agent might be restricted to UI components and their associated tests.

These boundaries help separate responsibilities and reduce unintended changes. They are especially useful in large repositories or workflows involving several specialized agents.

Skills for repeatable workflows

Instructions explain what the agent should know and how it should behave. Skills describe how it should execute a particular process.

A skill typically consists of a SKILL.md file defining a workflow, its required inputs, the tools to use, the expected outputs, and the completion criteria.

For example, a feature-development skill could require the agent to inspect the relevant code, produce a design proposal, generate implementation tasks, apply the changes, run tests, and report any unresolved issues.

The important distinction is that a skill turns a general request into a repeatable procedure.

Together, these mechanisms provide a useful control hierarchy:

Instructions establish project-wide conventions and constraints.
Path-specific instructions apply rules to relevant files.
Agent definitions establish responsibilities and boundaries.
Skills define structured, reusable workflows.
MCP tools provide access to external actions and verifiable results.

This control layer is not merely documentation for the AI. It is part of the development infrastructure.

3. Use MCP to Give Agents Real Development Capabilities

Even a capable language model cannot reliably verify its assumptions using generated text alone. It needs access to the actual development environment.

The Model Context Protocol (MCP) provides a standardized way for AI applications to discover and invoke external tools. An MCP server exposes tools with defined interfaces, allowing an agent to request operations and receive their results in its context.

Instead of expecting a model to predict whether its code compiles, for example, an agent can run the project's compiler or test suite and inspect the actual output.

Useful MCP integrations

Several categories of tools can substantially improve development workflows.

Test runners: Allow the agent to execute unit, integration, or targeted regression tests after making changes. Test failures become actionable feedback rather than something the model must guess.

Linters and formatters: Expose existing code-quality tools, such as ESLint, Ruff, or other project-specific utilities. The agent can check its changes against established conventions and address reported problems.

Codebase search: Provide fast, targeted access to relevant files and symbols. Semantic search can retrieve useful code snippets based on meaning, reducing the need to load an entire repository into the context window.

Security and policy checks: Enable checks for exposed secrets, prohibited patterns, license requirements, or other project policies. These checks are particularly valuable because they can apply deterministic rules independently of the model's reasoning.

MCP servers can be configured globally or for an individual project. Global tools are useful for common development utilities, while project-specific servers can expose domain knowledge and operations tailored to a particular codebase.

Build a feedback loop, not just a toolbox

The real value of MCP comes from integrating tool use into the development process.

A typical agent loop might look like this:

Search the repository for the relevant implementation.
Read the existing code and surrounding interfaces.
Implement a focused change.
Run the relevant tests and static checks.
Inspect the actual results.
Correct failures and repeat the verification.
Summarize the changes and any remaining limitations.

This process is particularly valuable for local models. A smaller model may be less reliable at reasoning about an unfamiliar codebase, but it can compensate by gathering evidence and checking its work.

Tool calling is not infallible, though. Models can produce malformed arguments, choose the wrong tool, misinterpret results, or continue iterating unnecessarily. The model, inference runtime, and tool definitions must all work together correctly.

Clear tool descriptions, well-defined input schemas, explicit usage rules, and reliable error reporting make these interactions more predictable.

MCP tools should also follow normal engineering security practices. Restrict filesystem access and command execution where appropriate, avoid exposing secrets, and require human approval for destructive or high-impact operations.

The principle is straightforward: use the model to coordinate work, but use real tools to establish facts.

The hierarchy: instructions vs agents vs skills vs MCP

These four layers serve different purposes and work together:

Layer	What it provides	Scope
Instructions	Project context, conventions, rules	Every interaction
Path-specific instructions	Scoped rules for specific file types	Matching files only
AGENTS.md	Agent persona and behavioral constraints	Directory-scoped
Skills	Structured multi-step workflows	On-demand
MCP servers	External tool access (tests, lint, search)	Always available

Instructions set the baseline. Agents define the role. Skills define the protocol. MCP provides the tools.

Practical tips:

Be explicit about file paths. Don’t say "the content directory". Say "src/content/service/myapp3/".
State constraints as rules, not preferences. Don’t say "it’s usually better to…" Rather use "Always…" or "Never…"
Include the schema. If your content has a validation schema, put it in the instructions. The agent can’t check a schema it doesn’t know about.
Keep it current. Update it. Stale instructions are worse than none. If you refactor your project structure, update the instructions. The copilot-instructions.md file is a living document, not a fire-and-forget thing.

4. Apply Spec-Driven Development to Control AI-Generated Drift

As AI agents become capable of modifying multiple files and executing longer workflows, a new challenge emerges: keeping implementation aligned with the original requirements.

A vague request can lead to a plausible implementation that misses important constraints or expands beyond the intended scope. This is often described as specification drift or implementation drift.

Spec-Driven Development addresses the problem by separating the definition of a change from its implementation.

Rather than immediately asking an agent to write code, first establish what needs to change, why it matters, which constraints apply, and how success will be verified.

A specification becomes a durable reference point that both developers and agents can consult throughout the lifecycle of a feature.

A practical propose–apply–verify–archive workflow

Tools such as OpenSpec and Spec Kit can help formalize this process.

A typical workflow consists of four stages.

1. Propose

The developer describes the desired change. The agent investigates the existing implementation and generates structured artifacts, such as:

A proposal explaining the motivation and scope.
A design document describing the architecture and trade-offs.
A specification describing expected behavior and constraints.
A task list breaking implementation into manageable steps.

The developer reviews the proposal before implementation begins. Missing requirements, incorrect assumptions, and architectural problems are cheaper to resolve here than after code has been written.

2. Apply

Once the proposal is approved, the agent follows the task list to implement the change. Each task should have a clear scope and, where possible, explicit acceptance criteria.

The agent uses repository instructions to respect existing conventions and MCP tools to inspect the code, run tests, and validate the results.

Breaking a large change into smaller steps also makes failures easier to isolate and review.

3. Verify

Verification should establish whether the implementation satisfies the specification, not merely whether the agent believes it is finished.

Depending on the project, this may include unit tests, integration tests, type checking, linting, security checks, and manual review of important behavior.

Failed checks should lead to investigation and correction. If the specification itself proves incomplete or incorrect, revise it deliberately rather than silently changing the implementation's intended behavior.

4. Archive

After the change has been completed and verified, archive its proposal, specification, and task history according to the project's conventions.

This preserves a record of the intended change, the decisions behind it, and the work that was completed. It also helps future developers understand why the code looks the way it does.

Choosing a specification workflow

OpenSpec and Spec Kit offer different approaches to organizing specifications and changes.

OpenSpec supports a more centralized change-management workflow, with proposals, tasks, specifications, and archived changes organized around individual changes.

Spec Kit offers a convention-based approach that can keep specifications closer to the code they describe.

Neither approach is universally better. The choice depends on repository structure, the desired level of formality, and whether the team prefers centralized change records or specifications associated directly with implementation areas.

The critical requirement is consistency: the agent should follow a defined protocol rather than inventing a new planning format for every request.

Why local inference complements SDD

Specification-driven development is inherently iterative. Requirements need clarification, proposals need revision, and implementations often require several rounds of testing and correction.

With cloud inference, repeated interactions can introduce meaningful usage costs. With local inference, the marginal cost of additional iterations is primarily electricity and hardware usage once the infrastructure is available (and time^^).

This can make it practical to explore alternatives, reject weak proposals, and improve specifications before committing to implementation.

Local inference does not guarantee better results, and hardware still imposes limits on speed and model capability. Its advantage is the ability to experiment with a more predictable cost structure.

5. Putting the Pieces Together

The four elements become most useful when treated as one coherent development system rather than independent AI features.

A typical architecture looks like this:

Copilot CLI provides the agent interface and coordinates the workflow.
A local inference server runs the selected language model.
Repository instructions and agent definitions establish project context and behavioral boundaries.
Skills define repeatable development procedures.
MCP servers expose testing, linting, search, and other development tools.
Specifications and task lists preserve intent and guide implementation.
Version control and review provide traceability and human oversight.

A developer can start with a feature request, use the agent to explore the existing code, generate a proposal, review the design, and approve a task list. The agent then implements the change incrementally, checks its work through real development tools, and produces results that can be reviewed against the original requirements.

This is a substantially different workflow from asking a chatbot to generate a block of code and copying the answer into an editor.

It is also important to preserve human responsibility. Developers should make architectural decisions, review significant changes, validate security-sensitive behavior, and decide when the implementation is ready to merge. Automation should reduce repetitive work without removing accountability.

Conclusion

Reliable AI-assisted development depends less on the model alone than on the engineering system surrounding it.

Local inference provides control over model execution and can make iterative workflows more economical. Explicit instructions and agent definitions establish the project's rules. MCP connects the model to real development tools, enabling evidence-based feedback. Spec-Driven Development preserves intent, structures implementation, and makes progress traceable.

Each element addresses a different weakness: models need context, agents need tools, complex changes need structure, and automated work needs verification.

The result is a development workflow in which AI does more than generate code. It can investigate, plan, implement, test, and refine changes within clearly defined boundaries.

The objective is not complete autonomy. It is repeatable, verifiable engineering with AI as a capable participant in the development process.