We’ve been working on a project called Extensions: customers can now connect their MCPs into our platform then load skills that tell us how to use them, with both skills and connectors (our word for a connected MCP or API) woven into all our agent surfaces. Basically, anything you can do in your Claude/Codex/etc should be doable in our platform, and vice versa.

It’s been a fun project but challenging, in part because until now we (incident.io engineers) have been passive consumers of these technologies. We used them, sure, and lots of people had written a skill, but were they good? Did they actually work, beyond just for whichever person had written them? The honest answer is we didn’t know, probably not, and no one could confidently say they had mastered the tools.

Extensions meant this needed to change, so we speed-run the process by:

• Building skill support into our agent harnesses
• Created a product with a strong conceptual model around skills
• Dogfood’d the hell out of it: even hooking @incident up to an office printer
• Arrived at opinions on best-practices
• Encoded those opinions into product
• Exposed feedback to customers on how their skills were performing
• Helped those customers actually improve their skills

Like much in the AI technology age you don’t need to know how this stuff works to use it, but you will be much more effective if you do. So after having the same “Skills and MCPs 101” chats with the team, I’ve taken the explanations that worked best and written them into this post.

If you want to know how this stuff actually works, this post is for you.

LLM 101

Before we can talk about how skills work, we need to recap some LLM fundamentals. Feel free to skip if you already know, but…

All LLMs expose an API that looks much like a message thread. Each message has a ‘role’ which tells you what provided the message, be it instructions given by the system, messages containing data for this user, or assistant messages which are generated by the LLM model in response to the former.

The standard example of asking an agent for the “weather in London” ends up looking like this:

system: |
  You are a helpful assistant. Use your tools to answer questions.

tools:
  - name: get_weather
    description: Get the current weather for a location
    input: { location: string }

messages:
  - user: What's the weather in London?

  # The model can't run anything itself: it responds by asking
  # the harness to make a tool call on its behalf...
  - assistant:
      tool_call: { name: get_weather, input: { location: London } }

  # ...the harness executes it, appends the result to the thread,
  # and asks the model to continue.
  - user:
      tool_result: { temperature: 14C, conditions: light rain }

  - assistant: It's currently 14°C with light rain in London.

In this example we set up the agent with a purpose via the system message and a single ‘tool’ called get_weather. LLMs don’t execute tools themselves. Asked a question like our user message (“What’s the weather in London?”), the model runs a standard flow:

1. Do I already know the answer, either from my training or from what’s in the thread? If so, respond with it.
2. If not, do I have a tool that could get me the answer? If so, return a tool_call describing how to call it.

Tools are how the LLM can see outside of its context and communicate with the outside world. They can be informational like get_weather or take action, e.g. send_message.

LLMs return a request to call the tool with parameters and expect the surrounding code that is running the agent (often called a harness) to look up the tool’s implementation and call it with those parameters. The result is then appended to the message thread and the entire thread sent back to the LLM, at which point it can run the same process as before but now the ‘answer’ is available in context, so it can immediately respond.

There is no concept of ‘memory’ baked into the LLM models themselves, they are ‘stateless’ in that all state is carried in the thread. The part that most people don’t clock is that an agent loop that executes tools repeatedly calls back to the model, gradually appending messages as it goes, as this is hidden behind a loading spinner in most LLM powered applications.

That’s all you need to know, other than one important subtlety around system messages which is that models are trained to treat the system message as their highest-priority instructions, outranking anything the user says. If you’re ever writing prompts it’s critical you realise this is baked into the core of these models, they take it very seriously.

A common mistake when naively writing a system prompt is saying something like:

Generate between 3 and 5 tags for this document.

Sounds sensible, right? You’re tagging documents and ~3 keywords is ideal… until someone uploads an empty doc and your LLM spits out something like “god” “help” “me” because it is DUTY BOUND to follow the system prompt to the letter, and you didn’t anticipate this edge case. In my experience building agent products, when a frontier model hallucinates it’s almost always because a prompt trapped it like this, which is why you must be extremely careful of how you write them.

More of that later, but that’s your LLM 101 done.

Agents are a loop

Now we’ve covered LLM fundamentals, we can define an agent, and it’s not anything crazy: an agent is the flow from the previous section run in a loop. Call the model, execute whatever tools it asked for, append the results to the thread, call the model again, and stop when it stops asking for tools.

An agent is this loop and nothing more: send the thread, run whatever the model asked for, append the result, send it all again. The green arrow is the only part you ever see.

Claude Code, Codex, Cursor: underneath the branding, they are all this loop, provided with a set of tools that make up what people call the ‘harness’.

Given everything these agents can achieve, what tends to surprise people is how mundane the tools are. Here’s an abridged version of what Claude Code hands the model:

tools:
  - Bash     # run a shell command
  - Read     # read a file
  - Write    # create a file
  - Edit     # find-and-replace within a file
  - Grep     # search file contents
  - WebFetch # fetch a URL

Every impressive thing a coding agent has done for you boils down to a model with access to a shell and a text editor. That might seem odd until you realise that’s all programmers need to build any of the systems you depend on daily, so really this is about giving LLMs access to the magic programmers have already been using for the last half a century.

Ok, so this makes sense, but I want to stress-test your understanding with a question: the bash tool, do you think it actually needs to run bash?

The answer is no! An LLM’s entire reality starts and stops at the messages that are provided to it, which means it never needs you to actually use bash to execute this command, it only needs you to return a tool result that looks like what bash would have returned.

This turns out to be quite important as it means even agents running in limited environments without bash can act like coding agents, provided they implement tools that can emulate how bash would behave.

At incident we’ve built a shell package into our Go app that implements a virtual filesystem and can interpret bash commands, calling ‘binaries’ that are actually just Go functions. It doesn’t matter that we’re not really running commands, as soon as we give a bash tool to our harness, the model can use it as if it were real.

Pretty neat, but the lesson is important for what we’ll cover next: these models are trained on patterns, and if you can provide a pattern they recognise (like a shell) then the rest of their training kicks in, and you can start leveraging the model to do smart things.

We’ll return to that later, but now…

MCPs are just APIs in tools

Everyone and their dog are talking about MCPs. It’s tiring and a bit off-putting, especially as the technology is barely two years old and has a loud contingent of people already declaring it dead (MCPs are dead! long-live CLIs!). AI, huh.

Let me start, then, by saying that MCPs are dead boring. It stands for Model Context Protocol, which won’t tell you much. Mechanically, an ‘MCP’ is just a server that lists a set of tools which an agent can connect to and make use of.

Take an example like our incident.io MCP. We host this MCP at an endpoint (mcp.incident.io/mcp) that serves the MCP protocol over (mostly) normal HTTP, where the protocol allows us to configure a list of ‘tools’ that other agents can connect to and make use of.

When someone connects Claude Code to the incident.io MCP, what happens when you boot the agent is:

1. Claude contacts our MCP server, asks “what tools do you have?”
2. Those tools are added to the existing tools in the prompt sent to the LLM
3. Claude’s harness adds some hints about the MCPs into the agent thread so it’s aware of what is loaded

There is some magic I won’t touch on (such as tool search, where tool schemas are left out of the prompt until the model searches for them) but that’s almost all there is to know. MCP servers are as vanilla as your standard HTTP API, just in a protocol that an agent can load and make use of.

They are so similar to HTTP APIs that we have implemented support for any OpenAPI specified HTTP API in Extensions, and expose the endpoints of those APIs to our agents just like MCPs. Genuinely nothing new here.

So now we have the tools, but what tells us how to use them?

Skills are just instructions

Have you ever tried building an integration with an HTTP API just from the reference docs? Where all you have is a page per endpoint and the OpenAPI specification?

I have, many a time. It works and you can gradually piece together what you want to achieve, but it’s a bit slow and error-prone. With the MCP tools added to our agent we are now in this world of “all the ~gear~ tools and no idea”.

The best API docs go beyond reference pages and offer guides and tutorials that teach best-practices and how to use the endpoints together. We need some way to give this to our agent, not just for using MCPs (though this is a key use-case) but for any work where the agent could use some explanation and context to help it achieve a thing.

That’s where skills come in.

Skills are a collection of files containing instructions that agents can read and use. They are stored in a directory that doubles as their name, with a core SKILL.md entrypoint.

As part of dogfooding Extensions we built an MCP server that exposes a bunch of tools to check health of our system queues: for example tools to view all queue topics (queue_topic_list) and purge a subscription (queue_subscription_purge) if we’re having issues.

With this MCP server connected we get those tools, but it would be naive to expect an agent like Claude to know how to use them without proper explanation. Our queues run on both NATS and Google Pub/Sub for redundancy, which is important to know if you want to interpret “how many messages are in X subscription” as you’ll get results back for both technologies. You probably want to caution the model on when to actually purge a subscription to avoid any over-eager calls, too.

We can and do put all of this into a queue skill, which looks like this:

ops/skills/queue
└── SKILL.md

Skill markdown supports ‘frontmatter’ which is a small YAML snippet at the top of the file. Skills are expected to provide their name and a description, and the harness places a listing of these into the agent’s context — much like the tool list from our LLM 101 example — which is how the agent knows a skill exists and can decide when to ‘load’ it.

Our queue skill looks like this:

---
name: queue
description: >
  Inspect and purge event queue backlogs across Pub/Sub and NATS with
  the Toolbox connector's queue tools. Use when events are delayed, a
  subscriber has a backlog, someone asks to purge a queue, or an
  incident involves event processing lag.
---

# Event queues

Every event publishes through a load balancer across two brokers: Google
Cloud Pub/Sub and NATS JetStream. Each message lands on exactly one of
them, so a subscriber's backlog lives on **both brokers**, and neither
alone is the whole picture.

## The tools

- `queue_topic_list` — what exists, merged across brokers.
- `queue_topic_show` — one topic's stored messages plus every
subscription with its backlog, per broker. Start here for "is X backed
up".
- `queue_subscription_purge` — the write tool. Preview with `dry_run:
true`, execute with `confirm: true`; a call with neither is rejected.

## Reading results

- `null` counts mean the broker can't say, never zero. Pub/Sub's admin
API carries no sizes, so get real numbers from telemetry instead.
- Read `warnings` before acting on any purge result.

Before purging, check the backlog isn't already draining: watch
`core_event_subscription_oldest_unacked_seconds` — falling means riding
it out costs nothing. Most backlogs don't need a purge.

...

Note that only the name and description sit permanently in the agent’s context: the body of the SKILL.md is only read when the model decides the skill is relevant. Even frontier models degrade as their context grows which is why it’s important to use it sparingly: every token costs money and competes for the model’s attention, so you want each conversation to pay only for the skills it actually uses.

Skills can contain files beyond the SKILL.md and should if they grow large, extending the same trick another level. This practice is called ‘progressive disclosure’ where the SKILL.md acts more like an index of what is available so the model can choose what it loads next.

An example of a more complex skill is ai-providers which helps debug AI provider incidents and understand how we load balance between providers:

Descriptions are cheap and always present; bodies are expensive and load on demand; reference files extend the same trick another level. Each conversation pays only for the skills it uses.

Recalling how agents are given a bash tool that can run shell commands just like a normal terminal user, it’s totally valid for an agent to ‘load’ a skill by just cat SKILL.mding the file. That brings the skill content into the session and the agent is free to use the instructions as it chooses.

In practice, agents often wrap skill loading in a skill tool. Naming it that helps — the tool’s description can explain what a skill is and how to treat one, and skills are a pattern models have seen plenty of by now - but mostly the wrapper exists for practical reasons, like tracking skill usage, or so the harness can inject useful data alongside the skill load that saves the model work.

The incident harness, as an example, when calling our skill tool will:

1. Handle resolution of the skill by name so you don’t need to know the exact path
2. Track that a skill has been loaded so we can provide observability and grade how it performs
3. Returns to the model:
   1. The full SKILL.md content
   2. A small directory index of files so the model can avoid a subsequent ls -R

This works really well across OpenAI, Anthropic and Gemini models. It’s all very simple, easily explainable, and needs nothing provider-specific, which is exactly why the same skill can run on all of them.

Plugins bundle skills

You now understand how LLMs work, how MCPs are loaded into them, and how skills can bundle behaviour that an agent can load when appropriate to extend its base instructions.

That’s almost the entire story, with just one thing missing which is about how to distribute skills so you can install them in your agents.

Anthropic have solved this with their plugin format which bundles together skills and other files into a distributable format. If you’re used to npm packages it looks much like this, where you turn a directory of skills into a plugin by adding a .claude-plugin/plugin.json.

Our ops plugin with production operation skills looks like this:

# ops
.claude-plugin
└── plugin.json
skills
├── ai-providers
│   ├── references
│   │   ├── how-we-use.md
│   │   ├── live-health.md
│   │   ├── output.md
│   │   └── usage-spend.md
│   └── SKILL.md
├── ...
...

Lots of products implement MCP and skill support by consuming these plugins: Anthropic’s Claude Cowork is one, and incident.io Extensions is another.

This isn’t rocket science but it is exciting, mostly because arriving at a consistent format for sharing these skills is finally meeting the promises made by AI tools since they first arrived.

We have an internal AI team at incident who have built a ‘data brain’, which gives natural language querying into all our data systems, enhanced by docs for each data model and guidance on how to use and interpret queries against those models in the context of our business. Everyone at the company uses it daily now, and I was one of those users, but admit I didn’t clock how it was built until planning how we’d build Extensions.

It turns out that ‘data brain’ is just a ‘data’ plugin containing skills and reference documentation, paired with an internal MCP that gives query access into our data warehouse and other data sources. Once we had a prototype of Extensions ready, I could in three clicks load the plugin and connect the MCP, and suddenly the ‘data brain’ was running in all our agents.

You might think it was janky with sharp edge cases but… it worked almost entirely out the box. This was a use case no one in the internal AI team had ever planned around, but by leveraging these standards I could plug their work into our product and suddenly we’re Data Ready™.

There are complexities to this, such as being careful to write ‘portable’ skills, which I’ll cover in a follow-up post. But this was the first time in my career that I’ve been able to integrate with such ease against a complex system and have it work out the box with zero tweaking. AI can be magic, if you know how to use it.

Skills replace system prompts

Finally, for those who build agents, it’s worth noting how skills offer a new way of building than you might have used before.

Until now, we’d created specific agents for each part of our system. We had a scheduling agent that knew how to work with our On-call product, and a telemetry agent that could execute natural language queries against telemetry providers and interpret the results.

Those agents had system prompts that taught them how to behave, and you could access another agent via a tool (e.g. scheduler_query) that would send messages to that agent and receive them back, segmenting context and control so we could scalably build complex agents without overwhelming any single model’s context.

With skills, we are now killing those agents in favour of a single agent with a collection of skills built from those specialised agent system prompts. Each conversation loads only the skills it needs (progressive disclosure again), and the change comes with some real advantages.

Left: context fragments across agents and every instruction is law somewhere. Right: the same instructions become skills in one shared thread — below the system prompt, so the model weighs them against the user instead of obeying.

The first is flexibility. Recall from LLM 101 that models treat the system message as their highest-priority instructions: exactly what you want for safety and identity, and exactly what you don’t want for behavioural advice, because every set of instructions has edge cases you didn’t anticipate, and when they live in the system prompt the model can’t route around them.

Our telemetry agent is a good example. It powers all types of observability query (“what is the error rate of web requests in production in the last 15 minutes”) across the ~20 observability platforms we integrate with (Datadog, Grafana, Honeycomb, etc), and we have to be careful planning our queries so as not to overload the provider, with an example being advice to split long-range queries into many queries over smaller time ranges.

With that advice loaded via system prompt, users of our chatbot would sometimes request a specific “get all error logs in the last week” that they know can be queried safely, and our agent would stubbornly refuse to query the entire range. This was frustrating, with users telling us to “just do the f***ing query” and the agent holding fast to its system instructions.

Now that advice loads as a skill, and remember loading a skill is just bringing the instructions into the thread as ordinary messages, below the system prompt in the hierarchy. The model treats them as strong guidance to weigh against what the user is actually asking for, rather than law it must obey, and this chat interaction now succeeds. Smarter models do better when given guidance rather than strict rules, and skills are how you hand them guidance.

One caveat: some rules should stay non-negotiable, and those don’t move into skills. The guarantees that must hold no matter what belong in the tools themselves — like our purge tool refusing to run without an explicit confirm — where nobody can talk the model past them. Constraints you must enforce belong where the model can’t negotiate, such as in the tool layer.

Flattening these agents into a single session also removes the downside of context segmentation, which is where agents would have incomplete information as you called into their sessions. Now one agent sees everything, understands the nature of the task beyond a single instruction, and can be much smarter about how it plans tool calls and interleaves intermediate results for a better outcome.

That’s it, really

That wasn’t that bad, right?

Models only ever see a simple message thread per request. Agents assemble that request in a loop, MCPs add tools to it, skills add instructions when they’re relevant, and plugins ship the two together.

AI is moving so fast it can feel really overwhelming. Most best-practices have a half-life of ~six months before they’re replaced, and keeping up with the latest trend can be exhausting. Thankfully, I feel we’re arriving at a more stable place where tools like MCPs and skills are going to stay for the longer term, and it’s worth building an understanding that is deeper than surface level if you want to use them well. Hopefully this post gives you that, along with some insights that come from implementing these features in a harness used by real people in production.

I have a follow-up post planned on treating skills like the software they’ve become: writing them so they run on any platform, and what you should expect from products that run them for you — starting with telling you when a skill misled an agent, and helping you fix it.

For now though, you have the mental model, and my previous post on building AI skills like checklists covers writing a good one.

If you liked this post and want to see more, follow me on LinkedIn.