I’m having a blast working with the internal tools Lisa has been building to accelerate our AI work at incident.

In a slightly unconventional working pattern, we’ve ‘ring-fenced’ Lisa to focus exclusively on building internal tools that accelerate the team as they work on products like our incident Chat flow or Investigations.

What this looks like is:

1. I run a backtest of investigations to target a new feature (finding causing code changes, for example)
2. Precision/recall isn’t where I’d like it to be, so I try digging into why
3. Most of the time our tools help quickly finding it, but…

If at this point I can’t easily find the problem, I stop and think “what would be the ideal productised flow for me to debug this?”

I then yeet this feedback into an ai-tooling Slack channel where Lisa normally responds within 60s saying “yep, on it” and several hours later we have the tools available.

So far this has resulted in:

• Widget that collect notes against our AI work so reviewers can say “This one is actually a hard one to find, we shouldn’t expect this to pass”
• Easily diagnose prompt errors and re-run them against many models, generating leaderboards that help select the model
• Automated diagnostic tools that give a first-pass of why a test may be failing

This interaction model may not work for everyone, but it works in our case because:

• Lisa is ridiculously good at building product like this, both extremely quickly and at high-quality
• Separating tooling investment from advancing the product is how you ensure even ambitious tooling projects get complete (you don’t think “urgh, do I really need this”)
• We’re working really closely together as a team, where we’re dogfooding these tools 10-30 times a day. This is how we built incident anyway, we know it works.

Honestly though it’s just really fun. The interaction model is high-contact but the pace is great, and the tools are helping us wrestle even our most challenging AI into submission.

You can see a bunch of this tooling in the attached screenshot, if you want to see it in action!