# Second-Source Your Models

> Hardware teams never design around a single-source part. Your agent stack shouldn't depend on one model either. How to keep the model swappable and test it.

- Canonical URL: https://webofmike.com/second-source-your-models/
- Author: Mike Moore (https://webofmike.com/about/)
- Published: 2026-10-25
- Last modified: 2026-10-25
- Tags: AI Agents, Generative AI, Platform Engineering, Evals
- Cite as: Mike Moore, "Second-Source Your Models", Web of Mike (webofmike.com), 2026-10-25. https://webofmike.com/second-source-your-models/


Every hardware engineer learns the same lesson early, usually the hard way. Don't design a board around a part only one company makes.

It's called second sourcing. For the parts that matter, you pick a primary and an alternate before the design is done. If the primary goes end of life, gets put on allocation or doubles in price, you swap in the alternate and the board still ships. The bill of materials says it right there, line by line.

Here's a small piece of one:

| Ref | Part | Primary | Alternate |
|---|---|---|---|
| U1 | Microcontroller | Vendor A | Vendor B, pin compatible |
| U4 | 3.3V regulator | Vendor C | Vendor D |
| C12 | 10uF capacitor | Vendor E | Vendor F |
| R47 | 10k resistor | Purpose unknown | DO NOT REMOVE |

Your agent stack has a part like U1 in it. It's the model. And most teams have no alternate on the BOM.

## Models change under you

A model isn't a foundation. It's a component, and components change.

New versions ship and behave differently on the same prompt. Prices change. Older versions get deprecated on someone else's schedule. Providers have outages. Rate limits change. Every one of those is a supply chain event, and if your agent only works with one specific model, every one of them is your incident.

I've written about what happens when the first responder at 3am is [an agent instead of a person](/pager-goes-off-nobody-wakes-up/). That loop is only as reliable as the model behind it. If the model is single-sourced, so is your on-call.

## Developers don't fully trust it either

Here's a number I keep coming back to. Stack Overflow's [2025 Developer Survey](https://survey.stackoverflow.co/2025/ai) found that 84% of respondents are using or planning to use AI tools in their development process, up from 76% the year before. Usage is close to universal.

Trust isn't. More developers actively distrust the accuracy of AI tools (46%) than trust it (33%), and only 3% report "highly trusting" the output. The biggest frustration, cited by 66%, is "AI solutions that are almost right, but not quite."

Read those together. Nearly everyone uses it, and most people don't trust it. That's exactly when you want the ability to compare and swap, not a long commitment to one option.

## What model independence looks like

The goal isn't to use every model. It's to make the model a setting instead of a dependency.

That means the important parts of your agent live in your layer, not inside one model's quirks:

- **Tools.** The agent's tools have clear names, clear inputs and clear outputs. Any capable model can call them.
- **Context.** The instructions, repo docs and runbooks the agent reads are in files you own and version.
- **Guardrails.** Permissions and approvals are enforced in your systems, not in a prompt that only works on one model. I covered that in [who gave the agent a badge](/who-gave-the-agent-a-badge/).
- **Tests.** You have your own task set and your own pass criteria. More on that below.

If all four of those are yours, swapping the model is a config change and a test run. If they aren't, it's a rewrite.

## Pick by task

Second sourcing also lets you stop pretending one model is best at everything.

Hardware does this all the time. You don't use your most expensive part in every spot on the board. You use the cheapest part that meets the spec for that spot.

For agents that might look like:

- A fast, cheap model for triage. Read the alert, sort the ticket, summarize the log.
- A stronger model for the hard fix, where getting it right the first time saves a lot of rework.
- A different model to review the first one's work. A second opinion from a different source catches different mistakes.

The point isn't the specific split. It's that you can make it at all, and change it when the numbers say so.

## Measure it like a bench test

When you qualify an alternate part, you don't take the datasheet's word for it. You put both parts on the bench, run the same tests, and compare.

Do the same with models:

1. **Build a fixed task set from your own work.** Real bugs you fixed, real tickets you closed, real reviews you did. Twenty good tasks beat a thousand generic ones.
2. **Define pass criteria up front.** Tests pass. The diff is reviewable. No forbidden files touched.
3. **Run every candidate model against the same set.** Same tools, same context, same limits.
4. **Track pass rate, cost per task and time per task.** All three. The cheapest model that passes is often the right one.
5. **Run it on a schedule.** Monthly is fine. Also run it any time a model version changes.
6. **Change one thing at a time.** If you change the model and the prompt in the same run, you don't know which one moved the number.

The test set only works if the tests themselves are honest. If your suite is flaky, every model looks worse than it is, and some look better. That's the whole point of [your tests are lying to your agent](/tests-lying-to-your-agent/). Calibrate the instrument before you compare the parts.

## Lock in your process, not your vendor

Second sourcing never meant you didn't have a favorite part. You did. You used it on most boards. You just made sure the board didn't die if that part did.

Do the same with models. Pick a primary. Qualify an alternate. Keep your tools, context, guardrails and tests in your own hands. Run the bench test every month.

Then the next time a model changes under you, it's a line on the BOM, not an outage.