My AI agents do almost all their work on Opus, the biggest model in Claude, at the highest effort level. I wanted to know how much it changes to use a smaller model, or to have it think less. So I had ten finished jobs redone, from the smallest model up to the biggest.
Why I ran the test
I wanted to see what difference the model and the effort level make, depending on the kind of work I do. I also wanted to see the cost. There are models outside the Claude and OpenAI subscriptions, like Kimi and DeepSeek, that you pay for as you go. Does it make sense to drop the subscription and pay per use?
This first test stayed with Claude, which has three models. To me they’re like quarterbacks. You’ve got the starter, the backup and the other backup. Here that’s Opus, Sonnet and Haiku. And their effort level is just that, effort. The higher it is, the longer the model thinks before it answers, and the more it costs.
How I tested it
I took ten jobs my agents had already shipped with Opus at the highest effort. Small ones and big ones. A bug in the phone menu, a site’s page map, a page redone with new copy, a three-question quiz, game times in my football app.
Each job started over from the same point, with the same instructions. Then it climbed the ladder. Haiku at low effort, Sonnet at medium, Sonnet at high, then Opus at medium. The test stopped at the first model that passed.
Passing meant doing as well as what had shipped. The same checks pass, every ask in the job is done, and the desktop and phone screenshots show nothing worse. Opus graded every run.
Before the test, I thought Haiku was built for simple, repetitive, extremely well-defined stuff.
What came out
Seven jobs out of ten passed on Sonnet at medium effort or lower. Three passed on Haiku, a small configuration file and two layout fixes. I really didn’t expect that. It looks like it can do a bit more.
For those seven jobs, the right model cost US$3.53 in all. Opus had done them for US$34.72, almost ten times as much.
The whole test would have cost about US$59 pay-as-you-go. $23 to redo the jobs, and $36 to have Opus grade them. Grading the work cost more than doing it. On my subscription, it cost me nothing extra.
What failed
The three biggest jobs failed on every cheaper model tried. I had set a US$20 cap, and the test stopped before trying them higher up the ladder.
What went wrong were details. Haiku broke a scheduled task in my football app. Sonnet left labels hidden under a band of shade, on two pages. Sonnet at high effort got everything right on one page except a form field.
The worst part is that Sonnet’s failed runs passed every automated check. On one of them, the model’s own report said it had fixed the exact thing that didn’t work. It took Opus looking at the result to see it.
I don’t think it would have cost me a client.
I see it like handing something to an intern without checking what they did. It’s risky.
Kimi and DeepSeek weren’t in this test, so I can’t compare them yet.
Why I keep my subscription
What I take from it most is that a subscription is worth a lot more than paying per use. Billed per use, my agents’ work from October 5 to 15 would have cost about US$593. My Claude Max subscription costs me around CA$300 a month.
Test the models that offer a subscription, then go all in on the one you like best. That’s what I do with Claude. When it’s down, I’m comfortable using Kimi, through OpenCode.
As for the plan, it really depends on how much you use it. It’s easy for me to hit the limits of the US$20 a month plan, because I do a lot of programming. Someone else could get by just fine on it. If you get blocked by the limits, upgrade. If you hit the limits of the biggest plan, it’s worth taking an honest look at how you use it and optimizing.
What I do now
- Opus first. I stay on Opus. As soon as I see I’m getting close to the limit, I optimize with Sonnet, rarely Haiku.
- When in doubt, Opus. A failed job I have to redo costs me more than the few dollars saved. Whatever the job, Tom Brady can do it all.
- The mistake I see. People take the biggest model at the highest effort without being able to explain what they need. It gets expensive fast, because you go around in circles more often.
If you start on Monday
Get a ChatGPT or Claude subscription, whichever you’re comfortable with. Take the biggest model, at the highest effort, and give it all the information about your business. Then ask it how it can help you. Ask it for a game plan detailed enough to hand off to agents running Sonnet and Haiku. And that’s it.
