I have four AI agents running meaningful chunks of my life, and until recently I could not have told you, with a straight face, which llm model any of them should actually be using - I was just going with the cheapest frontier model available, which turns out the be the "china-maal" (Made In China models). My personal agent, Nora handles my large parts of my personal life including task, errand scheduling, managing calendar, coordinating with contacts, keeping track of my health and fitness and more such. The unimaginatively named itguy, is my VPS Admin, he deploys and manages my servers and promptly fixes everything I break on it. Vikraya is a specialized Sales CRO agent which i created for my founder/entrepreneur friends to get their sales motion in play. WeGro is a multi-agent revenue growth engine focusing only on bottomline growth. Four very different jobs but every leaderboard I consulted to pick the brains for these agents, gave me the same answer regardless of the question, which is a bit like asking four different tradesmen which hammer to buy and being handed the same catalogue each time.
The leaderboards, if you haven't looked lately, are obsessed with things like Humanity's Last Exam and abstract reasoning puzzles that neither me, nor any contact of mine, has ever asked for (I barely scraped through humanity's first few exams). So, a model can top the chart for solving olympiad level mathematics and still write a sales email that reads like it was copied from an interns draft. Benchmarks measure capability in the abstract. Agents however, need competence and efficiency in the specific activities they are built for. Those are not the same thing, and the industry has spent a great deal of money pretending they are (every lab insists their model is state-of-the-art).
So I stopped referring to leaderboards and instead built a sandbox testing environment for myself. Nothing grand, just a folder on my server, with pre-defined script that fired real tests at candidate models and recorded as-is what came back. This I felt, was going to my way of deterining the definitive, scientific answer to which LLM deserves to run my agents. I did gave it rules to follow, created synthetic data, sandboxed everything in one folder, capped the budget capped at a five dollars per model per run (the actual spend across twenty runs came to thirteen cents, which tells you something about how cheap curiosity has become). Temperatures of all models was set to zero so that results are comparable (as apples are to apples). And when a model wrote code, that code ran in a locked subprocess with no network and no filesystem access, because I wanted to know if a model could write correctly, not whether it could talk its way past me.
That discipline, synthetic, sandboxed, budgeted, deterministic and jailed, is the whole point of this test - at least in the beginning. It's boring on purpose and thats also the reason I trust the results at all.
From there I built out a proper framework, five tasks, each one mapped to something each of the agents actually do (or at least, are supposed to do): research with citations, a planted bug to debug, creating a sales GTM play, figuring out how to hold a reasoning chain together, and a content-gap analysis for a real (synthetic, but realistic) business. I called this framework, the "Ass Whole Outline", partly because I am twelve years old at heart and partly because a name you can't say out loud with a straight face is a decent filter for whether you're taking yourself too seriously (you generally aren't, once you've named your project ASS). The testing operation is now named AyeBee's ASS - AyeBee's AI Screening System.
So under AyeBee's ASS test, every task gets scored the same way: sixty percent for quality, twenty-five for percent cost/result, fifteen percent for latency (human patience is a dying quality). I felt this kind of weighting will be the determining factor, more than only analysing any single result the runs have produced.
Why? Because a model that's brilliant but twenty times slower and ten times more expensive does not win here, because people dont want to spend extra money; someone would quietly swap it out the first time the API bill arrived. So instead, I wanted a scanner that rewards the model you'd genuinely want to run at 2am when nobody's watching, not the one that photographs well on a benchmark podium.
The first proper run bore this out with a bit more drama than I expected. Grok 4.6, the US flagship in the race, took eight minutes to write a single sales email and cost ten times what the cheapest model did for the same suite, with no quality edge that justified either number. Tencent's Hy3 passed every objective task at a tenth of the cost with minimum quality deviation (again, we are not testing for proficiency in Humanitys Last Exam). DeepSeek v4pro wrote the best prose, if you had the budget and the patience for it. None of these results came from a spec sheet. It came from watching four models actually do the four jobs I needed done, and it's the only reason I now know, with some confidence, which model sits behind which agent.
Here's a summary of the results:
So, I've concluded with a high degree of cretainty, that the model matters less than people actually think, and the workload matters more than anyone admits. Most people ask "which model is best" as though there's a single correct answer. unfortunately there isn't and never will be. There's only "best for this job, at this cost, at this speed," and that answer changes agent by agent, task by task, quarter by quarter as pricing shifts under you. Benchmark culture sells the fantasy of one true answer because one true answer is easier to publish. Real work doesn't cooperate with that fantasy, and pretending otherwise is expensive.
The sandbox has since grown a public face, a live site at abass.ayebee.xyz that shows the results, a blind-ranking page where you judge the outputs before you know which model wrote them, and a name that still makes me laugh every time I say it out loud. But the site is downstream of the thing that actually matters, which is that I stopped asking vendors what their models could do and started asking my own work instead. It answers faster, and it doesn't have a marketing department or lobbyists.
If you're running agents on someone else's leaderboard, I'd ask yourself the same question I eventually asked myself: has anyone actually tested this on the work you do, or just on the exam everyone agreed to study for together? To try out AyeBee's Ass, head over here.