What we did. We gave 13 AI models the same sales and marketing jobs, for example: “Create a GTM pitch for [product].” Each model built its own version. Then two AI judges compared the results side by side, without knowing which model made which. Since September 3, that has added up to 353 test runs, for pitches but also for documents and images.
What we learned. There is no best model. An open model was as good as Claude Opus, and cheaper. A score from 1 to 10 moves around more than many models differ. AI judges favor their own model family. And three small rules for layout and text did more than any change of model. If you use AI for content, first decide how you will recognize a good result. Choose the model second.
- 353test runs since September 3
- 99pitches compared blind
- 13AI models tested
- 17/20wins for three small rules
What we measured
The Brand Studio is our brand AI. It builds pitch decks, brochures and documents in a company’s own design, using its images, product data and rules. That takes four jobs that an agency, a designer and a copywriter would normally split between them: planning the content, writing the text, choosing or making images, and laying out the pages.
We have measured each of these jobs since September 3. For us, a test run means one complete build under fixed conditions: same job, same data. Then we score the result and write down the time and the cost.
353 test runs, one dot each
Pitches 233Documents 30Images 63Speed and cost 27
There were probably more. We only count runs we can show in logs or result files. Where a later test corrected an earlier one, you see the corrected number.
Lesson 1: A score tells you less than you think
At first, an AI judge gave each pitch a score from 1 to 10. Then we ran the exact same setup twice. Once it scored 6.74, once 6.34. Nothing had changed except chance. Those 0.4 points are more than the gap between many models.
So we stopped giving scores and let pitches compete instead. Two pitches for the same job sit side by side. The judge doesn’t know who made them and picks the better one. Two judges look at every pair, twice: once on the left, once on the right. Each of these is a match, and we count how often a variant wins. 50% means equally good. As a check, we pitted two identical setups against each other. They came out at 55%. That is how much room chance has.
That’s how we built and compared 99 pitches from ten models on October 10, plus three GPT models in a second batch. The best pitches came from Kimi K3, an open model anyone can download. Claude Opus 5.5 kept up: at 16 matches to 14, it’s too close to call. But Opus cost €1.42 per pitch, Kimi €1.19. The Opus result comes from a corrected test; more on that below. DeepSeek V4.1 Flash was much faster and cheaper: 52 seconds and €0.13, against 250 seconds and €1.17 for Kimi. But it only won one match in five.
The model garden: the taller the plant, the more often the model won
Kimi K3, our yardstickcan’t tell apart from Kimiclearly weakernot compared
This picture is more honest than any ranking. DeepSeek V4 Pro (35%) and V4 Pro 0813 (30%) look weaker than Kimi. But with so few matches, we can’t say so for sure: their ranges reach past 50%. Only Claude Sonnet 5.5, DeepSeek V4.1 Flash and the two GPT Sol models are clearly weaker. Five models never made it into the comparison. Qwen 3.8, GLM 5.3 and MiniMax M3 built only 3 of 5 pitches, GLM 5.3 Flash none at all. And GPT-6.1 Sol couldn’t use the tools it needs to build anything, at least not in our setup.
Lesson 2: AI judges have favorites too
If an AI does the judging, you have to check the judge too. Kimi gave its own pitches 7.7 points on average. A GPT judge gave the same pitches 6.6. It works the other way round as well: the GPT judge let GPT’s pitches win 14 of 20 matches against Kimi. With Opus as the judge, they won none.
Same pitches, different judges
Points for the same Kimi pitches, from 0 to 10
How often GPT’s pitches beat Kimi, out of 20 matches
A second mistake was much more mundane. Five screenshots of Opus pitches showed a notice from our development tools: “2 Issues”. The judges took points off for it. Only after we fixed that was Opus level with Kimi. Since then, no model judges its own family alone. And we check what the judges actually get to see.
Lesson 3: Small rules beat big models
The biggest improvement didn’t come from a new model. We added three small rules for layout and text, all at once. With them, the Studio won 17 of 20 blind matches against its old version. Both judges agreed. On top of that, it got faster, 187 instead of 250 seconds per pitch, and cheaper, €1.08 instead of €1.17.
How often each change won in blind matches
If you want to save time, check what you lose. When Kimi thinks briefly instead of thoroughly, a pitch is done in 79 instead of 199 seconds. It still wins 40% of matches. We also let two models share the work: one plans, one writes. If the fast model plans and the good one writes, the pair wins only 20%. If the good one plans and the fast one writes, it’s 28%. So save where you like, but don’t skimp on the planning.
A second count points the same way. In our pitches, the planner decides the layout of 84% of the pages. The design step afterwards changes only 16%. If you want better pages, start with the planning.
The gap between an ordinary slide and a well-designed one is easiest to see with the same material. Here are two examples, each with the same photo and the same content:
Example: an ordinary slide and a designed slide from the same material
There is no best model, only the right one
The question we hear most: which model is the best? Our tests say it’s the wrong question. A model is good or bad at a specific job, not in general. You can see that when you put cost and quality side by side.
What a pitch costs and how often it wins
Kimi K3, our yardstickanother modeltwo models or a shortcutno match won
Three things stand out. First, more expensive doesn’t mean better. Claude Sonnet 5.5 costs more than Kimi thinking briefly, and wins less often. Second, very cheap is possible, but not for everything. DeepSeek V4.1 Flash is nine times cheaper and wins one match in five. That can be enough for a first draft, but not for the pitch to your board. Third, some models drop out completely. GPT-6.1 Sol couldn’t use the tools it needed in our setup and didn’t build a single pitch. Being at the top of a public ranking doesn’t make a model right for your job.
For us, this has turned into a kind of lineup. Each job gets the model and the method that won the test for exactly that job:
- Planningthe strongest model, with time to thinkIf a fast model plans, the result wins only 20%. The planner decides the layout of 84% of the pages.
- Quick drafta strong model that thinks briefly79 instead of 199 seconds, and it still wins 40%.
- Building with toolsonly models that handle tools reliablyA top model didn’t build a single pitch in our setup.
- Judginga judge from another family, blind and in pairsIts own family got 7.7 instead of 6.6 points and 14 of 20 wins.
- Checking layouta program measures, instead of an image AI lookingThree image AIs missed cut-off text. Measuring found it.
- Images with letteringgenerate from scratch instead of fixingEditing a photo: 0 of 12 labels error-free. Generating from scratch: 7 of 7.
This lineup fits our jobs and our data. Yours will look different. That is exactly why it pays to test for yourself instead of following a ranking from the internet.
Lesson 4: The most common text problem: stock phrases
We also score the text in every pitch. A strict reviewer reads each page and notes what’s wrong. In one overnight run with 34 pitches, it flagged stock phrases in 22 of 27 reviews. Next came empty closing pages (13), repetition (12) and text that was hard to read on top of an image (12).
In a second series with 38 reviews, other problems came first: repetition (49 spots), images that were missing or didn’t fit (42), empty space (31) and results that missed the brief (31). So good text doesn’t come from a better model alone. It comes from a check that finds stock phrases and repetition and sends them back for another pass.
Lesson 5: For images, start fresh instead of fixing
Image AIs make products look deceptively real. What gives them away is the small print on the product. We counted this on product images with fine lettering, in 63 runs with one image model. The obvious route is to edit an existing photo. That gave error-free lettering in 0 of 12 cases. Generating the image from scratch, with the exact text in quotation marks in the prompt, worked 7 times out of 7.
Error-free lettering, one dot per try
Whole scenes are still hard: 36 tries for 4 usable images. And details depend on precise words. One part had to sit on the correct side. With the technical term, that worked in 2 of 5 cases. With a plain, explicit description, in 3 of 3. If the product takes up less than about a third of the image width, the lettering almost always breaks. These are small numbers, three to twelve tries per case. The direction is still clear.
Lesson 6: First look at what users actually do
For eleven rounds, we improved how Claude builds brochures and Word documents through our Studio. PDF brochures now need half as many steps, 16 instead of 32. A Word document is done in 120 instead of 224 seconds and costs a third less. Breaches of the brand rules are gone; before, there were up to four per brochure. Without the Studio, Claude needed 17 to 28 minutes for a Word document.
Then we looked at the logs of how people actually use it. They build almost nothing but slide decks. We had perfected a route that hardly anyone takes. Since then, every test series starts with a look at real usage.
One last test shows how much depends on small things. In claude.ai, the model sees only the first 80 or so characters of each tool description. When the tool’s purpose came later in the text, Claude picked it for brochures in 0 of 3 cases. With the purpose at the start, in 3 of 3.
How we counted, and where the limits are
Every percentage comes from blind matches. The judges see two results for the same job without knowing which model made them. They see each pair twice, once with the order swapped, so being on the left or right gives no advantage. For every value we give a range. It shows how sure we can be. We calculate it with Wilson’s method as a 95% interval, because it gives sensible limits even for few matches and for values close to 0 or 100%. Times are middle values (medians). Costs are the amounts actually billed in each test series.
| What we tested | How many | Result | Range |
|---|---|---|---|
| Three small rules vs. old version | 20 matches | 85% | 64–95% |
| Claude Opus 5.5 vs. Kimi K3 | 30 matches | 53% | 36–70% |
| DeepSeek V4 Pro vs. Kimi K3 | 20 matches | 35% | 18–57% |
| DeepSeek V4 Pro 0813 vs. Kimi K3 | 20 matches | 30% | 15–52% |
| Claude Sonnet 5.5 vs. Kimi K3 | 20 matches | 20% | 8–42% |
| DeepSeek V4.1 Flash vs. Kimi K3 | 20 matches | 20% | 8–42% |
| GPT-6 Sol and GPT-5.6 Sol vs. Kimi K3, each | 10 matches, one judge | 0% | 0–28% |
| Sanity check: two identical setups | 20 matches | 55% | 34–74% |
| Kimi thinks briefly vs. thoroughly | 5–10 pitches per variant | 40% | 30–52% |
| Kimi plans, Flash writes | 5–10 pitches per variant | 28% | 19–38% |
| Flash plans, Kimi writes | 5–10 pitches per variant | 20% | 8–42% |
| GPT judge lets GPT’s pitches win | 20 matches | 70% | 48–85% |
| Opus judge lets GPT’s pitches win | 20 matches | 0% | 0–16% |
| Stock phrases in the reviews of one overnight run | 27 reviews | 81% | 63–92% |
| Error-free lettering, photo edited | 12 attempts | 0% | 0–24% |
| Error-free lettering, generated anew | 7 attempts | 100% | 65–100% |
| Same setup, scored twice | 2 builds | 6.74 / 6.34 | gap 0.4 |
50% means as good as the variant it was tested against. If the range reaches past 50%, we can’t show a difference.
Being honest means naming the limits. Many variants rest on only five to ten pitches. We tested images with a single image model. Costs vary from one test series to the next; for Kimi K3, between €1.13 and €1.19 per pitch. And AI judges are models with preferences of their own. That’s why judges from two families always decide together. Where a later test corrected an earlier one, for example after the faulty notice on the judges’ screenshots, you see the corrected value.
What you should do now
- Decide what good looks like. Then pick the model. Take 10 to 20 real jobs from your daily work, for example a pitch, a quote and a product text. For each, write one sentence on how you’d recognize a good result. Without that list, you’re choosing a model by gut feeling.
- Pick a model per job. The strongest model for planning, a cheap one for quick drafts, one from another family for judging. A single model for everything is either too expensive or too weak.
- Compare instead of scoring. Put two results for the same job side by side, without names, and pick the better one. In our tests, the score for the same setup moved by 0.4 points. A direct comparison shows differences where scores are just noise.
- Check the judge. Don’t let a model judge its own family alone, so no GPT judging GPT. And look at what the judge actually gets to see.
- Rules first, new model second. Before you switch models, write down three clear rules for layout and text. In our tests, three such rules won 17 of 20 matches. The best change of model only got a draw.
- Look first, then improve. Check the logs to see what your people actually build. Otherwise you perfect a route nobody takes.
An AI that makes content looks finished fast. Only testing shows where it really stands. If you want to set up your own test for your pitches, documents or images, we’re happy to pick the first jobs with you.
