Think hard once, or sample widely?

A creator publishing a reel most days needs a continuous supply of ideas, and the supply is the bottleneck — not the filming, not the editing. So we tested the question underneath it: is it better to think hard once, or to sample widely and select? We drew 150 ideas, one per call, across five frontier models, and measured what came back against every reel the creator had already published.

What follows is the whole experiment, including the part where it contradicted the assumption it was built on. We measured how varied the ideas were and how far they sat from work already published. We did not measure whether they are any good — nobody has shot them yet, and that limitation is load-bearing enough that it gets its own section.

This is the write-up. The paper it comes from — the same run, set out formally, with the reference list, the silhouette selection and the interactive figures — reads in full here.

01Why run this at all

The creator we ran this for posts a teaching reel most days, to an Indian audience learning AI and data science. A companion analysis of 35 reels found a median of 1,039 views against two breakouts at 34,405 and 58,264. The breakouts shared a shape: they reframed a job title the viewer already holds, and attached a one-word comment keyword to a real deliverable. The floor shared a different shape — eight reels between 458 and 672 views, five of which took exactly zero comments, each explaining a term or reporting a model launch without asking the viewer to decide anything.

That gives an unusually well-specified problem. We know what the target looks like. We know the dominant failure mode. And we have a labelled corpus of prior attempts to measure novelty against. The open question is procedural, and it is the one every person using these tools actually faces: think hard once, or sample widely and select?

02Method

The prompt

One prompt, roughly 5,700 characters, holding the creator’s mission, the measured performance of all 35 prior reels with real view counts, and the rules distilled from them. Two choices in it are load-bearing. It leads with the losers’ shared mechanic rather than the winners’, because “five of the eight worst reels took zero comments” is a more correctable instruction than “career topics win”. And it explicitly retires the winning sentence: the 58k reel’s construction is named as used up, so the models reuse the mechanism underneath it instead of producing 150 variants of one headline.

Sampling

One idea per call, not a list of ten. A model asked for ten ideas optimises the set — it pads after the first strong entry. One idea per call makes each draw independent, which is what makes the 150 rows comparable and the saturation analysis meaningful.

Thirty identical prompts at temperature 1.0 do not explore a space. Models have strong modes and re-sample around them, so the draws collapse onto a handful of ideas in different words. Each call therefore enters through one of 20 rotating lenses — a required angle of attack: the failure mode, the verification problem, the career door outside software, the thing a senior will not tell a junior. Call n uses lens n mod 20, so every model covers the full set at least once.

Models and parameters

We ran five models through OpenRouter, 30 draws each, at temperature 1.0, reasoning enabled, structured JSON output with a parse retry. Two configuration details cost real debugging and are worth recording. Setting provider.require_parameters, the pattern used elsewhere in this codebase to pin routing to schema-honouring providers, removes every viable endpoint for some vendors — Claude Opus 5 returned a 400 for unsupported structured outputs, and GPT-5.6 Sol a 404 for no endpoints found. And reasoning tokens bill against max_tokens, so an initial ceiling of 1,200 was consumed entirely by DeepSeek’s reasoning, returning empty content that looks exactly like a parse bug. The ceiling went to 6,000 after measuring Grok at 3,958 tokens on this prompt.

Measurement

All 150 ideas and all 34 recoverable prior reels were embedded with text-embedding-3-large at 3,072 dimensions, then row-normalised so a dot product is cosine similarity. Ideas were clustered with agglomerative clustering on cosine distance, average linkage, the cluster count chosen by silhouette. Cluster labels come from one model call per cluster; exemplars are medoids, meaning a real idea nearest the cluster centre rather than a synthesised description. Every random component takes a fixed seed (20260802).

03Results

Yield

150 of 150 calls returned a parseable, non-empty idea. No draw was wasted, and no vendor needed a second attempt beyond the parse retry that never fired.

Thematic structure

Silhouette selected 37 clusters. The peak is shallow — 0.1142, with neighbouring values within 0.004 — which is normal for short-text embeddings and means the exact count should be read as “roughly this many themes”, not as a discovered constant. Of these, 23 contain ideas from more than one model and 9 are singletons: one idea no other draw came near.

Novelty against the back catalogue

For each generated idea we took the cosine similarity to its nearest prior reel. The mean is 0.392, the maximum 0.5498, the minimum 0.22.

A posted reel’s nearest neighbour among the other posted reels scores 0.5267. A generated idea’s nearest posted reel scores only 0.392. The generator is therefore working outside the creator’s existing territory, by a margin larger than the territory’s own internal spread. That is the result the raw number was trying to claim, and it survives the control.

The same table carries a caution the headline hides. Generated ideas are more similar to each other (0.6375) than posted reels are to each other (0.5267). The pool is novel with respect to prior work while being internally tighter than the creator’s real range. That is a predictable consequence of the prompt: 20 lenses all pointed at agentic systems, so the output is concentrated on-mission rather than spanning everything he actually posts about. Read it as a deep seam next to the existing catalogue, not a wider catalogue.

Saturation, and whether 30 draws was the right number

This is the result the experiment was run to get. Resampling the draw order 400 times gives the expected number of distinct themes found after n draws.

The marginal return falls sharply and legibly. The first 30 draws yield 19.4 themes, about 0.65 per draw. Draws 31 to 75 add 10.7 more, about 0.24 per draw. Draws 76 to 150 add 6.9, about 0.09 per draw. A run half this size would have found roughly 81% of the themes.

Read the two curves as answering different questions. The gap at 30 draws (3.2 themes) is what mixing vendors buys at a fixed budget. The rise along the pooled curve is what buying more draws buys. The second is much larger than the first, which is the opposite of what this experiment was set up expecting.

Which model contributed what

Within-model similarity (0.3856) exceeds across-model similarity (0.3464). The margin is small but consistent, and it means each model has a recognisable signature: its own draws resemble each other more than they resemble another vendor’s. That is the mechanism behind the mixture gain.

One asymmetry in that table is worth naming. The least expensive model in the set reached 18 themes; the most expensive reached 15. On this task, at this sample size, price bought no coverage whatsoever. If this is repeated, the efficient configuration is to scale draws on a cheap model and keep one expensive model in the mix for contrast, rather than to spread a budget evenly across vendors.

The themes

Sizes are uneven by a factor of ten. The head of the distribution is where several models converged on the same territory; the tail is where a single draw found something on its own. Both ends are useful for different reasons — the head as consensus, the tail as reach. The 15 themes above the median hold 71% of all ideas, and the 9 singletons at the bottom are where a wide sample earns its keep over a deep one.

04What it means

Separating scale from mixture

The design confounds two variables, and the tempting summary — “one model reached 16 themes, five models reached 37” — silently compares 30 draws against 150. The honest accounting is less dramatic than that sentence.

• Mixture, at fixed budget. 30 draws spread across five models find 19.4 themes; 30 draws on one model find 16.2 on average. Pooling buys 3.2 themes, about 20% more coverage for the same money. Real, worth having, not transformative.
• Scale, at fixed mixture. Going from 30 draws to 150 takes coverage from 19.4 to 37.0 themes. That is the larger effect by a wide margin.

So the headline is that scale mattered more than model choice, with mixture a secondary bonus. This inverts the intuition the experiment was designed around. If a budget is fixed and something must give, take more draws before taking more vendors. The practical caveat is that mixture is nearly free to add — it is a list of model ids — while scale is linear in cost, so there is no reason to give up the 20%.

The lens rotation did its job

Mode collapse was the expected failure of this design, and it did not occur. A companion word-overlap analysis of the same 150 rows found zero cases of a model repeating itself, and all 20 lenses were represented in the output. The cost of that insurance was writing 20 lines of prompt.

Where the models did converge is informative rather than embarrassing. Two themes were reached independently by all five vendors: AI agent loop analogies (8 ideas) and AI Data Literacy (7 ideas). All five vendors landing separately on the same ground suggests those are optima of the evidence itself rather than a quirk of one model’s training. Against that, 14 of the 37 themes came from a single model — which is the clearest argument in the run for not standardising on one vendor.

05What this does not show

• We measured diversity and novelty, not quality. This is the important one. Nothing here shows these ideas will perform. Distinctness in embedding space is a necessary condition for a useful idea pool and nowhere near a sufficient one. The honest test is to shoot several and compare against the 1,039-view median, which has not been done.
• The cluster count is soft. A silhouette peak of 0.1142 is weak in absolute terms and the neighbouring candidates are within 0.004 of it. Read 37 as an order of magnitude for the thematic range, not a measured constant. Every downstream count inherits that softness.
• The saturation tail is partly an artifact. The curve resamples a finite pool without replacement, so it is forced to reach exactly the true total at n=150. Real additional draws would keep finding new themes at some low rate past that point. Only the early slope is a trustworthy rate; the flattening is real in direction but overstated in degree.
• One prompt, one topic, one day. All 150 draws share a single prompt on a single niche, run in one session. Nothing here establishes that the pooling advantage generalises to another domain, another prompt, or these same models next month.
• The pool is narrower than the creator’s actual range. Generated ideas cluster more tightly around each other (0.6375) than his posted reels do (0.5267). A run intended to broaden topical range needs a different lens set, and this experiment says nothing about how that would perform.
• Embedding similarity is a proxy for sameness, not a definition of it. Two ideas with different wording and the same underlying premise can land far apart; two with shared vocabulary and different intent can land close. The clusters were spot-checked and read sensibly, but no human labelled all 150.
• Prior work is 34 reels, not 35. One posted reel had no matched script and therefore no topic text to embed, so it is absent from every novelty comparison.

06Conclusion

The question was whether to think hard once or sample widely and select. On this task the answer is sample widely, with two corrections to the naive version of that advice.

The width that matters is draws, not vendors. The experiment was built on the assumption that mixing five frontier models was the lever. It is not. At a fixed budget of 30 draws, mixing buys 3.2 extra themes over a single model — roughly 20%. Multiplying the draws fivefold buys 17.6. Both are real; they are not the same size. Mixture survives as a recommendation only because it is nearly free to add, while draws cost money linearly.

The sampling needs structure or it does not scale. Thirty identical prompts at temperature 1 re-sample one region. The 20 rotating lenses cost twenty lines of prompt and produced zero within-model repetition across 150 draws, with every lens represented. Without that, this experiment would have measured the cost of rediscovering the same four ideas thirty times. Any attempt to replicate the yield reported here without a diversification scheme should expect a fraction of it.

What the pool is. 37 distinct themes, sitting genuinely outside the creator’s existing work: an idea is further from its nearest posted reel (0.392) than his reels are from each other (0.5267). It is a deep seam rather than a wider catalogue, because all 20 lenses point at agentic systems. For zero failed calls out of 150, that is a favourable trade against the alternative — a creator staring at a blank page.