Posted on 

 by 

 in , , ,

Creating an AI Braintrust

Years ago, when I first read (Pixar co-founder) Ed Catmull’s brilliant book Creativity Inc, I remember really loving their ‘Braintrust’ idea. This is where a group of Pixar’s finest creative brains come together regularly to review outputs and provide candid, constructive feedback on films in development. Ed Catmull described at the time how the job of the Braintrust was to push beyond mediocrity and use radical candor to take a film ‘from suck to non-suck’. Catmull wrote that the Braintrust is valuable ‘because it broadens your perspective, allowing you to peer, at least briefly, through others’ eyes.’

Many people use AI as an answer engine. Ask a question, get the output. I’ve argued many times that it can be much more. Used thoughtfully, LLMs become a thought partner that takes you to places you couldn’t have got to alone. This matters most for strategy and innovation work, where value sits in the quality of the thinking rather than the answer itself, and where you have to stay the driver of the process rather than outsource judgement to the machine.

So I decided to create my own Braintrust to help me do this, and it’s become a regular part of my work. My hypothesis was that using a range of synthetic personas of renowned thinkers could help me to source different perspectives and challenge my own thinking on client projects and strategy. I began by trying to emulate Andrej Karpathy’s LLM Council which is a tool that the well-known AI researcher created where different LLM tools debate their answers to the same question to give you a better answer. Each model is fed the same question and responds independently. They then peer review each other’s answers, before a ‘Chairman’ reads all of it and gives you the verdict.

I soon realised however, that there were some flaws in applying this approach to sourcing different inputs from personas to inform a strategic process. The LLM Council gets its diversity from running the same question across genuinely different models but when you’re simulating a council with one model wearing six masks inside a single context window you have to work much harder to preserve the variance you need to get genuinely challenging outputs. Each persona response that gets generated within the same conversation will, in subtle ways, be conditioned on the previous responses. The risk is that what you get is performed disagreement rather than real disagreement because the model’s instinct is to be reasonable and to find common ground. By the time the Chairman synthesises, you’re likely to find that you’re summarising views that have already converged part of the way toward each other. The risk with the Chairman’s role in this context is that summarisation tends to produce the average of the views in front of it, which is often the least useful output, because the value of a multi-perspective process is precisely in the tensions or views that don’t reconcile.

My objective in setting up the Braintrust was to use a curated set of synthetic personas (expert thinkers, rendered with enough behavioural specificity to reason in their distinct way) as a structured way of opening up new perspectives on strategic challenges. The principle at the heart of it is that the process should be designed to preserve genuine intellectual variance, and force productive friction rather than to produce a consensus. The value of this comes from variance, not volume, so three carefully selected but deeply consulted thinkers will outperform six that are questioned in a shallow way. The personas act as scaffolding for distinct frames of reasoning, and the frames are what is actually doing the work. The aim is not to reach agreement but to make the irreconcilable parts of each perspective visible, so that you can uncover ways of thinking about a challenge that are new to you, or hidden by your own experience and bias.

I used Claude, which is my go-to for strategic work, and curated a deliberately broad range of personas in different categories that apply to my work, and that Claude could select from. It’s worth spending some time on list curation – you need enough to provide intellectual variety and weight, and perhaps to be able to cluster them around some key themes (although you don’t have to do this). I ended up with twenty key thinkers across transformation and change, advertising and marketing, AI and technology, and strategy. To make a good synthetic persona, the subjects needed an extensive corpus of work that the LLM can draw from but they also needed a unique intellectual approach or angle. Someone like Peter Drucker, for example, has a huge corpus but it’s so varied that the persona risked becoming generic management wisdom. A couple of others that I thought of had a significant collection of work but there was a level of abstraction about how they expressed themselves which somehow didn’t lend itself to clarity and uniqueness of perspective.

Remember that you’re using the personas as a way of giving you a different framing so distinctiveness matters. As an example, for my advertising cluster I have perspectives that give me instinct-led craft and research-led creative judgment (Bernbach, Ogilvy), commercial shrewdness (Wells Lawrence), behavioural insight and marketing science. The personas include sections on their intellectual foundations, things they reject, how they diagnose a problem, and notable angles on the topics I’m going to be using them for. Claude mapped out the tensions and points of complementarity between all of the personas for me (it built me a whole tensions matrix) and helped me make adjustments to ensure that the process best served the aim of productive variance. It was actually really interesting to look at these intellectual relationships. The strategy and transformation clusters of personas for example, generated higher levels of productive complementarity within-cluster but were particularly rich in question-level disagreement across the other clusters.

I then designed a four stage process that I could capture in a Claude Skill to make it repeatable and broadly applicable whenever I need new perspectives.

Stage one: Triage and selection. Claude reads the challenge and proposes a shortlist of thinkers from the persona library. I created a ‘quick lens’ section at the start of each persona so that Claude could do the triaging quicker. Each choice is justified against what the challenge specifically requires. The selection includes two or three obvious fits and one deliberately surprising choice, a thinker who doesn’t naturally belong but whose frame might cut across the conventional reading of the problem. This adds an entirely different, and unconventional framing. I then confirm or adjust the shortlist.

Stage two: Independent consultation. Each chosen persona is run in isolation, in a fresh context, with no awareness of the others. This is important to prevent cross-contamination between personas and to produce the cleanest possible expression of different angles. Each is given the full challenge and asked, before responding, to interrogate the framing of the question itself (what would they refuse to accept about how the problem has been set up, what would they reframe, what would they want to know that hasn’t been provided). And then they respond.

Stage three: Productive tensions. This is designed to find the most revealing tension and also to surface divergence and convergence by examining the relationships among the perspectives generated in stage two. I ended up (with Claude’s help) with five relationship types: direct conflict (incompatible answers to the same question), question-level disagreement (personas operating on different prior questions, so each makes the other’s answer invisible), asymmetric attention (engagement with different parts of the problem without contradiction), unexpected convergence (different starting points reaching similar conclusions), and shared omission (something all the consulted personas collectively skipped). The AI then engages with whichever relationship is most active, with the explicit instruction not to resolve. Forced confrontation is one option but only one. Question-level disagreement gets the prior questions named and laid against each other. Convergence gets tested for robustness or coincidence. A shared miss gets named and interrogated. In this way the most revealing relationship between perspectives gets laid bare.

Stage four: Sceptical review. Rather than a chairman who synthesises, the final stage is a critic whose explicit job is to identify where the personas converged too easily (and what shared assumption that convergence might rest on), what important framing none of them brought, and what the strongest objection is that no persona raised. The output is a short brief to me that names the sharp edges, identifies the trade-offs that are hard to reconcile, and flags the gaps that the council shared.

This is a structure deliberately designed to preserve variance and to generate genuine friction – the kind of friction that can reveal totally new ways of looking at a challenge. Rather than being a summary of views the output is a structured working document that contains the original framing critiques from each persona, the substantive responses, the productive variance and tensions, and the sceptical review. I do the integrating because integration is a strategic act and shouldn’t be outsourced to the machine. This way I stay as the driver of the strategic process but the AI is playing a key role in helping me to explore new and varied perspectives. I turned the whole thing into a Claude Skill, meaning that I can trigger the process whenever I want to (simply by using a relevant trigger phrase) and Claude will do the work in context automatically and present me with outputs.

My experience is that this technique is most useful for strategic questions where there might be an assumed framing, a need to ask a better question, or when it’s a complex challenge that would benefit from multiple perspectives to help navigate it. I’ve used it in client consulting work, prepping talks, and providing thought-starters for leadership workshops. Just recently I used it whilst prepping for a session I’m doing with the leaders of independent agencies on how AI will reshape their agency operating model and it gave me three unique angles I hadn’t considered (including that in the AI-world the real, lasting advantage that independent agencies have is senior judgement, applied to the problems that matter, faster). Every time I use it it comes up with something completely new to me that I genuinely hadn’t thought of.

The Braintrust is one example of something I think matters more broadly. The current sources of AI advantage, task-based automation and individual or team efficiencies, will top out. When everyone has similar tools doing similar work, differentiation disappears with them. The harder-to-copy advantage will come from deliberately designed practices that use AI to strengthen human judgement rather than substitute for it. Practices that broaden our perspectives, surface what we’d otherwise miss, and let us think more rigorously than we could alone. The Braintrust is one such practice. I suspect there will need to be many more.

A version of this post appeared on my weekly Substack of AI and digital trends, and transformation insights. To join our community of over thirteen thousand subscribers you can sign up to that here.

To get posts like this delivered straight to your inbox, drop your email into the box below.

Leave a Reply

Discover more from Only Dead Fish

Subscribe now to keep reading and get access to the full archive.

Continue reading