Posted on 

 by 

 in , ,

The age of the expert generalist

In 1936 a 20 year-old Claude Shannon came to MIT from the University of Michigan to work with Vannevar Bush on his differential analyser, an early, room-sized mechanical computer. This analog computing machine could solve differential equations through a series of rotating discs and shafts whose operations were governed by a bank of around a hundred electromechanical relays. Engineers working on the analyser knew from practice which combination of switches produced specific outputs but there was no formal theory of how to design or arrange them. A lot of it was based on trying things out and intuition.

Whilst at Michigan Shannon had taken an elective course in philosophy which had introduced him to the symbolic logic ideas of George Boole, a mid-19th Century mathematician who had developed an algebra for reasoning in which propositions could only be true or false. Boole’s work had existed for 80 years by the time Shannon learnt about it, but it had lacked a practical application. But when he studied switching circuits for the analyser machine at MIT, Shannon realised that a relay was like a physical embodiment of a Boolean proposition. It was either open or closed. Switches that were arranged in a series could behave like AND, and switches that were arranged in parallel could behave like OR. An entire circuit could effectively be designed as an algebraic expression with ones and zeros, creating a series of logic gates.

Shannon wrote up his ideas in his 1937 thesis ‘A Symbolic Analysis of Relay and Switching Circuits’, a piece of work which laid the foundation for digital circuit design and established the theory behind digital computing. Some have called his paper the most important master’s thesis of all time and the ‘birth certificate of the digital revolution’. Shannon became known as the ‘Father of the Information Age’ and went on to help found the whole field of Artificial Intelligence.

Claude Shannon’s skill in bringing together two previously unconnected concepts from different domains of knowledge is an excellent example of what the Artificiality Institute calls ‘multi-domain entanglement’, or the ‘braiding of different fields’. An idea which has long been central to creativity and breakthrough thinking. In The Act of Creation for example, Koestler wrote about ‘bisociation’, or how the creative act is a collision between two self-consistent but habitually unconnected frames of reference. Standard association involves making connections within a single, familiar framework but bisociation forces a collision between two incompatible or different realms. Bisociation can be a source of humour (juxtaposing or shifting abruptly from one context to another), art (like the Cubists bringing geometric architectural drawing together with traditional African masks), or even scientific breakthroughs (Archimedes in his bath and the measurement of irregular volume, or Kepler bringing together planetary motion and the physics of magnetism).

It’s a concept that has real value. The American sociologist Ronald Burt came up with his theory of ‘structural holes’ after his research showed that people who bridged otherwise unconnected groups in a social network generate disproportionately more good ideas, do better in performance reviews and get promoted faster. Their competitive and creative advantage comes from the arbitrage of information between groups that generally don’t talk to each other.

Similarly, in his book ‘Range: Why Generalists Triumph in a Specialized World’ David Epstein explores the decision-scientist Robin Hogarth’s idea of ‘kind’ and ‘wicked’ learning environments. Kind environments (like golf or chess) have clear rules, are more predictable and are characterised by immediate feedback. Wicked environments (like rapid tech-driven change, the current geo-political situation in many parts of the world, the stock market, our career planning, or starting a business) have shifting patterns, changing rules, hidden information and delayed feedback, thus making past experiences less useful. Narrowly focused specialists thrive in kind environments but generalists are better equipped for wicked environments because their broader, cross-disciplinary experience enables individuals to draw from a wider toolkit, adapt well to new situations, and to connect seemingly unconnected ideas.

In the video I linked to earlier Helen Edwards makes the point that LLMs know every field but they know them separately and they can’t pull on these different fields in the same way that a human can. She uses the example of the architect who holds different aspects of their work simultaneously – the budget, the structure, the light, the regulation, the client’s unsaid wants. An AI model can give you each piece and a small element of crossover based on words and text but it can’t give you the meaning surrounding a particular situation. Pulling the work apart like an AI does kills that meaning. It’s human imagination that can make the meaningful connections between different domains in ways that create something new in specific context. Helen calls this becoming a next-level or ‘expert generalist’, something she notes is common in academia but surprisingly rare in business: ‘…once you can hold two fields at once, you start seeing things that neither specialist can’.

Epstein also makes a related point about what he calls ‘match quality’, this being the fit between what you do and who you actually are. He shows that people who ‘sample’ more (as in trying many different things out) before they commit to a specialism end up with a better fit. The short-term earnings penalty that results from trying things out then reverses into much greater advantage as the benefits of a good fit compound over a career. When we look for connections from adjacent or even unrelated domains we’re effectively sampling. AI has made it much easier to become conversant enough in an adjacent field and it has also collapsed the cost of doing so. So the constraint has shifted. What’s scarce now is the judgement needed to select the right domains across which connections may bring new value, and to know which information within that domain can generate new meaning within your own field of interest – the pairing that can create real value.

An increasing range of businesses, roles and disciplines are operating now in wicked environments but there are some roles where multi-domain entanglement is an essential part of the job. Strategists, for example. A couple of years ago the annual WARC Future of Strategy survey (always an interesting read, and the latest one is now open for contributions) found that planners needed to master an unprecedented number of (up to 14) different areas and disciplines to do their job effectively. This places a huge pressure on planners to become multi-domain specialists (Michael Lee talked about this in his Google Firestarters episode). But the best strategists that I’ve worked with have been expert generalists who know that they don’t need to know about every related domain in depth. They just need to know enough to be able to hold different domains in tension with each other and recognise how they can connect them to generate new meaning.

Tey Bannerman has described four distinct ways of thinking with AI (thanks to Piotr Bombol for the heads up): compression, for when you’re overwhelmed or needing clarity; expansion, when you’re exploring or need options; reflection, for when you need to challenge or check your own thinking; and execution, when you just need to get something done. Most organisational AI right now sits firmly within compression and execution, but expansion and reflection are where real growth opportunities lie. Yet few are being taught it. In Hogarth’s kind environments where the conditions are stable and feedback is fast, using AI to compress or to execute makes a lot of sense. But in wicked environments, where the context is changing rapidly and feedback is harder to get, we need something else.

Psychologists Daniel Kahneman and Gary Klein spent years arriving at different conclusions about expert intuition. Klein studied firefighters and nurses who showed remarkable judgement, Kahneman studied experts whose intuition was consistently poor. Rather than argue it out in public they worked together adversarially and concluded that the difference lay in the environment rather than the expert. This idea of adversarial collaboration, deliberately looking for where you disagree or what you’ve missed, is exactly how we need to work with AI in wicked environments.

This involves techniques that take you beyond using AI as a research assistant. One of my favourites is norm or constraint switching. Here you take the rules or restrictions from one category or discipline and apply them to your own to open up entirely new ways to think about your processes or supposition. AI is pretty good at analysing your own thinking for gaps, assumptions, or the impact of variables on a situation and so a habit of adversarial prompting builds a habit of deeper thinking. I love using synthetic personas to open up new perspectives or to see how a completely different profession (a comedian, a lawyer, an architect) would look at a situation. Setting up persona debates can expose weaknesses in arguments or angles that you haven’t thought of because it forces disagreement and counter-arguments. Creating an AI brain trust turns this into an automated perspective generator. Looking at how adjacent or different domains have dealt with challenges that are structurally similar to yours (structural analogy generation) is really helpful whenever you get a bit stuck. Transposing typical failure or risk scenarios from another topic area can identify hidden vulnerabilities or unappreciated strengths that you may have missed.

What’s common with all these techniques is that none of them are asking the AI for the answer. They are instead all designed to help you, the human, think better. You are using the AI for what it is brilliant at – structured thinking and assimilating huge amounts of information. Shannon’s pairing came to him almost by accident, a philosophy elective and a job rewiring relays, but ours doesn’t have to. Breadth used to be expensive but it’s now almost free. But it’s the expert generalist’s judgement that sets up and turns these cognitive collisions into entirely new value. And no AI model can do that anywhere near as well.

A version of this post appeared on my weekly Substack of AI and digital trends, and transformation insights. To join our community of over thirteen thousand subscribers you can sign up to that here.

To get posts like this delivered straight to your inbox, drop your email into the box below.

Image: Tekniska Museet of Sweden, Item 43069 via Flickr, CC BY 2.0 via Wikimedia Commons

Leave a Reply

Discover more from Only Dead Fish

Subscribe now to keep reading and get access to the full archive.

Continue reading