Balancing efficiency and serendipity for business innovation

Executive overview

Optimising for efficiency through AI and data patterns locks organisations into the recent past. Algorithms are good at recognising what has worked — they are poor at discovering what comes next.

The core insight: structured efficiency and deliberate serendipity are not opposites — organisations need both, and must actively protect time and space for each.

The efficiency paradox

• AI pattern recognition reflects the recent past; markets shift faster than the patterns update.
• Hiring or selecting for proven fit produces homogeneity — the organisation loses the ability to adapt.
• Diversity failures in AI systems are a symptom of a deeper structural problem, not an isolated bias issue.
• Algorithmic optimisation accelerates convergence toward the same customers, products, and people.

The Thalberg model: balancing intuition and discipline

• Thomas Edison controlled film production through a patent trust; his obsession with cost efficiency caused him to miss the rise of feature films.
• Visionary directors like von Stroheim had no budget discipline and destroyed their own projects.
• Irving Thalberg held both simultaneously: market intuition (pioneered preview screenings), financial discipline, and willingness to invest more to earn more.
• Thalberg started as a private secretary with no professional film background — cross-domain entry was the asset.

Structured time versus free play

• A highly structured day is not the problem; the problem is leaving no unallocated time within it.
• Reserve explicit time for free exploration — treat it as a protected calendar block, not leftover time.
• Sources outside your industry (museums, humanities, non-business newsletters) surface ideas that sector-specific reading cannot.
• J.K. Rowling's Harry Potter series looked like creative inspiration; it was built on exhaustive diagramming and pre-planned grids before a single publisher saw it.

Cross-domain talent and internal mobility

• Michelin's steel-belted radial tyre originated with a graphic artist moved into tyre R&D — his geometrical imagination, not rubber engineering, was the breakthrough.
• People in different departments often carry transferable talents that are invisible to efficiency-oriented hiring processes.
• When solving a constraint, engaging the whole team — not just leadership — surfaces approaches that stay hidden in top-down analysis.

Desirable difficulty and analog thinking

• Desirable difficulty: making a cognitive task harder (e.g., handwriting notes) produces deeper comprehension than fluent, frictionless processing.
• Longhand note-takers outperform fast typists on comprehension tests — paraphrasing forces understanding; verbatim capture does not.
• A printed page functions as a spatial territory; readers form a different mental map than when scrolling the same content.
• Analog and digital modes are not in competition — each retrieves different kinds of understanding from the same material.

Web access and the democratisation paradox

• The web made chess mastery far more accessible globally; it simultaneously eliminated the income that supported professional players.
• Greater access to skill-building does not translate to greater reward for those skills — supply expands faster than demand.
• The same dynamic applies across knowledge work: easier access to information raises baseline competence while compressing the advantage of expertise.

Applying this to business scaling

• Identify constraints or blockers clearly, then hold them in mind while seeking inspiration from unrelated domains.
• Avoid solving the constraint by brute-force analysis within the same frame — let cross-domain exposure surface the mechanism.
• Build intentional variation into hiring, sourcing, and process design — not as culture work, but as a strategic hedge against pattern lock-in.