Seven lessons from building 50+ AI automations

Executive overview

Most AI automation projects fail not from bad technology, but from bad priorities, vague processes, and no definition of success. The fix is applied before you write a single line of code.

Start with the highest-impact bottleneck, document it fully, define binary success metrics, then prove feasibility with a top-tier model before optimising.

Prioritise by impact, not interest

• Identify the business goal and set a time horizon (12–18 months).
• Map the bottlenecks standing between current state and that goal.
• Rank bottlenecks by impact — time saved, revenue unlocked.
• Decompose the top bottleneck into its sub-activities.
• Determine which activities are fully or partially automatable.
• Automate the highest-impact subset first.

Document before you automate

• A vague process description produces a wrong automation.
• Require documentation from the client before starting work.
• If they lack bandwidth, run a paired documentation session side-by-side.
• Reverse-engineering a process mid-build wastes time and risks harm.

Define binary success metrics

• "If you can't measure it, you can't improve it." — Peter Drucker
• Set quantitative success criteria upfront, after prioritising and documenting.
• Make metrics binary (pass/fail), not rated on a scale.
• A 1–5 rating obscures whether AI output is actually acceptable; binary does not.

Start with the largest model, then scale down

• Avoid open-source models at the start — hosting overhead masks performance problems.
• Prove the automation is possible with the best available commercial model first.
• Once proven, incrementally step down to smaller, cheaper models.
• Most small and medium businesses don't need open-source infrastructure.
• Focus on business value, not what is technically interesting.

Use AI only when it is actually necessary

• Clients often arrive with an inflated view of what AI can do today.
• Ground conversations in the business and its bottlenecks — not in AI.
• When the problem is identified without reference to technology, the right tool becomes obvious.
• Sometimes the answer is not AI.

Keep automations simple

• A PDF-extraction automation built with chained models and ~1,000 lines of regex reached 83% accuracy after weeks of work.
• Replacing the entire system with a single Gemini 2.0 call took one hour and reached 97% accuracy.
• Always test the simplest possible version first.
• For extraction tasks, a single model usually outperforms elaborate pipelines.

Set hard cost limits

• A single overlooked edge case burned $100 in API costs in under an hour.
• Use provider-level hard caps where available (e.g. Claude, OpenAI dashboards).
• Where provider caps don't exist, implement token tracking and cost cutoffs in code.
• Emit alerts to yourself when a limit is hit so you can debug before costs compound.