AI System That Generates YouTube Ideas From Competitor Comments

Executive overview

Most creators generate ideas from personal interest alone, leaving audience demand on the table. This system scrapes comments from competitor YouTube channels, clusters questions into themes, ranks them by engagement signals, and surfaces scored video ideas in a Google Sheet. The core insight: audience questions are a free, real-time demand signal — automating their extraction turns viewer curiosity into a prioritised content pipeline. The build uses Apify scrapers, Gemini 2.5 Pro for analysis, and Google Apps Script to keep it self-contained with no custom database or UI.

System architecture overview

• Input: a Google Sheet lists 6–8 competitor or admired YouTube channels
• Apify "Fast YouTube Channel Scraper" pulls the most recent 3 video URLs per channel
• A second Apify actor scrapes comments from each video (up to ~1,000 comments per run)
• Comments are passed to Gemini 2.5 Pro in chunks of 250 for theme extraction
• Gemini identifies recurring questions, groups them by theme, then scores each theme
• Scoring weights: frequency of question (50%), upvotes (30%), reply count (20%)
• Final output: topic, idea description, suggested titles, and intent score written back to Google Sheets
• Everything runs in Google Apps Script — no separate server, database, or UI required

Output format in Google Sheets

• Column A: idea description derived from the dominant question theme
• Column B/C: multiple AI-generated video title options, with one selected as the strongest
• Columns D–G: intent score, frequency count, vote count, reply count
• Top-ranked idea from one run: "CAG vs RAG — will RAG become obsolete as context windows grow?"
• The sheet also has a logs tab for Apps Script debugging

Build process: spec → blueprint → to-do list

• Start with research: use Perplexity or OpenAI o3 to gather docs, API references, and model details before writing anything
• Explicitly include the target model (Gemini 2.5 Pro) in research notes — AI coding assistants default to older models they were trained on
• Feed research plus intent, user, and job-to-be-done into an interview prompt with o4-mini
• Interview constraint: "ask one question at a time, each building on the previous answer" — prevents AI from dumping 10 questions at once
• Expect 15–20 back-and-forth turns; if you don't know an answer, ask the AI to suggest options
• Paste the finished spec into Gemini 2.5 Pro (chosen for high token output) to generate a blueprint
• Blueprint prompt instructs 2–3 rounds of progressive detail — high-level phases first, then finer sub-steps
• Final blueprint section produces phase-by-phase system prompts ready to paste into Cursor
• After blueprint is complete, request a to-do.md checklist with checkboxes per phase

Iterative coding with the to-do list

• Paste phase-1 prompt into Cursor (Claude 3.7 Sonnet or Gemini 2.5 Pro); build and test
• After each phase succeeds, return to the to-do list, check off completed items, then proceed to phase 2
• Alternating between the strategic checklist (high-level) and active coding (low-level) prevents the model from drifting off-plan
• This "forest and trees" discipline is the main reason complex multi-phase builds stay on track

Key tool and model choices

• Apify: cheapest and most reliable scraper for YouTube channels and comments; use actor documentation as context in your spec
• Gemini 2.5 Pro: preferred for blueprint generation (high output token limit) and comment analysis (long context + cheap reasoning)
• o4-mini: fast reasoning model suited for iterative spec interviews
• o3: used for deep research with simultaneous web search and reasoning
• Google Apps Script: keeps the tool zero-infrastructure — no hosting, no database, no auth complexity for a personal proof of concept
• Cursor: IDE with AI coding assistant where phase prompts from the blueprint are executed

Replicating or adapting this system

• The interview and blueprint prompts are from Harper's publicly available blog process (with personal modifications)
• The two Apify actors needed: "Fast YouTube Channel Scraper" and "YouTube Comment Scraper"
• Copy actor documentation into your spec so the AI has accurate API context
• Adjust the comment chunk size (250) and scoring weights (50/30/20) based on your volume and priorities
• Swap in your own channel once it has sufficient comment volume; until then, use competitor channels in the same niche