After thousands of interactions with ChatGPT, patterns emerged — some exciting; others worrying. At times, AI acted as a true thought partner, helping refine ideas and solve problems more effectively. Other times, like many users, I fell into automation bias — accepting AI-generated answers without questioning them. Curious about this dynamic, I analyzed 2,500 of my own conversations with ChatGPT to answer three key questions:

• How do we use AI in decision-making?
• Where do breakdowns occur?
• How can we work with AI effectively without losing critical thinking skills?

The AI Decision Loop: A Smarter Way to Use AI
 

Through this study, I developed a structured approach for maximizing AI’s potential while avoiding automation bias. The AI Decision Loop is a five-step process designed to help users get better, more reliable results from AI:

1. Frame the Decision Context: Clearly define the problem, constraints, and assumptions.
2. AI Output Generation & Thought Partnership: Treat AI as a collaborator, not a magic box.
3. Apply Human Judgment: Challenge AI’s responses and ask for justification.
4. Verify & Validate: Fact check AI-generated insights, especially for high-stakes tasks.

5. Refine & Iterate: Learn from the interaction, improve your prompts, and automate selectively.
    

   

This structured approach mirrors techniques such as Pair Programming, where iteration and refinement lead to better results than a one-and-done approach.

AI Decision Loop Examples

Standard Example: Developing a Product Roadmap for an AI-Driven Note-Taking App

Step 1 – Frame the Decision Context & Guide AI Collaboration
Human Prompt:
“We’re building an AI-powered note-taking app for busy professionals. Our business goals include high user engagement, seamless integration with productivity tools, and scalable monetization. Please outline three distinct product roadmap options that balance technical complexity, market timing, and resource constraints.”

Step 2 – AI Output Generation & Thought Partnership
AI Response:
“Option 1: Core-Feature First: Focus on essential note-taking, search, and basic integrations before expanding into advanced AI features.
Option 2: AI-Driven Innovation: Launch with advanced functionalities like auto-summarization and contextual suggestions, followed by phased integration development.
Option 3: Hybrid Rollout: Combine a core feature set with a simultaneous beta for advanced tools to gather early feedback.”

Step 3 – Apply Human Judgment & Challenge AI’s Perspective
Human Follow-up:
“For Option 2, what assumptions are being made about user readiness for advanced features, and what technical challenges might extend our timeline?”

Step 4 – Verify Outcomes & AI Reliability Check
Human Verification Prompt:
“Compare these roadmap options against market research and technical feasibility studies. Identify any potential biases or overlooked risks, and verify assumptions using historical data.”

Step 5 – Refine, Iterate, and Automate Where Possible
Human Iteration Prompt:
“Based on feedback from our technical and market teams, refine Option 2 to include a phased rollout with pilot testing of advanced features. Document the revised process for future product roadmap decisions.”

Low-Risk Example: Crafting a Social Media Post for an Upcoming Webinar

Step 1 – Frame the Decision Context & Guide AI Collaboration
Human Prompt:
“We need a concise, engaging social media post to promote our upcoming webinar on digital marketing trends. The tone should be friendly and professional, and the post should include a call-to-action to register.”

Step 2 – AI Output Generation & Thought Partnership
AI Response:
“Join us for an exclusive webinar on the latest digital marketing trends! Discover innovative strategies and insights that can boost your business. Register now to secure your spot!”

Step 3 – Apply Human Judgment & Challenge AI’s Perspective
Human Follow-Up:
“That’s a solid start. Can you offer two alternative versions that slightly vary the tone — one more upbeat and one more informative?”

AI Follow-Up Response:
“Upbeat: ‘Ready to supercharge your marketing game? Don’t miss our live webinar on cutting-edge digital trends. Join us and get ahead — register today!’
Informative: ‘Explore the evolving landscape of digital marketing at our upcoming webinar. Learn practical strategies and insights to drive your business forward. Register now for a deep dive into the latest trends.’”

Step 4 – Verify Outcomes (Optional for Low-Impact Decisions) 
Note:
For a low-impact decision like a social media post, you might choose to skip an extensive verification step since any minor oversight is unlikely to have significant consequences.

Step 5 – Refine, Iterate, and Automate Where Possible
Human Iteration Prompt:
“Let’s finalize the post with an upbeat tone. Slightly refine it to emphasize the date and time of the webinar.”

Final AI-Generated Post:
“Ready to supercharge your marketing game? Don’t miss our live webinar on digital marketing trends, happening on [Date] at [Time]. Join us to get ahead — register today!”

Key Insights

Analyzing my own AI usage revealed surprising trends:

• 34% of the time, I stopped after Stage 1, highlighting a potential automation bias, where I accepted AI’s first response without deeper engagement.
• 66% of the time, I continued beyond Stage 1, engaging in deeper iteration and refinement rather than treating AI as a one-shot answer machine.
• 59% of engaged conversations completed the full AI Decision Loop, reinforcing that structured collaboration with AI leads to more refined results.
  • 74% of these that had AI partnerships resulted in successful completions, compared to just 26% for non-partnered engagements, proving that AI is most effective when treated as a thought partner.
• 63% of the time, I used AI for decision-making working with it as a co-pilot rather than letting it take the wheel. 
  • 73% of AI-involved decision-making sessions succeeded, reinforcing that AI works best as a co-pilot rather than an autopilot.
• Only 34% of the time did I ask AI to critique its own output or surface missed opportunities for deeper analysis.
  • 98% of AI-assisted critiques led to successful refinements, showing that critical iteration — rather than passive acceptance — significantly improves outcomes.
• Although these structured engagement behaviors strongly correlate with higher loop completion rates, further research is needed to determine whether loop completion itself directly improves output quality. However, the high success rates in AI partnership (74%) and AI critique (98%) suggest that structured collaboration may lead to more refined results.

The Strange Loop of AI-Assisted Thinking

AI-assisted decision-making works like a strange loop, a self-reinforcing cycle where thoughts build on themselves. The more structured your engagement, the smarter the loop becomes. But if you accept AI outputs passively, the loop collapses, and AI becomes an automation crutch instead of a thinking tool.

• Engaging in structured iteration makes us smarter.
• Skipping steps turns AI into an automation engine, reducing critical thinking.

This creates a paradox: AI makes iteration easy, but it also tempts us to shortcut the process, reinforcing automation bias.

The Takeaway: AI as a Thought Partner, Not an Automation Engine

AI doesn’t make us smarter, but structured engagement does.

• AI is a force multiplier, not an autopilot.
• Its true power lies in adaptive collaboration where human reasoning and AI-generated insights refine and improve each other.

If we want to maximize AI’s potential in knowledge work, software development, and decision-making, we must shift from passive AI consumption to active, structured collaboration. The AI Decision Loop trains structured thinking, turning AI from an output generator into a thought partner.

Read the Full Case Study and Explore the Data

For a deeper dive into the methodology, findings, and technical details, check out the full case study and git repo with the code used to do the analysis. And for more on “Strange Loops,” see Chapter Two of John Horgan’s Mind-Body Problems.