AI Error Checking for a New Google IDE

Zero-to-one research on an AI feature for a high-priority, high-visibility launch

At a glance

Role: Lead UX Researcher

Team: 2 Product Managers, Staff Interaction Designer, UX Engineer

Methods: Concept evaluation, task-based usability testing, 1:1 remote interviews with 12 Google engineers

Impact: 65K+ weekly active users; now a default extension

Overview

Google was building a new web-based IDE with AI assistance, extensions for customization, and major UI improvements. Google’s quarterly engineering satisfaction survey had already shown that checking code for errors is slow and breaks developer flow. The team proposed an AI Error Checking extension to address this, and I led research to make sure it delivered real value before launch.

Challenge

This was a brand-new product with high priority and visibility. It was a zero-to-one launch, and it introduced AI features to developers for the first time.

Goal

How can we deliver a strong value-add that boosts productivity, in a user-friendly way, while letting developers stay in the “flow state” of development?

Research goals

  • Understand how AI can automate mundane, rote tasks to improve developer productivity

  • Evaluate two different designs of the AI Error Checking feature to align product development

Round 1: Concept evaluation

  • Method: Qualitative concept evaluation of an in-line design that surfaced suggestions on screen as users wrote code, plus 1:1 remote moderated in-depth interviews

  • Participants: 6 Google full-time software engineers (6+ months tenure) with a mix of development types, technical levels, and IDE usage

  • Findings: Developers liked the concept, but the in-line design interrupted their coding flow and had usability problems.

  • Recommendation: Fix critical design and usability issues before launch, and weigh the risk of launching a premature feature on a high-visibility product.

  • Outcome: The research shaped product prioritization. The team postponed the initial launch to Beta users and redesigned the feature.

Round 2: Task-based usability study

  • Method: Qualitative task-based usability study of a redesigned feature that let users accept AI-generated suggestions at the end of their session, plus 1:1 remote moderated interviews

  • Participants: 6 Google software engineers: 3 using the legacy Google IDE and 3 using the new one, with a mix of development types, technical levels, and programming languages

  • Findings: Participants saw the feature as a value-add to their workflows. They liked being able to finish coding first and then choose whether to use the AI, and the feature was clear and easy to discover.

Impact

  • Launched to Beta users of the new IDE, with over 10% adoption (1,000+ users) within 2 weeks

  • Featured as “Extension of the Month” in the IDE newsletter and used as a selling point for adopting the new IDE

  • Became one of the most-used extensions, with 65K+ weekly active users (80% usage), and is now a default extension supported by the IDE team

Learnings and reflections

  • Next time I’d add a quantitative measure, such as the System Usability Scale (SUS), to track improvement across design iterations.

  • Building and keeping stakeholder trust and buy-in is critical to product success, especially when the recommendation is to delay a launch.

  • Advocating for the user protects long-term product success.

Next

Building a Research Program for the Federal IDR Platform