What happens when AI starts doing the work your engineers used to do — and what does that mean for quality?
In part two of our first installment of the “Q&A with QA” series, we continue our conversation with Pete Draheim, Director of Quality and Engineering Tools at Best Buy. This half focuses on AI – how Pete’s team is driving roughly 30% efficiency gains across their engineering organization, why unit testing is the sweet spot, and what it really takes to get engineers to stop fighting the robot and start using it as a tool.
How do you see AI changing your role and your team’s role?
We’re responsible for the implementation of AI within how we develop our software and push business value, and we’re leading the way in that. It’s absolutely fascinating. We have an AI/ML team at Best Buy that focuses on delivering AI solutions for business, but they don’t have bandwidth to develop AI solutions for engineering functions. We take that over.
We’re not building models—the tools and technologies available on the market are amazing and have changed dramatically over the last year. We’re basically a GitHub Copilot shop. My team has become the experts internally, and we teach these concepts out to engineering teams. It’s fun to see teams leapfrog each other and us in their understanding of how to use AI to improve the software development process.
For testing specifically, we’re a little behind. I talked to my boss about a proposal to use an AI testing-focused tool as opposed to just the coding agent. We might do something later in the year. We are using GitHub Copilot for functional and unit testing, and unit testing is a sweet spot for AI. It’s going to change everybody—the software development lifecycle, the process, what engineers do daily. It’ll become more about managing your AI agents and less about doing your own work.
Where does AI add the most value, and where do you see gaps that still need human intervention?
We’ve pushed that decision down to individual test engineers. The contexts are too different to have the same strategy everywhere. What you automate versus what remains manual is based on the return on investment of the test case. In some cases, we’re going to do a test we may never run again—why automate that? There are edge cases and parts of code you want to verify that probably will never change.
I think AI is going to push us toward making lower-level tests at the unit and API level so much easier that our coverage will increase. This will take pressure off functional testing. Our functional testing will actually decrease, automation of functional testing will remain the same, and the manual part will cut down. So your percentage of automation is going to increase, but that doesn’t mean you’re automating more tests—you just need to run fewer tests. That’s the operating theory I’m going under as AI rolls through the development space.
What advice would you give an organization trying to mature from a testing-focused approach into true quality engineering?
That gets back to proactive versus reactive quality. How you write code, how you integrate code, how you scan for code—both for quality and security—how you build and deploy the application, all those CICD processes are important. The better and faster we do those, the more cycles we can get through and the higher level of quality you’ll have.
One principle is working in small batches. Making smaller changes and testing those smaller changes more frequently allows you to achieve both higher velocity and higher quality. You get both if you do it right. We’re seeing AI increasing a team’s velocity by 30% or higher just by removing lower-value tasks and more mundane tasks like writing documentation, writing tests, executing tests.
The ability to be proactive and shift left really reduces that reactive quality testing piece. We have some things we still need to learn in the testing space, like how to make our testing quicker, because we’re going to keep manual testing around. I firmly believe you’re always going to want to keep a human in the loop, but we’re going to minimize it.
Can you elaborate on that 30% efficiency gain you’re seeing with AI?
We have seven AI pilots I’m co-leading and monitoring, looking at different use cases. Sometimes we’re looking at Java upgrades—can AI do a Java upgrade basically hands-off? We go from taking three weeks to three hours. I say 30% because I’m averaging everything down. Some things used to be 120 labor hours and are now three—that’s a lot more than 30%. But if I take the totality of what engineers do, it’s roughly around 30%.
There are also things we chose not to do or didn’t do well that AI makes easy. Unit testing is the key. Before AI, some teams’ unit testing wasn’t always as consistent as you’d like. When push came to shove and we got behind, we’d cut unit testing out. AI enables us to always have that there at zero cost to the engineer. So we can both get faster and get better.
That 30% is really not digging into the agent-based approaches that are coming out. It’s really within the IDE, offloading the more basic tasks to AI, which enables each engineer to get about 30% faster.
What’s critical for human beings adopting AI technology?
The human beings have to embrace the AI technology and effectively learn to use it to their advantage. If you have team members that are resistant, you’re not going to get that value. Each person, each engineer has their own version of a learning curve. They have to convince themselves that there’s actually value.
There’s a little bit of a human-versus-robot mentality that takes over. You have to be able to say: the robot’s a tool. It’s not replacing you, but it is going to replace some of the things you do. So how do you effectively use that tool?
I think of all the saddle makers that employed thousands of people in the 1890s and early 1900s. When the automobile came by and we didn’t need as many horses, all those people got put out of work. Rather than making saddles, you need to make automobiles. Or think about before oil was pumped from the ground—we had thousands of whaling ships that would harpoon whales for their oil. I think of all the industries that have changed over time and how dramatically they’ve changed. This is actually pretty mild compared to horses versus cars and whale oil versus crude oil.
What’s one lesson you learned the hard way about scaling quality in a large enterprise?
I don’t know if you can scale quality. Could I say I’ve scaled quality at Best Buy? No, I don’t think I could say that. Quality is all of these functions—the proactive functions and reactive functions: CICD, version control, small batches, unit testing, API-level testing, code quality scanning, and all the different levels of testing. It’s all of that, scaled to everyone consistently.
We’re not there. We have 100-plus different engineering teams, probably close to 200. We have some that are probably C-minus and some that are solid A’s. Just like you have your natural curve in school, you have that in engineering teams. How do you bring the poor-performing teams—who probably don’t even know they’re poor-performing and think they’re doing a great job—up to a level you see a different team at?
You focus on capabilities: AI adoption, version control, consistently integrating and building and deploying code every day. Not necessarily to production, but you should be doing it every day. I don’t think we have all teams there yet, and the transparency across it—I don’t necessarily know who is and who is not.
We’ve done something we’ve never done in the past 15 years I’m aware of: measure on 25 or 30 different aspects how an engineering team is doing. Number of defects escaping into production, security vulnerabilities assigned to each team—those are all functions and features you need to holistically look at.
You don’t want a team to abandon quality for velocity—they’re super fast but none of their code ever works. Or the other way—their code’s always perfect but they’re six weeks behind everybody else. How do we balance that? How do we provide that information to the right leaders to make the changes that may be required? Maybe a team needs to slow down for six weeks while they build a continuous integration framework. We might take a momentary slowdown to achieve higher velocity at the end.
And of course, AI makes all this so much easier. What would take you six weeks before, you can now do in six days with respect to continuous integration and test automation.
———–
A huge thank you to Pete for taking the time to share such candid, hard-earned insight from 15 years on the front lines of enterprise quality. If this conversation got you thinking differently about how your organization approaches QA, stay tuned – this is just the beginning. Next up in our “Q&A with QA” series, we’ll be bringing in more voices from across the industry to keep the conversation going.