I just got key account manager for a medium-sized software vendor when we faced the challenge of creating an approach for agile usability testing. My client had changed their development methodology to SCRUM a few months before (although they are pretty innovative, they were quite late with that).

They shook up their company to be more collaborative, responsive and fast. Most of their UX research was done by service providers which led to the problem that the user research measures were not able to match the new speed of the development teams.

Back then, most of our clients’ usability testing efforts were one-off projects at a very late stage in development. Dev teams built their solutions without any feedback loops until they were almost finished and wanted to get quick approval with one final usability test. Of course, that very rarely worked out and as a UX consultant, it was very frustrating to enter projects in those late stages since time constraints permit only minor adjustments so close to product launch.

At this point in time, the client already had a dedicated user research team that took care of the groundwork and had senior management on board for regular UX testing. As the client’s key account manager at GfK, I was brought in to consult on agile UX and trigger the organizational changes needed for agile UX.

We sat down and had a look on what caused the delays in research projects. We were able to identify three factors that lengthened the projects’ durations:

• Purchasing process: Organizational obstacles lead to 2–5 days of delay. We wrote a customized proposal each time a project request came in and our client had to issue an official purchase order before we were allowed to start working.
• Participant recruitment: Sometimes we had to plan with three weeks due to the client’s very specific target groups and the traditionally larger sample sizes of around N=10.
• Reporting: We had spent around five days on average to report findings from our usability tests.

We decided to use a fixed hourly rate for the agile UX work of our consultants. Purchase orders contain a fixed contingent of hours that the client can request at any time to be more flexible. After a little back and forth with our procurement and finance departments, writing proposals and waiting for POs were removed from the list of delays.

Reducing the time needed for recruitment was a little trickier. In the past years, we had already streamlined all the processes for participant recruitment and the client had set up a customer database. We had also established online usability testing. We no longer invited participants to our lab but set up a screen sharing tool (that allowed the user to remotely control the cursor) and called them for the interview. In our experience, remote testing works absolutely fine for most software products and it makes the recruitment for hard-to-reach target groups a lot easier.

The only lever we had left was sample size. Since we were about to shift to iterative testing anyways, we didn’t hesitate to reduce the average sample size to N=5.

For the topic of reporting, we decided to try one of these crazy new approaches (user-centricity 😉 ) and actually talked to the dev teams about what they need in an agile environment. It turned out that most of them are absolutely fine with or even prefer a very lean documentation. We went from detailed 50-page reports to very lean workshop protocols.

For our agile usability testing testing approach, we made it mandatory for the team to watch the interviews collectively from a conference room. We planned breaks of 30-45 minutes in between interviews and instantly discussed and summarized the observations with the team. At the end of each field day, the findings were prioritized by the team.

Moreover we included the development teams in the testing process to enhance their commitment. In the first iteration of a project, the UX consultant trained the dev team for the following iterations. The team’s UX designer or another team member who stepped forward (not all dev teams had a dedicated designer) had to learn some UX basics and how to write the recruitment screener and session guides. Once the training phase was finished, the researcher’s work would be reduced to

• feedback loops on the screener and session guide,
• conducting the online interviews, and
• moderating the collaborative analysis with the team.

We now had an organizational framework and a basic structure for our agile projects. We were excited to test it in practice and see where we would need to make adjustments. I managed the first project myself to test the agile format with the client and joined a dev team.

I worked with a very open-minded team that was really willing to make the agile usability tests a cornerstone of their development efforts. Especially the very ambitious UX designer helped to make the project a success. She took over the moderation of the analysis in between sessions as it turned out that it’s a little too much for the interviewer to do both. With that team, we tested and iterated their product in over ten cycles.