User testing on the cheap 

User testing is the cornerstone of every successful project. And, say digital officers Charlotte Brewer and Geraint Northam, it needn’t break the bank. 

User testing tends to be thought of as a lot of work. Planning, finding volunteers, making sure they turn up, preparing and managing the session, reporting and summarising it afterwards – all take significant time and effort. 

It’s worth it. As a digital team we want to find problems, solve them, and improve the user experience. This then allows the University to meet its business objectives. 

But does it have to be so time-consuming? We’ve started to embrace ‘user testing on the cheap’ – quick, small tests done often. 

What is user testing ‘on the cheap’? 

The basic premise is that instead of setting up a big workshop or laboratory test, you just approach people and ask them to test something. 

You still have to plan in advance, making sure questions are fair and are not leading, and that they are the right questions. But the whole process takes much less time than traditional methods. 

For example, recently we took part in a workshop which involved diving headfirst into some user testing with only 20 minutes prep time. Before we knew it, we had been thrown out onto the street and were asking random strangers to help us test a charity website. The experience was terrifying, but half an hour later we had returned to the workshop with lots of ideas about how we could improve the website. 

Surprising insights 

By user testing in this way, you can see what users do and how easy or hard they find it. 

We found several surprising insights during our various user testing: 

• Users preferred using a different system to the one we were developing. 
• Users downloaded the wrong form, and even worse, were confident it was the correct form.  
• International students didn’t know common English terms, such as ‘GP’. 

Small tests can produce useful results 

This graph, courtesy of Ida Aalen, shows the probability of spotting usability problems. Five user tests will uncover 80% of usability problems, while just eight will uncover over 90% of usability problems. 

 

How little you need to do to get a very good idea of how usable your website is. 

Tips on how to run user testing on the cheap 

Be prepared 

Even though this is ‘guerrilla’ testing you still need to be well prepared. Select relevant questions and tasks, test them on your colleagues, and charge your laptop before you head out. Take a colleague if possible (one talking, the other taking notes), and buy each other a coffee before you begin the user testing session. 

Find a location that works for you 

This might be standing on the street or making use of libraries, cafés or study centres. Get a feel for some different options. The best location for one study might be different from the one for another study. 

A good location can really help. We found the University library café to be particularly good as students were on a break and more receptive to being approached. Another good location was where students were leaving a study area. 

Keep asking 

One of the hardest parts of guerrilla user testing is randomly approaching people. This can be scary. Practise your opening lines until they sound natural. Most will say no, but if you keep going you’ll eventually get a yes.  

The strike rate seems to vary from day to day. Sometimes as many as one in three agreed to help, and on other days it seemed that nobody would speak to us. What you’re testing may also have an effect. For example, students may be more willing to help test wellbeing content as this is often an important subject to them. 

Make the user comfortable 

Some quick testing can be done on the fly, standing up or sitting on a wall with a mobile phone or tablet. For more in-depth user testing you’ll need to have comfortable seating (cafés are ideal – grab a table and chairs before approaching anyone). Then make sure the space isn’t too noisy and (easy to forget) that the screen isn’t affected by glare from the sun. 

Start the session by summarising how long it’ll take and what you’re testing. Ask them to speak their thoughts out loud, tell them they can stop at any time, and reassure them that any information will only be used for improving the website. And will be anonymous. At the end of the session thank them profusely for their time. You may even want to offer them some fruit or sweets (although we haven’t done this – yet).  

Don’t do too much, and remember to write it all up 

At the end of each session, talk to your fellow user tester and summarise what was said. Write it down straight away. A summary will be more useful than in-depth notes.  Don’t do more than four or five tests in one go. You need to be fresh and this is tiring work. Any more and the sessions will start to melt into each other. 

Act on the findings – then go out and test them again! 

Don’t just sit on the findings and not do anything about them. Make immediate changes to your web content and then test again. Continue making iterative changes like this and use analytics and other tools to measure how (or if) your pages are improving. 

Respond to any scepticism about doing user tests 

Some people may think that their website is fine and doesn’t require any user testing at all. Proactively explaining the user testing process and inviting senior management to witness testing themselves can be a good way to highlight problems.  

We invited two product developers to sit in on a user testing session. Both found it very revealing. It made it clear where users new to the system came across navigational issues or attempted new routes through the system. Similarly, it helped them focus on better use of text, avoiding terminology that though common to those close to the project was confusing or ambiguous to everyone else. They wouldn’t have picked these issues up without the user testing. 

Other types of user testing

There are many other types of user testing that can be done quickly and cheaply

• Card sorting. Online, or in person, card sorting can help you organise your content and make sure your terminology is clear.  
• Surveys, including pop-up surveys, are also quick and useful ways to test on the cheap.  
• We’ve even on occasion asked other members of the team or division to test potential content. While they’re not our true users, they can still offer a different opinion. 

Our experience so far is that cheap and fast user testing results in the identification of usability issues that would otherwise be missed. The small budget associated with it also means that you can always fit testing into your process.