4 Things To Consider When You User-Testing Your Service

Websites, mobile applications, professional software, operation systems, and basically each and every software that has been developed for the world to use as a component of the interaction with the user. This can be as basic as a command line or very complex like a screen with hundreds of buttons and options.

The user-service in general and the user-software, in particular, is known to be an extremely important component in every product. Indeed, a lot of research both in academia and the industry has been done to find the secret sauce which makes a user experience a gourmet dish rather than a home-cooked one.

User-testing helps you validate your assumptions and measure the true impact of new features and changes. This isn’t just something you should do because it’s good practice, this is how you know if it’s good at all.

If you don’t run tests for each change, all you can rely on is topline metrics. You can release a new experience and hope the dashboards will show an overall increase in the expected metrics. But how will track long-term changes? Performance regressions? What if there just happens to be a holiday that drove the actual increase?

Instead, run an A/B test to measure all the effects. And most importantly by learning from the test, you will gain meaningful insights about your users. This knowledge will help you make better decisions in the future.

When it comes to user-testing, big companies like Facebook enjoy the fact they have a lot of users so they can user-test them a lot. On the other hand, good startup leaders know how to extract a lot of information out of a relatively small amount of users as they have more personal access to them. We shall present five things to consider when user-testing from the knowledge gathered from these two very far but surprisingly completing perspectives.

Just a second before we begin with all the fun, it is worth mentioning the two main types of user experiments. First, a hypothesis oriented experiment where there is a parameter \ feature \ metric which is in question and the tester trying to measure the effects. A good example is an A/B test. Second, an exploratory experiment where there isn’t a specific question to be answered rather an observation and an attempt to extract useful insights.

Baby steps

Experiment with one clear thing at a time. Start by dividing any infrastructure and product changes. Make sure the things you change are well documented and are the smallest building blocks that still have a logical or operational effect on the product or scenario in question.

For example, migration to a new UI framework, it can be tempting to build the new version with all the fancy features, but it makes understanding the change difficult. Instead, it is preferable to rebuild the current UI with the new framework. On top of that you can test additional changes to the UI, the purpose here is to isolate the effects of the new UI framework from the actual user experience changes.

Pick the right users

Find a well-represented group of your users to test. It may sound trivial, but asking your friends and family may be easy but not necessarily useful to extract meaningful results from the experiment. Understanding who your users are and picking a group that well represent the diversity of this group should provide more accurate results.

In an auto-marketing tool developed for an online marketing company providing services for Facebook and Instagram professional publishers, we have done a user-testing with eight professionals (as this is a professional-oriented software) suggested by the client. This was a mistake as the real users were professional-ish and therefore the learning curve of the software was 3 times longer than the one recorded in the test as a result of picking the wrong users...

Prepare for the worst

If you ask our good friend Murphy he’ll tell you "Anything that can go wrong will go wrong". For example, we at the “Facebook Lite” team were doing user research in the field, and some users didn’t give us the correct username (it might have been their family members’ details). This meant that they aren't receiving the intended experience on their own devices for the experiment. To overcome this, we had a backup device set up with the correct experience.

When running an A/B test make sure you don’t open the test on too many users. If the test proves negative, you’ll be hurting the experience for too many people. Instead, first, open the experiment to a smaller percentage and only then increase the exposure. Of course, it’s advisable to have a quick and simple way to close the experiment if worst comes to worst.

Don’t holdback - holdout

Keep a long term holdout to see the effect over time, and detect regressions. A common mistake of a lot of product managers and developers is to study either a small number of users or run the study for a short period of time. It is commonly seen that users change their behavior over time, as a result of the learning curve of the product, changes in people’s behavior over the day and so many other factors that cannot be included in the user-testing but effects it.

For example, while developing a clinical product that has a very central screen presenting real-time clinical info for the medical in two ways; the extended table one and a summary as figures. A user-testing has been conducted to understand the optimal layout of the data on the page. In the beginning, the data from a 5 minutes review (using Hotjar) of 10 medical teams using the system showed that the table should be at the top of the page and the summarized data in the bottom (required scrolling) as 71% of the time had been spent on it. Then a longer test has been done on the same 10 teams but this time 4-minute review has been done each 2 hours. In the first 1-minute review the table won with 76%. But, in total, the table has been viewed 34% of the time while the summarized data the rest 66%.

It is fair to claim that user-testing is a really important tool for product managers, developers, designers and so much more professionals in the field. The ideas above should be a guideline for one starting to do user testing but should not be considered as hard rules that by following them the result of your test will be easily accomplished and significant because currently, this subject is more an art than a science.

If you struggle with your user-testing and would like some advice from people that walk these roads already contact us and we will be happy to help (if we can)!