When 60 Seconds Don’t Add Up to One Minute – Benefit Calculations for User Centred Design

Usability engineers and their clients (internal and external alike) have a legitimate interest in the benefits of user centred design approaches. One typical question that is encountered in this context is: “What benefits does the new interface that has been developed via user centred design bring compared to the old interface?”

Potential Benefits of User Centred Design

Looking at an in-house application, areas to investigate for benefits are, e.g.

• support costs
• time to learn
• costs of development / late changes during development / maintenance
• user efficiency
• errors
• productivity
  

(An overview of approaches to cost-benefit analyses for usability can be found in
Rajanen M., Jokela T. (2004). Analysis of Usability cost-benefit models. In T. Leino, T. Saarinen & S. Klein (Eds.), Proceedings of the Twelfth European Conference on Information Systems. Turku School of Economics and Business Administration, Turku, Finland. [PDF])

If user centred design works in the way intended, the parameters mentioned above should decrease – except for the last one, productivity, which should increase.

Measuring Benefits: Time to Complete Tasks

The interesting part of the analyses is to “measure” the benefits in order to report them. Often the goal is to end up with quantifications that show how the diverse areas are affected by user centred design. For user productivity, the time to complete key tasks is a measure that is often used. The idea is to compare the time needed to complete a certain task (e.g. entering information into an online form) with the old and the new interface. To arrive at those times, different approaches can be used. One is actually measuring the times, which requires both interfaces to be implemented. In another approach, the tasks are modelled, e.g. with keystroke level modeling [PDF].

The benefit is then calculated by subtracting the time needed with the new interface from the time needed with the old interface and to hope that one ends up with a positive value. This benefit for one user is then multiplied by the number of transactions per day that is carried out and the total number of users to arrive at the overall daily benefit for the organization in which the interface is used. Multiplied by working days per year and you arrive at the time per year that becomes available as additional time for productive work…or do you?

The idea seems straightforward. Example:
An interface that is used by 2000 users of an organization is improved so that 1 second is saved for a task that is performed approximately 30 times per user and day.
Overall (for the whole user group), this results in
1 second x 30 X 2000 = 60000 seconds saved per day
With 230 working days per year, one ends up with
60000 seconds x 230 = 13800000 seconds = ca. 479 working days for the whole user group (8 hours work per day)
saved per year.
(Multiply it with the daily income of users and you get the benefits in terms of money.)

Now what is the problem with this neat quantification, which shows that even minor changes to an interface can add up to a lot of time saved overall given a large enough user base?

When the Whole is Less Than the Sum of Its Parts

The issue is that you cannot simply add up individual seconds saved. In this particular case, the whole is less than the sum of its parts. It makes a difference whether 30 seconds become available “en block” or distributed over the day. And it makes a big difference whether one has an hour of additional time in one day or distributed over several weeks or months. In the case where additional time becomes available as a whole, users might really start working on additional things whereas with only seconds or minutes per day, they might hardly even notice that anything has changed. And this does not even regard the fact that the time benefit is not only distributed over days and months, but also over the whole user group.

Indirect Benefits

This is not to say that one should neglect those “small benefits” altogether. Users may realize that the system is more responsive or that their work can be carried out “smoother” even though they may not be able to put their finger on the exact cause for this impression. This may result in an improved overall perception of the interface which can increase user motivation and through this contribute to user productivity indirectly. This effect cannot be described with simple calculations as above however.

I certainly believe in the benefits of usability and user centred design. But I also believe that they are not always easy and straightforward to quantify. A lot of improvements from usability are effective in indirect and / or hard to measure ways such as perception and motivation as described above. This does not mean that these effects are less important (in fact, they may even be amongst the most important effects). But constructs such as “perception” and “motivation” don’t look as good dollar signs and numbers in Excel sheets…

I guess assessing the benefits of usability and user centred design really is a combination of quantifications (“hard numbers”) and qualitative observations , which may be based on observations from, e.g., industrial psychology (“indirect benefits”). One should not assume that certain effects / benefits are more relevant or “true” because they can – seemingly – be easily quantified. As shown above, numbers can be misleading.