How much user data do you really need for the UX design process?

In a perfect world, you would first do extensive user research, systematically derive requirements, present initial ideas and designs, and test them until the results say, “Now it’s perfect!”. Unfortunately, we experience this perfect case relatively rarely in the real world.

In reality, there are deadlines and limited resources.

This creates the need to keep the design process as large as necessary but as small as possible. But if you’re like most others, you probably have difficulty deciding how much is “enough.” To help you with this, we have developed the flexibility and risk matrix.

In this article we do the following:

• Present you with this matrix as a decision support tool to the question “What is the right amount of user data for my product?”.
• Define the two dimensions “flexibility” and “risk” according to our understanding in this context.
• Clarify for you how to best work with the flexibility and risk matrix.

Here we provide you with illustrative examples that show you what conclusions can be drawn from a specific location on the matrix.

Enjoy reading!

Introducing: The flexibility and risk matrix

The idea behind the matrix is to evaluate your system, product, or service concerning the two dimensions of flexibility and risk and draw conclusions about the effort you need to invest in Data-Driven UX Design. This will allow you to estimate the data required for the rest of the process. First, we would like to introduce you to the flexibility dimension.

Als kleine Hilfestellung haben wir dafür die Flexibilitäts- und Risikomatrix entwickelt.

The Flexibility of Your Solution

By flexibility, we don’t mean how far you can stretch or bend your product but how adaptable it is after it’s released.

If you manufacture a washing machine, for example, large parts of the product are fixed after development and delivery: The dimensions are defined, the control knobs are fixed, and even the electronics of your device can no longer be changed without great effort. Of course, you could send out a service technician and have all the washing machines modified by hand – but the effort involved would most likely be disproportionate to the benefit.

This product would therefore have very little flexibility in our perception. In addition, the life cycle of a washing machine is usually relatively long – namely ten years – which ensures that customers who have just bought a new one will not buy another one for a while. So again, there is very little flexibility here. Your washing machine can only be less flexible if you also use the washing machine in a regulated market (let’s say, for example, the medical market). Then, in addition to the technical and organizational hurdles to change, there are regulatory hurdles that may even prohibit you from changing the washing machine – at least without retesting it by a regulatory administration office. That means your product would not be flexible enough to be changed without incurring significant costs and effort for quite some time.

But if your washing machines were “smart,” for example, and you, as the manufacturer, could remotely change and adjust various cleaning programs via software updates, your device would gain significantly in flexibility. You would still not be able to change the hardware, but you would be able to change the functionality, at least within certain limits.

Hardware flexibility is also possible in principle. For example, a manufacturer of hot tubs offers the option of exchanging the backrests of the individual seats with a click system. The new backrest with different massage jets creates a unique experience, which the customer can thus flexibly control himself.

There are complete systems that are based on hardware flexibility and have standardized interfaces. Examples are system cameras with exchangeable lenses or computers with exchangeable RAM and slots for add-on cards.

On the other side of the flexibility scale are products that can be easily changed even after market launch. For example, unregulated software products can usually be adapted very quickly and flexibly. Manufacturers can make small or far-reaching changes via patches or updates, integrate or remove functions, and even completely replace the user interface.

Determining the flexibility of your own solution – an aid

Therefore, the higher the flexibility, the more uncertainty you can accept in development. If it’s easy to fix bugs with one click and add features even though the product is already in the field and used, securing the design with data is less relevant. Unless, of course, you are operating in a risky environment.

If you want to determine whether flexibility is more likely to be high or low in your case, you can use the following questions to rank it:

• How long do your customers and users expect to work with an unchanged version of your solution?
• Can and may your solution be used without prior training of the users?
• Is it possible to update your solution without regulatory or organizational constraints that require (re)testing?
• Does your solution also have internet access during use, which you, as the manufacturer, can access from the outside?
• (If no internet access is available): Does your product have the possibility to have new product versions installed on-site by the users?

The more often you have just been able to answer “yes,” and the shorter the period customers and users expect to work with an unchanged product version, the more flexible your product is. However, if you have had to say “no” frequently, you should spend more time defining user requirements early on and testing them, as your solution will need to be classified as less flexible.

The Risk of Your Solution

The risk dimension refers to how dangerous it is if errors occur while using your product, system, or service. Risk in this context is defined according to ISO 14971 as:

Risk = Probability of occurrence × Severity

Therefore, the question is how often this error is likely to occur and how bad its possible consequences result. The risk can relate to different areas:

• The health or life of persons is endangered by operating errors caused by poor design.
• The economic basis of a company is jeopardized by poor design. This could happen, for example, if production losses occur due to faulty operation or if a lawsuit is sought because of these production losses.
• The reputation of a company is jeopardized by poor design. How bad the loss of reputation is for the company must be examined individually based on the brand value and promise.

The easiest way to understand the risk is probably if you are developing a system, product, or solution whose use is directly related to human health or life.

Perhaps you now have the feeling that this can’t happen to you. However, as UXers, we have worked directly on such projects in many places! Especially in the medical sector, high-risk products are commonplace: developing a heart-lung machine, a defibrillator, or a patient monitor. But even outside the medical industry, we frequently encounter high-risk projects, such as an aircraft emergency landing system or an airport baggage scanner. In all these cases, incorrect operation and misjudgment can result in glaring damage. Therefore, solutions that endanger the health or human life are automatically (high-)risk for us.

The risk of economic loss

Also, risky, but of course in a different way, are applications that, if misused and misinterpreted by users, cause damage to a company’s economic foundation. This can be your own, but more importantly, it can be your customers’ businesses. This category of products includes, for example, process controls in production or software systems that map a company’s primary work processes in the productive area. For manufacturing companies, this could be production planning or machine control; for service companies, it could be the project management tool or central data storage.

So, whenever the purpose of the business cannot be fulfilled or can only be partially fulfilled if your system, product, or service is operated incorrectly, we would also rate the application as risky. Compared to the risk of direct harm to people, however, more of a medium risk.

The company that runs into economic difficulties due to a poorly designed system, product, or service can also be your own. For example, if you only have the capital for one product development, this last throw should fit and not flop. You could not “afford” a poor design in this case.

The last point – admittedly a relatively soft one, but no less important – is when poor design threatens your company’s reputation to such an extent that short-term or lasting damage is to be expected, affecting the perception of the brand and its quality. How severe the damage is, of course, depends heavily on several factors.

In our view, these include:

• How established is the brand already?
• What does the brand stand for?
• How closely is the brand linked to the product?

In our view, the potential damage from a poorly designed solution is significantly higher for established brands with outstanding quality than for unknown players.

For example, buying a premium sedan and then struggling with the super-expensive but stubborn-to-operate user interface of the infotainment system hurts the brand. It’s not uncommon in this case to tell friends or acquaintances that the vehicle itself is excellent, but the operation is horrible. Suddenly, it’s not so “premium” anymore.

If you don’t have a well-known brand yet, you can’t lose brand equity through poor design. But of course, you won’t build one either. However, if you work for Apple, for example, and thus even stand for usability, user experience, and innovation, the whole thing looks different: The damage caused by a problematic product would be immense.

The same applies if the poorly designed product is closely linked to the customer’s brand. Then bad design radiates strongly and clearly onto the core brand – just like good design. An example of products closely related to the core brand is Apple’s products, such as the Apple Watch. Here, it is clear from the name alone that there is a close connection between the product and the core brand. So if the Apple Watch is poorly designed, it reflects directly on Apple.

However, the potential damage is lower if your company operates more in the background and presents many different brands to customers. An example of this could be Proctor & Gamble, whose in-house brands include the diaper manufacturer Pampers and the cleaning product Swiffer or the chest rubs from Vick. If a batch of Pampers diapers does not work as intended, it is unlikely that customers will stop buying and using Swiffer.

Of all the factors mentioned, damage to the brand is the hardest to measure. Nevertheless, we would also rate an application as risky here if it is published by a strong brand closely linked to the product and the brand itself stands for high product quality in the premium price segment.

How risky is your product?

If you want to assess how risky your product, system, or service is, you can use the following questions to make an assessment:

• Can people’s lives be endangered by operating your product incorrectly?
• How many lives will be put at risk if there is a mishandling?
• Can incorrect operation of your product threaten the health of people?
• How many people’s health is at risk if something goes wrong?
• Can incorrect operation endanger the economic basis of another company or your company?
• Can poor design (not only in terms of usability but also user experience) cause lasting damage to the value of your brand?

The more often your answer to these questions is “yes,” and the higher the number of people potentially affected, the higher the risk associated with your product, system, or solution. If you could answer “no” to all questions, it is probably a solution with relatively low risk.

Working with the Flexibility and Risk Matrix

With the definition of flexibility and risk from the previous sections, working with the flexibility and risk matrix is now relatively easy to explain. To work with the matrix, you should ask yourself the following two questions:

• How flexible is my solution once I have finished developing it and brought it to market?
• What would be the risk if my solution leads to misoperation and misinterpretation?

Using the matrix is not about being able to justify your position in the matrix down to a tenth of the inch. So, it is not about whether your solution scores a 58 or a 59 on the flexibility scale. It’s about getting a sense of the flexibility and risk of your solution.

In general, the further you move your solution to the top right of the matrix, the more data you should obtain during the development of your solution to ensure the quality of your designs (see Figure 1.13). The further you are at the zero point of the matrix at the bottom left, the less effort you need to invest in testing your solution in advance. This has several consequences:

• If you have a low-flex but high-risk application (top right), you must ensure everything works in advance. This is where you have to invest the most effort, and in the best case, you even perform a final test before release (of course, for a medical device, you have to do this anyway, regardless of our matrix). This gives you multiple safeguards and also allows you to show that you have invested time and effort to ensure that your product not only looks good in the end but also works safely. An example of a product in this category would be a heart-lung machine: a regulated, approved hardware product that can’t be changed once shipped. What’s more, people die if it’s operated incorrectly.

• If you have a very flexible but potentially risky application (bottom right), you should also ensure that everything works so that no unnecessary damage occurs. This is where liability issues and due diligence can suddenly come into play. Again, a final test before release is a good idea. An example product in this area would be a Tesla, whose hardware is fixed but can be widely updated via “over the air” software updates. Nevertheless, people can still be harmed in the event of operating errors.

• If you have an inflexible but low-risk solution (top left), the worst that happens to you is that you have to live permanently with a worse solution. Ultimately, only you (and your customers) can decide whether you want and accept that. In most cases, a final test is unnecessary, and the number of iterations can be reduced compared to risky applications. An example product in this category would be the washing machine from a non-premium manufacturer mentioned earlier. It is not flexible due to fixed hard- and software. However, no great (image) damage can be expected without a premium brand as long as the product works.

• If you have a flexible and low-risk application (bottom left), you are very comfortable. You can get to market quickly and then address potential issues downstream. From our perspective, you don’t need a final test, can test less or not at all, and in the best case, work with usage data from live operations. An example product would be a to-do app without a well-known brand. It can be changed flexibly anytime and does no critical damage if errors occur.

Conclusion

You now have an excellent idea of where you can locate your product on the matrix and what that means for your particular case regarding user data. Now, we suggest you read up on what precisely valid data is, how to evaluate it critically, and what kind of data you need (Objective vs. Subjective or Qualitative vs. Quantitative data). And then, of course, there’s the question, “What are the best methods for collecting data, and how do they work?”

The book “Usability and User Experience Design – The Comprehensive Guide to Data-Driven UX Design” (published by SAP Press) written by Michaela and Dr. Benjamin Franz deals with precisely these topics (and many more).

This article is a part of this handbook. It belongs to the fifth chapter: “Data-Driven UX Design.” In this chapter, you will get the process explained in depth and answers to the above questions regarding user data.

Need help with your UX design? Feel free to get in touch with us via our contact form. We look forward to hearing from you!