Evaluating Qualitative Interviews – with Sense and Practical Use

One of our main tasks in the field of user research and psychological product design is to uncover a person’s motivations and attitudes. To achieve this goal, the qualitative interview is the method of choice. Due to the targeted design of the questions and the possibility to respond freely and flexibly to the participants’ answers, there are almost unlimited possibilities to get to know your interview partners better.

In addition to the challenge of conducting a good interview, the challenge of then evaluating this data in a targeted but also economical manner often arises in contact with the client. Unfortunately, conventional scientific methods for content analysis offer only a very limited solution for this, as the evaluation is often disproportionately time-consuming. This results in the problem that scientific evaluation is often expensive, tedious and not very practice-oriented.

Today, I would like to introduce you to a method that we use to evaluate in-depth interviews, with a focus on comprehensibility, practical applicability and cost-effectiveness. Perhaps this method will also support you in the future when evaluating your data.

The method I would like to introduce to you I call “pseudo-quantification”. The goal is to crystallize the most important answers from the qualitative interviews and to get a feeling for how many people show these motivations or attitudes.

Let me explain the method with an example. Imagine you have developed a new app to help parents make arrangements for their children’s leisure activities. To test the quality of the app, you conduct a series of usability tests or interviews. Within these interviews you talk to a total of 3 parents.

Here you also see directly the problem of the topic: 3 interviews are by no means quantitative data. Nevertheless, clients are often interested in finding out how many people have this perception. However, conducting a large number of qualitative interviews is extremely costly. On the one hand because of the time needed to conduct them, on the other hand because of the time-consuming analysis. In addition, it often turns out that many of the motivations can already be identified in an interview with just a few participants. Large companies usually want a combination of qualitative and quantitative methods that allow real quantification. However, this approach is complex and expensive and thus not very suitable for small companies and also means a considerable time delay.

Pseudo-quantification gives an impression of the importance of the reasons without the effort of duplicating the survey and is thus suitable for less elaborate surveys and smaller companies.

Now, back to the 3 interviews:

Imagine that you ask in the interviews what would be important for the users in this app.

The first parent answers you the following:

“Oh, you know, our son is on the road a lot and plans change spontaneously. So it would be very important for me that we can both look at the calendar together and also enter changes spontaneously. Since we both work it would be great to be able to see drive times to activities depending on our own calendar and assign the activity to a person so we know who’s taking care of it.”

Parent two responds:

“As you already know, we have three kids. That makes it really hard to keep track sometimes and we have a hard time figuring out if we have time or are actually already off somewhere with another kid. It would also be cool if we could get suggestions for activities. For example, if our middle daughter is at a friends house, the app could suggest what we could do nearby with both of our kids during that time. For example, a playground or a friend nearby.”

The last parent tells you:

“We are both self-employed. Unfortunately, something else happens all the time and often we have to change our plans spontaneously. So it would be important to us that we both have access together and at the same time and on short notice. On the other hand, we often rely on the help of family and friends.”

Now that you have the data, take the first text and go through it for important aspects. The first statement is “both can look at the calendar together.” For the qualitative statement, you would now take this quote and leave it as a video excerpt or audio recording if necessary. If necessary, you would still interpret it, but you would look at the responses on a case-by-case basis. For this method, here’s what you do: You consider, what is the next level of abstraction of this utterance? For this specific example, you might want to introduce the category of “cooperative use.” Then create a table – on your PC, on a sheet, wherever you want – and write the following in it:

Next, you could add a new category and call it, for example, “Schedule Management”.

Then add the individual parent statements in the last column so that later, without having the full quotes or reading the transcripts, you know which responses all fall into that category.

This way, you go through the entire interview. For example, after Interview 1, the table might then look like this.

So far, the table is purely a listing, but you can of course start pseudo-quantification at this point. To do this, expand the table as follows:

Here you can now suddenly see how often a category was mentioned at all and how often the category was mentioned at least, but also at most in the individual interviews. Of course, the numbers do not yet make much sense after the first interview, because it is clear that every reason that appears in the table was mentioned at least once. But as you can see now, two aspects of the schedule management topic (a calendar and an integration of a driving schedule) were already mentioned in the first interview.

Now you continue to complete the table with the other interviews. In parallel, you keep a second table that shows you the evaluation of the data on the interview level. Here you enter the reasons that were given for each interview conducted. This table will be constantly expanded as the interview progresses. Here is an example

So you end up with a table in which you keep a list of answers to which you assign numbers and individual answers that you have packed into that group. On the other hand, you have a table in which reasons are assigned numerically to each interview.

On the one hand, this gives you a good overview of what was said, but on the other hand, it also makes the evaluation much faster. The first interviews with this method are quite time-consuming, because you have to discover and document new reasons permanently. After a few interviews, however, many of the reasons repeat themselves, so that you only have to assign numbers and, if necessary, add missing individual details. This speeds up the evaluation process considerably, especially if there are several interviews. In addition, you retain the option of searching later for tendencies and patterns that are shared by several people. You quickly find out which aspects are crucial for many people and which are only crucial for single persons. You see how many different aspects there are and you still manage to create a summary that increases comprehensibility.

Overall, a super approach to evaluate qualitative interviews in a pragmatic yet scientific way. I hope it brings you as much exciting information as it did us.