Measuring design quality using heuristics

How do we measure design quality? I heard this question in all the companies where I worked. It wasn’t just the designers who asked it. Answers range from “we know when we see” to decisions based on analysis based on design principles or detailed and thorough product and design reviews. Other suggested solutions include surveys such as NPS (Net Promoter Score), CSAT (Customer Satisfaction Score), or individual user surveys such as “How much do you like our product?”

Each of these solutions has a number of problems. Internal evaluations can be skewed by intrinsic motives or blind spots. Surveys may fail to highlight design quality or may add unnecessary burden to the user.

Several years ago, I led a design and research team at a company that was just getting caught up in John Doerr’s philosophy and OKRs. We measured everything, including the quality of the design.

The scoring engine we ended up with was an extended set of usability and product heuristics based on Jacob Nielsen’s 10 UI usability heuristics. We concluded that while the design team may be biased in evaluating their own work, it is clear that designers need to know what “good design” is, as criticism is an important part of our practice.

Several colleagues and I developed a set of heuristics that fit our overall product design structure: design thinking, UX and UI design. An additional benefit was that we looked at the qualifications and skills of designers in similar teams, so when assessing quality, we could easily answer the question “where in this project did the designer demonstrate strengths or capabilities?”

To evaluate the quality of the design, we got together in a small group after the launch of a new product and gave a quick overview of the work done. We simply answered “yes” or “no” to each question, and applied the formula to determine the percentage of quality score (100 – yes, 0 – no).

Although this did not give an accurate result, we recorded resonant options from project to project and were able to give feedback to designers, as well as provide information about quality to non-designers.

This is a heuristic.

Design Thinking

User problem

• Is the user’s problem clear?
• Does this study support the proposed solution to the problem?
• Does the available data support the proposed direction for solving the problem?
• Does the product or feature solve the user’s problem stated in the product description?

The audience

• Does the product or feature solve the target audience’s problem?
• Does the product or feature avoid negative trade-offs for the target audience?
• Does the product or feature have a minimal negative impact on the global audience / other audiences?

Value

• Does the product or feature add value to the user experience?
• Does the product or function avoid value degradation in other areas?

Design principles

• Is the product or function overloading the user?
• Has the product or function been pressure tested?
• Does the product or function improve quality?

UX design

State of the system

• Is it clear where the users are located?
• Is it clear to users what happened earlier and what will happen next?

A clear task

• Can users easily find what is expected of them?
• Do users know how they are expected to complete the task (s)?

Simplicity use

• Is the minimum effort required to study the task at hand?
• Are the current tasks in a logical order?
• Does the current task (s) avoid unnecessary complexity?

User expectations

• Do the tasks set correspond to the user’s mental models?
• Does the product or function follow common patterns of interaction?
• Is the product or feature flexible or customizable to meet the needs of different user groups?
• Is the product or function associated with other related product or function systems?

Error processing

• Do users have a clear path when things go wrong?
• Is the error message clear, descriptive, and concise?
• Have edge cases been considered?
• Do users have access to help?

UI design

United standards

• Does the feature or product design meet accessibility standards?
• Does the design of a function or product use this design system?
• Does UX copywriting conform to content strategy standards (appropriate voice, tone, style)?

Recognize, not remember

• Does the design of a function or product make objects, actions, and parameters visible?
• Is it easy to get instructions or information?

Familiar elements

• Is the page layout, hierarchy, etc. of the function or product similar to other commonly used functions or products?
• Does the design of the feature or product comply with the interface standards for the respective platforms?

Craft

• Does the design of a feature or product meet industry standards?
• Is the design of a feature or product best in class?
• Is the design of a feature or product clear and minimal when possible?
• Is the design of a feature or product delightful when needed?

Note that they have been adapted to our understanding of design quality and may vary depending on the context. For example, when shaping design principles, we focused on how people learn to use a product.

While this approach worked for us, we admit that it cannot be scaled outside of our small team. For example, a team of hundreds of designers submitting designs daily is unlikely to be able to manually check them. However, as designers learn to moderate their work, we believe that this approach will enhance design quality – which will help measure product success and performance – for both the company and our users.

Thanks to jennie § yip and Casey Callow for help!