This causal chain positions design with one foot in the world of customer experience (CX), and the other in the world of “making things work”. This sometimes creates confusion between product and design responsibilities, since both designers and product managers need to collaborate with the whole spectrum of disciplines involved in creating products that work and that people use. However, the designer contributes to these outcomes by pursuing design quality (AKA user value), while the product manager needs to orchestrate and prioritise effort across all disciplines in pursuit of overall product outcomes.

The consequence of this view is that every contributing discipline — including design — should be held accountable for discipline-specific quality metrics, and have shared responsibility for product success metrics. While product managers should be held ultimately accountable for product success, and take a holistic view across the spectrum of activities — from sales and marketing to customer support through design to science and technology — that contribute to that success.

As a product company, we include societal and environmental impacts in our definitions of product success. This is referenced in the causal model through the concept of “systemic impact” beside user benefit and business value. For example, the products we are building at Evinova aim to deliver systemic impacts including reducing the carbon footprint of drug development, making clinical research more inclusive and equitable, and accelerating the development of new medicines. These goals are baked into our organisational purpose, the products we have chosen to build, and the way we define success of those products.

A different approach might be needed for design teams working in challenge arenas 3 and 4, where the type of intervention required is undefined and may end up not being a product or service at all, but perhaps a policy, a piece of legislation, or an organisation. This kind of work is outside the scope of this article.

I don’t fully agree with the maxim “you can’t improve what you don’t measure”. Artisans, cooks, artists, writers, musicians and tradespeople spend their lives honing and improving the quality of their work, without requiring objective validation. However, it is often true that “businesses won’t spend money on improving things they can’t measure”, which for many designers means they can only improve design quality by working nights and weekends! So, for everyone’s sanity and quality of life, we need ways of measuring our design quality dimensions. We currently have qualitative tools that give us insights across all dimensions, but can only quantify a subset. This is very much a work in progress, so I welcome feedback and pointers to measurement tools and approaches that could help.

Usability

Our target users can use it — quickly, without errors, with minimum training and support, in real-world conditions.

Measurement methods: heuristic assessment, usability testing, SUS/UMUX usability questionnaires (deployed both in user testing and real-world use)

Target: 80% agree/strong agree (top 2 box method) with Q2 of the UMUX-lite questionnaire ([This system] is easy to use).

Accessibility

All our target users can use it — regardless of their physical, sensory or cognitive abilities

Measurement methods: accessibility checklists, accessibility audits, user testing with diverse panels

Target: zero critical defects against WCAG 2.1 AA using Deque scoring method. Continuous improvement through recurring audit and remediation planning.

Usefulness

It helps people progress towards their goals

Measurement methods: user interviews, SUS/UMUX usability questionnaires (deployed in user testing and real-world use)

Target: 80% agree/strong agree (top 2 box method) with Q1 of the UMUX-lite questionnaire, alternate version ([This system] does what I need it to do)

Enjoyment

People feel good using it

Measurement methods: user interviews, contextual enquiry

Target: TBD

Additional notes: Measuring enjoyment in product use is an active area of research and a number of scales have been proposed, but we haven’t found any short, widely used validated questionnaire. Potential routes to explore include designing a simple 1-question measure to accompany the UMUX-lite questions, or using automated sentiment analysis on interview transcripts.

Context-fit

It works well within the existing technology, workflow, and normative landscape

Measurement methods: contextual enquiry, shadowing, user interviews

Target: TBD

Additional notes: Contextual enquiry often reveals context-fit issues which are otherwise hard to detect. User workarounds and patches, misalignments between intended and actual use, and blockers that prevent or limit use, can often be observed in the field. One possible approach to quantify this dimension would be to grade these from minor to critical, in a similar approach to that used for technical bugs or accessibility defects.

Information exchange and interoperability are specific types of context-fit which can be technically specified when domain-level standards (such as FHIR) or dominant platform players (such as the EPIC EMR) exist. However, technical interoperability does not guarantee context-fit, nor is it always a necessary condition for it.

Net-Simplicity

It removes more complexity than it adds to peoples lives and jobs

Measurement methods: contextual enquiry, shadowing, user interviews

Target: TBD

Additional notes: One possible approach to measuring this dimension could be to define a set of “net simplicity heuristics”. These could include aspects such as transferability of existing user knowledge; hand-offs between systems in cross-product workflows; avoidance of double data entry; avoidance of multiple sources of truth for the same information; and reduction in total number of systems used.

These metrics can tell us something about design quality, if contextualised with other sources of insight.

Customer Satisfaction (CSAT) — this is closely linked to design quality, especially if it is used for distinct aspects of the customer experience, e.g. product, customer support, and training. However, technical quality can have a huge impact on this, so if your product has performance or reliability issues this may drown out any signal about design.

Monthly or Daily Active Users (MAU, DAU) — generally speaking, rapid growth in user base is a sign of good product-market fit, of which design is an important component — but other aspects such as pricing and distribution may have greater positive or negative impact.

User Retention — user retention or churn metrics, with appropriately tailored definitions of what constitutes an “active” or “inactive” user, are closely related to design quality. If users find value in your product, they will keep using it. However, this has similar caveats to CSAT around technical quality issues.

Task completion/abandonment — one of my favourite product metrics — do users achieve the tasks for which we designed the product? Also one of the hardest measurements to define, as it requires both a clear definition of what the product is for, and a method of measuring if that has occurred.

Duration of sessions (measured in time or number of tasks) — this metric is a double-edged sword — sometimes the best design enables users to quickly find what they need and move on, so a “bounce” may actually be a happy user. However, for some products where the benefits are obtained through sustained use, longer sessions are better.

Customer support logs — an absolute goldmine for identifying and quantifying pain-points that the product is creating, some of which will be design related.

The ideal approach is to view these metrics in combination with direct measurements of design quality, and with contextual information about what else may be contributing (e.g. technical issues, customer relationship issues). However, I argue that design should never take sole accountability for these product-level metrics and they should never be used on their own as measures of design quality.

We’re using these measurement strategies in two ways within Evinova. Firstly to contribute to the overall scorecard and OKRs for the business. Secondly, within the product teams, by rolling out some of the well-defined measures (like UMUX-lite, CSAT, and accessibility audits) across all our products to create a consistent baseline of measurement. We also intend to explore and experiment with metrics for some of the harder-to-measure concepts like enjoyment, context-fit and net-simplicity. I’d love to hear from the community if you know of measurement approaches that we might be able to apply.

For design to be understood in a business context, its purpose needs to be defined in a way that is distinct from, and compatible with, the purpose of other professions. At Evinova we have defined our purpose with the statement:

Evinova uses human-centred design to create products and services that are valuable for the people who use them.

Design also needs quality criteria. Based on our purpose statement, we frame design quality in terms of user value:

Design quality = User value (of our products and services)

This means we take accountability for designing solutions that:

1. are barrier-free (usability and accessibility);

2. people like to use (usefulness and enjoyment);

3. improve messy reality (context-fit and net-simplicity).

We have qualitative methods to assess all of these dimensions, and are working on establishing more systematic quantitative measurement as part of our overall OKR framework.

Finally, we recognise that we are contributors within a larger, multi-disciplinary system, and share responsibility with our colleagues in other disciplines for making products that work, that people use, and that deliver outcomes. Ultimately our collective goals are user benefit, business value, and positive systemic impact.

I hope this essay provokes in-house designers to move on from “ROI of design” rhetoric and ask themselves some hard questions:

• What is design for [in my organisation]?
• What is good design?
• How can we measure it?
• What does that mean for the way we organise and incentivise design in our business?

I’m sure your answers will be different from mine; and I’m equally sure that examining these questions will help you better understand and position the value of design in your business.