In recent articles we have discussed the role of Generative and Evaluative research in experience design. Whilst the focus of those articles has largely related to qualitative research, in this article I want to explore the use of quantitative data in both Generative and Evaluative research. An overview of what I’ll cover is:
- What is quantitative data?
- Quantitative data can remove subjectivity from decision-making
- Statistical significance carries the day
- Matching behavior and attitudes to customer data
What is quantitative data?
When we talk about quantitative data, or research methods, we are typically referring to data that is numerical in nature that allows us to quantify attitudes, behaviors and opinions, and more. The number of respondents is large enough to allow statistical reliability and to relate the results of the research back to the general population, or the population that is of interest.
Taking the subjectivity out of decision-making
Quantitative data can deliver objectivity to decision-making processes that are more often subjective in nature. One of our clients, a major Australian university, was undergoing a digital transformation, which also tied in with a brand refresh. A key aspect of the transformation was a website redesign. They had been provided with three design treatments to choose from, and normally, the decision on which treatment to proceed with would sit with some key stakeholders based on internal discussion. However, using an online survey with over 600 respondents, together we were able to identify which treatment best represented each of their key brand attributes. This process yielded a very clear “winner”, whilst also teasing out which elements of the alternative treatments should be incorporated within the final design to deliver the best representation of their brand.
The use of quantitative data in this context removed individual preference or subjectivity from the discussion and provided an unambiguous path forward, something that was highly valued by the stakeholders. Through establishing key criteria that can be used to assess the suitability of a design treatment, the team was able to make a decision with ease, and it also provides a frame of reference for any future changes to the design of their website. Any future changes can be assessed against these brand attributes before implementing any change.
Statistical significance tells the story
Applying statistical significance testing to different approaches to interaction design can provide an unequivocal understanding of which approach to implement. We collaborated on such a project with a client operating in the financial services sector who needed assistance in choosing between two different interaction schemes for completing an online banking transaction. We designed an online usability test that allowed participants to experience both interaction design versions, with the order of presentation varied across participants to reduce potential for bias. With over two hundred participants it was possible to test for significant differences in task completion and attributes such as preference and perceived ease of use. A statistically significant difference was identified allowing the team to move forward with confidence and bypass any potential contention amongst internal stakeholders regarding the selected interaction design
Matching behaviour and attitudes to customer data
Organisations regularly use customer segmentation to assist them in understanding and communicating with their customers. These segments are often created via the identification of “clusters” of similar people based on a combination of shared demographics and characteristics. The issue we often find with these segments is that they lack the strategic differentiation required for making experience design decisions. For instance, we are specifically interested in the variants in digital behaviour and preferences when designing digital solutions and clients are interested in closing this gap.
We worked with a client to achieve this outcome and did so by running a large-scale survey with their customers that explored behaviours and attitudes related to use of digital. Through statistical analysis of the responses we were able to create a model that allowed our client to relate specific behaviours and attitudes to product holdings. We created “strategic segments” to aid them in experience design decisions, with the added benefit of also being able to tie these “strategic segments” back to their existing customer data.
The above case is an example of using quantitative data to support generative research. It was not particularly clear what the outcomes or predictive model would actually look like, it required an open exploration of the data and relationships between variables to reach the final model. For some, the application of quantitative approaches to generative research is not intuitive but it is an important, but often overlooked, tool in the experience design process, achieving outcomes not possible with qualitative techniques. The preceding two examples represent evaluative research where weight of numbers and statistical significance provided an indisputable path forward, removing ambiguity and subjectivity from decision-making. Whether it is qualitative or quantitative research, making informed evidence based decisions, rather than relying on instinct or the highest paid opinion provides a strong foundation for success of your experience design.