Engaging with (big) data visualizations: Factors that affect engagement and resulting new definitions of effectiveness
First Monday

Engaging with (big) data visualizations: Factors that affect engagement and resulting new definitions of effectiveness by Helen Kennedy, Rosemary Lucy Hill, William Allen, and Andy Kirk



Abstract
As data become more and more ubiquitous, so too do data visualizations, which increasingly circulate online and are an important means through which non-experts get access to data. This paper addresses the factors that affect how people engage with data visualizations, a relatively under-researched focus in visualization research to date. Drawing on qualitative, empirical research with users, we identify six factors that affect engagement: subject matter; source/media location; beliefs and opinions; time; emotions; and confidence and skills. In drawing attention to these factors, we bring HCI concerns together with approaches to media audience research, to identify new themes for visualization research. In particular, we argue that our findings have implications for how effectiveness is conceived and defined in relation to data visualizations and how this varies depending on how, by whom, where and for what purpose visualizations are encountered. Our paper aims to extend the horizons of visualization research, in its focus on factors that affect engagement and how these suggest new definitions of effectiveness.

Contents

Introduction
Research into engagements with visualizations
Methodology
Factors which affect engagements with visualizations
Implications of findings for definitions of effectiveness
Conclusion

 


 

Introduction

As data become increasingly ubiquitous (Kitchin, 2014; Mayer-Schönberger and Cukier, 2013), so do data visualizations — that is, the visual representation of data and datasets which communicates precise information and values. Indeed, the main way that ‘ordinary,’ non-experts access newly ubiquitous data is through visualizations, as Gitelman and Jackson note when they claim that data are ‘mobilized graphically’ [1]. It is important, therefore, to consider data visualizations as objects for critical scrutiny, not just as mechanisms to communicate data [2]. We do this in this paper by focusing on the question of how people engage with data visualizations. By ‘engage’, we refer to the processes of looking, reading, interpreting and thinking that take place when people cast their eyes on data visualisations and try to make sense of them. We propose that research about visualization engagement can learn from some of the approaches that are widely used in media and communication studies, especially in relation to audience research and their attention to factors (such as class, gender, race, age, location, political outlook, and education of audience members) which affect engagement with media and communications artefacts. Importantly, these factors extend beyond textual and technical matters.

In data and information visualisation research, studies exploring the effectiveness of visualizations tend to define effectiveness quite narrowly, if at all, measuring it, for example, through accuracy, consistency or speed of comprehension. On the whole, such studies provide little information about who users are and how this might affect their engagement with visualizations. Also, they almost never consider the factors that affect how people engage that concern us here [3]. Some research attempts to bypass social or cultural influences on engagements with visualizations, opting for techniques like electroencephalography (EEG) which go straight to the brain, perceived as a biological entity unaffected by society, culture or context (Anderson, et al., 2011). In contrast, we suggest that the study of engagement with visualizations, to date primarily carried out within HCI or computing, can benefit from adopting qualitative approaches which take seriously people’s perspectives on their experiences of engaging with visualizations and which are attentive to social and cultural influences on engagement with artefacts like visualizations. In so doing, we bridge the scholastic paradigms of HCI and media and communications studies.

In this paper, we argue that contextual, social and cultural factors matter when it comes to engaging with data visualizations, and the field of data visualization research needs to pay attention to them. To fully understand how people engage with visualizations, it is important to acknowledge these factors. Drawing on empirical research with users of visualizations, we identify six factors that affect engagement:

  1. subject matter;
  2. source/media location;
  3. beliefs and opinions;
  4. time;
  5. emotions;
  6. confidence and skills.

We argue that these findings have implications for how ‘effectiveness’ is defined in relation to data visualizations. We suggest that technical measures like memorability, speed, accuracy of recall or consistency of comprehension do not adequately capture what users experience as an ‘effective’ visualization, as they fail to consider factors beyond the visualization text as dimensions of effectiveness. Although challenging and not necessarily easy to implement, we propose that acknowledgement of such factors — of socio-cultural differences amongst users, as well as the contexts in which they engage with visualizations — will lead to better understanding of how we might think about their effectiveness.

This paper is based on qualitative, empirical research with users of visualizations on the project ‘Seeing Data’ (http://seeingdata.org/). The users with whom we worked were not experts in data visualization, although some did have interest or experience in related fields, such as data, visual design, or the subject matter of some of the visualizations that we examined. The project aimed to explore factors in visualization consumption and production processes that affect engagement, and through this identify how effectiveness could be defined in this context. It addressed these questions through a range of methods, including focus group research, interviews and diary-keeping. Below, we locate our research in the context of other studies of user engagement with visualizations, describe our methods and findings, and discuss their implications for visualization research.

 

++++++++++

Research into engagements with visualizations

Measures of effectiveness

A number of HCI studies attempt to evaluate how users engage with visualizations. Many of these focus on the assessment of a specific element in the visualization engagement process, such as memorability, speed of task completion or recall, or the effectiveness of particular visual elements. For example, Huang, et al. (2009) focused on cognitive load (that is, the amount of interpretative work the brain has to do) in their study of visualization effectiveness. They argue that attention to cognitive load is important as it helps to overcome some of the limitations of other performance-based measures of visualization effectiveness by accounting for individual differences in cognitive effort despite similar task performance. Based on this hypothesis, they developed a model to test cognitive load, which they applied in a study with 30 postgraduate IT students.

The memorability of data is one of the performance-based measures of visualization effectiveness to which Huang, et al. refer. In contrast to Huang, et al.’s assertion that such measures have their limitations, Borkin, et al. (2013) argue that being able to identify and quantify what makes a visualization memorable is important, in order to be able to design effective visualizations, although they also acknowledge that a memorable visualization is not necessarily a comprehensible one. They carried out a study based on 410 single-panel visualizations, categorized by visualization type (e.g., bar chart, line graph), collected from news media sites, government reports, scientific journals and infographic sources. Using Amazon’s Mechanical Turk, 261 ‘workers’ participated in the study. To tell their readers about their participants, the authors report on the age range and ethnicity of their participants and on the employment and education characteristics of the Mechanical Turk worker population in general, not the specific group involved in their study. They conclude that the inclusion of colour and a ‘human recognisable object’ enhance memorability, that visualizations with high visual densities are more memorable than minimal visualization styles, and that common graph types are less memorable than unique types of visualizations.

Another performance-based measure of visualization effectiveness is response time, or time taken to complete a task. Chin, et al. (2009) tested dynamic data visualizations to find out which visualization methods were most helpful in the quick accomplishment of tasks. Their aim was to identify ways of representing data that allow users to grasp dynamic data ‘in forms that are intuitive and natural’ [4]. They assessed visualization effectiveness by measuring time taken to complete tasks, assessing task accuracy and asking participants to identify their satisfaction rates on a Likert scale. Their 15 participants were all researchers within the authors’ laboratory (because the tools they were testing were targeted at this specialist audience) who, according to the authors, had limited experience with visualization tools. The authors conclude that some visualization techniques used in some chart types are more effective than others for representing dynamic, real-time data.

Some studies focus on the assessment of the effectiveness of particular visual features within visualizations. Haroz and Whitney (2012) tested how layout, visual features like colour and motion and the inclusion of specific visualization elements affected users’ task performance. They conclude that it is important not to overwhelm the user with visual elements and that grouping elements makes it easier to complete some tasks in relation to visualizations. As a result of their study, they produced guidelines for effective visualization design, all of which focus on visual elements within visualizations themselves. Attention is focused on the effectiveness of visual elements and presentation styles and thus the emphasis is on the visualization itself. In this paper, we argue that whilst visual elements are important factors in determining the effectiveness of a visualization, who users are, contexts of visualization use and other factors outside of the visualization text are also important in determining visualization effectiveness [5].

Who are the users?

The HCI studies of visualization use and effectiveness that we have cited thus far provide little information about participants. Huang, et al. (2009) tell us that their participants were IT students. Borkin, et al. (2013) tell us a little more about Mechanical Turk participants, as noted, but neither sets of authors consider the ways in which demographic factors might impact on visualization engagement. Haroz and Whiteney (2012) tell us very little about their five participants, only that three were female and all were graduate or postgraduate students in psychology or computer science or trained university staff. Given the small number of participants, a more precise description of each one would have been possible. Chin, et al. (2009) also tell us little about their users. This lack of description about users is surprising in these ‘user’ studies. We argue that this kind of information is vital to the interpretation of research data and to visualization engagement. Who are the users? What are their circumstances? What do they bring to the visualizations? What do they want to see? What factors affect their engagement?

One exception can be seen in a study by Dadzie, et al. (2009) which took advantage of user-centred design approaches to develop visualization tools that could analyse data in the context of knowledge management. Their study provides more information than others cited thus far about the subject matter of the visualizations and tools that they researched. The authors also acknowledge that uses of data visualizations are part of broader sense-making processes and so provide detail about contextual factors which is absent from the other studies discussed here. For example, they identify that the paper uses an example of aerospace engineers investigating the causes of an issue with gas turbines, thus identifying the subject matter of the visualizations. They gathered and report on demographic data about participants and used qualitative methods including observation and interviews [6]. In these ways, unlike other HCI research, this study is attentive to factors beyond the visualization text which may play a role in shaping visualization engagement.

Similarly, Ziemkiewicz and Kosara’s (2009) study pointed to the role played by user characteristics in the complex process of visualization engagement. Building on previous research, they evaluated participants for five personality traits: openness, conscientiousness, extraversion, agreeableness, and neuroticism. They conclude that gender plays a role in visualization engagement, as they found that women preferred verbal to spatial metaphors. Other work by Kosara and others [7] acknowledges that different people respond to different cues when engaging with visualizations and that comprehension of visualization-related tasks can also vary across users. Likewise, Shah and Hoeffner (2002) argue that existing bias affects users’ interpretations of data visualizations. Thus a minority of studies have attended to the role played by demographic and contextual factors in visualization engagement, but they have not considered the implications of these factors for definitions of effectiveness.

Learning from media audience research

The literature discussed in the previous section notwithstanding, HCI studies of visualization use generally do not consider factors outside the visualization text which might play a role in user engagement. People are not lab rats, and their engagement with designed artefacts does not occur in situations free of cultural or social values and contexts. As a result, we argue that it is useful to study engagements with visualizations with some of the methods and concepts used within media and communication studies, which actively consider how social, cultural and contextual factors influence how users — or audiences — engage with cultural products.

These methods and concepts can be traced back to the work of Stuart Hall and others in the 1970s, which focused on the different ways in which people read media texts. Hall (1973) highlighted the importance of two elements in engagements with media products: the encoding of meaning as the media product is produced, and the decoding of meaning as it is used or consumed. The encoding of texts takes place within the sociocultural milieu of producers, whereas decoding happens within the sociocultural milieu of audiences. Audiences may share the same milieu as the producers and decode the message encoded in the text by producers — this is known as the preferred reading. They may have different ‘frameworks of knowledge’, socioeconomic relations, ‘technical infrastructures’ or relationships to producers [8] and therefore decode a different, unintended message. Such readings may be negotiated (the dominant message is understood and accepted at an abstract level, but not entirely) or oppositional (the intended message is understood, but an ‘alternative framework of reference’ is used to make meaning [9]). Following from Bourdieu’s (2010) influential work, Hall suggested that how media products get decoded depends on the educational and class background of the viewer, amongst other factors.

Another important contribution from media studies is the recognition of the role of emotions in engagements with media artefacts [10]. Designed products like visualizations both impact and mobilize emotions as people process or make sense of them (Döveling, et al., 2011). Konijn and ten Holt (2011) argue that we need to pay more attention to the relationship between media and emotions, making a useful distinction between mood as an underlying emotional state that exists prior to engagement and emotional states that result from engagement with media artefacts. They report that Forgas and East (2008) found that people’s mood can ‘have an impact on the believability of news, with people in bad moods being better at detecting lies than individuals who are in neutral or positive moods’ [11]. They also highlight how certain emotional responses improve recall of images, but different studies contradict each other on this point and so are inconclusive. Other writers have highlighted that emotions are culturally specific and socially constructed [12] and so need to be understood as socially and culturally produced, not simply individual or psychological.

In this paper, we argue that the kinds of grounded, bottom-up approaches which are commonplace in media audience research can make it possible to identify new issues in research into visualization use and engagement. What’s more, such research to date, by focusing on aspects like cognitive load, memorability and speed or accuracy of task completion, has failed to take into account factors which have regularly been found to influence engagements with media products. User tests never only test chart types and arrangements — they also assess users’ abilities and capacities, and by association, their social, cultural and educational backgrounds. This is why it is important to gather and provide data about who users are, which in turn influences how the effectiveness of a visualization can be measured and defined. With these concerns in mind, we undertook grounded, empirical research into the factors that affect engagement with data visualizations and the implications of these factors for understanding what constitutes an effective visualization. We describe the methods we used in our research on the Seeing Data project in the next section.

 

++++++++++

Methodology

We used a range of methods to develop understanding of the factors that affect engagements with visualizations, including diary-keeping, focus groups and interviews. Our primary method of data collection was focus groups, because they allow access to a large number of attitudes, feelings, beliefs and reactions in a short period of time, and because participants may take the initiative in the discussion, something we valued (Gibbs, 1997). Participants were drawn from organisations or social groups that were already meeting out of shared interests prior to our research. These groups had varying degrees of institutional structure: the art group had a more formalised structure involving various officers including a secretary; the group of young farmers had similarly structured positions; the civil society/voluntary organisation members’ were joined through a business structure. We opted for some homogeneity within focus groups because we support the principle that homogeneity results in understanding of others’ lifestyles and situations and so facilitates discussion (Krueger and Casey, 2000; Sanders, 1997).

In the focus groups, we asked participants to evaluate eight visualizations, which we chose (after much discussion) because they represented a diversity of subject matters, chart types, original media sources, formats (print and online; all but one (Figure 9) are available online) and degrees of interactivity. Another criterion was to include visualizations that aimed to either explain key points using data or invite exploration. Three of our chosen visualizations are of migration data, as this topic was a case study in our research, two of which (Figure 2 and Figure 5) we commissioned from a leading European visualization agency. We believe that our chosen visualizations represent a cross-section of visualizations that might be encountered in everyday life by people who are not experts in data visualization.

Our approach to sampling was purposive, based on the objectives of our study and population characteristics: we aimed to recruit an equal balance of participants who a) might be assumed to be interested in data, the visual, or migration, and so ‘already engaged’ in one of the issues at the heart of the project (an art class, an open data group, migrant groups and groups in areas affected by migration) and b) about whom we could not make these assumptions. In the event, we were more successful in recruiting from the former group than the latter. We carried out our research in four geographical locations which, given our focus on migration as a case study, we characterise as: rural/high migration; rural/low migration; urban/high migration; urban/low migration. We did this in order to achieve a balance between rural and urban and high and low migration. We thought that migration rates or prior experience with migration (either personally or through day-to-day contact) might affect responses to migration data depicted in visualizations, but in keeping with our grounded, bottom-up approach, we did not have expectations about differences. We carried out nine focus groups with a total of 46 participants, in the groups listed in Table 1.

 

Table 1: Focus group participants.
Type of groupNumber of participants
Art class (potentially interested in visual representation)4
Open data group (potentially interested in data)8
Two East European groups (potentially interested in migration)2; 4
Asian/British Asian group (potentially interested in migration)6
Civil society group (potentially interested migration, given their focus)4
Young people involved in farming in high migration area (potentially interested in migration, given their location)6
Rural community (not assumed to be interested in data, migration or the visual)6
Pilot group with representatives from most of the above categories6

 

Twenty-seven participants were female and 19 male; ages ranged from 11 to 70, with the 30–39 age range best represented (18 participants). Employment sectors were extremely diverse, including fields like hairdressing and cleaning, local government, agricultural work, teaching, media, retail and information services. All participants except four (two of whom were under the age of 16) had qualifications of some kind; 19 had completed tertiary education and 11 had higher degrees. As the study took place in the U.K., most participants (n=30) self-reported as British, and other nationalities included German, Indian, Lithuanian, Pakistani, Polish and Thai.

Focus group participants kept diaries of their encounters with data visualizations in the week before their focus group meetings, to induct them into what a data visualization is and where it might be found. They were provided with a template that asked them to detail when, where and how they saw the visualisation, what their first impressions where, how they felt and what they thought about the visualisation. To assist participants in understanding what we were asking them to do, we provided two sample diary entries. We also thought that these diaries would help to initiate discussion participants could talk about what they had seen, what (if anything) they had learnt from visualizations and how they had felt about them.

Focus groups lasted for two hours. During the focus groups, we instructed participants to look at each visualisation for as long as they liked, and for each visualisation to complete a notes sheet which was similar to the diary template. We asked participants to record their initial responses on a grid which identified whether they liked or learnt from each visualization, a sample of which is shown in Figure 1, as previous experience of running focus groups had indicated that visual information about participants’ views helped to initiate discussion. Focus groups were recorded, transcribed, and analysed through iterative code development in Nvivo.

 

The grid on which focus group participants recorded initial reactions to visualizations
 
Figure 1: The grid on which focus group participants recorded initial reactions to visualizations.

 

After the focus groups, we asked 13 participants to keep diaries for a month and to be interviewed about their diary-keeping experiences, in order to provide us with further information about encounters with visualizations ‘in the wild’. We selected these participants based on diversity of demographic characteristics, quality of original diaries and degree of engagement with our study. Seven participants agreed to do this. Their demographic characteristics were diverse, but it is noteworthy that five of them were educated to degree level, they were predominantly readers of the left-leaning Guardian broadsheet newspaper and ‘already engaged’ as described above. There was one exception: J.C. (male, 24, white British agricultural worker) who self-identified as a Conservative and/or U.K. Independence Party (UKIP) voter, and read a more conservative tabloid newspaper, the Daily Mail. Participants were asked to record their encounters with all the visualisations they saw every day, using the same template as for the earlier diary, i.e., recording what, when and how they saw visualisations, their first impressions, how they felt about them and what they thought. During the interviews the participants were asked to reflect on their overall encounter with data visualisation, with particular visualisations and their recollections of the focus group visualisations. We draw on these focus group discussions, diaries and interviews in the next section, which discusses the socio-cultural factors which we found have an effect on engagements with data visualizations.

 

++++++++++

Factors which affect engagements with visualizations

Through iterative coding and interpretation of our research data, we identified six factors which affect engagements with data visualizations: subject matter; source/media location; beliefs and opinions; time; emotions; confidence and skills. We discuss each of these below.

Subject matter

Visualizations do not exist in isolation of the subject matter that they represent. When subject matter spoke to participants’ interest, they were engaged — this was particularly evident with Civil Society group members who were interested in issues relating to migration and therefore in the migration visualizations. Ishmael (female, age 30, Indian, works in IT for an NGO) looked at the visualization of migration in the news shown in Figure 2 and found it interesting because of the subject matter. She brought in other information from her background knowledge to try to make sense of what she was seeing, asking herself questions like:

‘Okay, when did 9/11 take place?’ Then ‘How many times did this show up? When did these different things happen? How bold is it? Why is this showing up? Why did they associate this person as illegal? Why did they associate it as that?’ [...] How does that associate with the Labour Association? How does that associate with all these different things? (Ishmael).

 

Migration In The News, produced for the Migration Observatory, University of Oxford
 
Figure 2: Migration In The News (http://www.compas.ox.ac.uk/migrationinthenews/), produced for the Migration Observatory(http://www.migrationobservatory.ox.ac.uk), University of Oxford.

 

Ishmael was deeply interested in the subject matter and this influenced her engagement. In contrast, Chris (male 38, white British, agricultural worker) was not interested in any of the visualizations we showed him in the focus group. He was generally taciturn and disengaged, and we suspect he attended our focus group because his partner asked him to. He doubted his ability to make sense of visualizations and was not interested or confident to spend time looking. He also made a telling quip, interrupting his partner:

Sarah: I watch quite a lot of the news and I think sometimes they use graphs and charts and things to highlight their issue (Sarah, female, 34, white British retail worker).
Chris: To confuse you (Chris).

Chris had no qualifications, whereas Sarah did. Whilst formal education is not the only route to understanding visualizations, we suggest there is a link between Sarah’s education and her greater confidence in engaging with visualizations and their use in the media; we discuss this link further below. However, Chris’ lack of confidence in his abilities and his mistrust of the media did not stop him completely from looking at visualizations: he told us that when he came across visualizations in the Farmer’s Guide, a publication he read regularly because it speaks to his interests, he would take the time to look at them. Again, interest in subject matter is important.

Other participants were interested in the uses to which visualizations may be put. Visualizations that would be beneficial in their work were valued, as Theresa explained in the focus group in relation to the visualization in Figure 4 about migration in the census:

I probably didn’t get as far as everybody else because I spent too much time on the first one, the Census immigration one. I have a specific professional interest in that, it was something I hadn’t seen before and it was like Christmas had come early for me, wow here’s all this information! (Theresa, female, 50, white British, worked in local government intelligence)

Theresa’s ‘professional interest’ affected her diary keeping — she chose to look at visualizations that were useful to her, and she was not alone in this. Noon (female, 32, Thai, Ph.D. student) also looked at a number of visualizations about shark attacks because she was going diving and wanted to know whether she would be safe, even though she would have found the topic dull at other times. For Angela, visualizations that interested her were those which she could use to start conversations with her family. In her week-long diary she related viewing a Dr Who visualization:

Immediately thought it was a fun way to share so much information, and shared it from the screen with my husband almost immediately. Together we went through each section, almost storytelling and reminiscing, but also finding new facts. (Angela, female, 43, white British, working in technology education)

Subject matter is an important factor in determining whether people are interested in engaging with visualizations — either domain-related (persistent interest in a topic) as in Theresa’s case, or as a result of topicality (time-specific interest) as in Noon’s case. These examples show how it is more helpful to talk about effective engagement with visualizations with reference to subject matter and user interests.

Source or media location

The source of a visualization is important: it has implications for whether users trust visualizations. This is evident in the quote from Chris above: his concerns about the media setting out to confuse the public were shared by many focus group participants, and led some to view those visualizations encountered within the context of certain media as suspect. In contrast, some focus group participants trusted the migration visualizations which we commissioned and which carried the logo of the University of Oxford, because they felt that the ‘brand’ of this university invokes quality and authority. During the period of extended diary keeping, a somewhat different picture emerged. Participants tended to see visualizations in their favoured media (such as the Guardian or the Daily Mail newspapers) which they already trusted, so they were likely to believe the visualizations they saw there too. J.C. (male, 24, white British agricultural worker), who regularly reads the Daily Mail, demonstrated this when he remarked in his interview that ‘you see more things wrong or printed wrong in the Sun I think’. Given the ideological similarities between these two publications — they are both relatively conservative-leaning — this comment points to the importance of media location in user trust and engagement.

Beliefs and opinions

Participants trusted the newspapers they regularly read and therefore trusted the visualizations in these newspapers, because both the newspapers and the visualizations often fitted with their views of the world [13]. This points to the importance of beliefs and opinions in influencing how and whether people take time to engage with particular visualizations. Some participants said they liked visualizations that confirmed their beliefs and opinions. The visualization about migration in the news shown in Figure 2 led some of our participants to reflect on what they already believed about how the media report migration, as seen in this quote:

I would say it reinforced, the one about the media, how I feel. It’s not great the way that refugees and migrants are portrayed. But I wouldn’t say it changed because I had that view before; it just reinforced it I think (Sally, female, 48, white British, worked for a civil society group).

But it is not just when visualizations confirm existing beliefs that beliefs matter. Jason (male, 34, white British, IT worker) was surprised by the migration data in the ONS visualization in Figure 3. He said that he had not realised how many people in the U.K. were born in Ireland:

I was surprised that Irish immigrants were the most common in the U.K. I think the last Census — it was the ONS thing again — it was surprising, it was something I hadn’t even thought of and it was like, ‘Wow!’. For all the talk of immigration things, the fact that Irish immigrants are the most common, for a lot of the time until ten years ago, it was something I didn’t expect (Jason).

These data questioned what he believed and he enjoyed that experience. This suggests that some people might like, or be interested in, data in visualizations that call into question existing beliefs, because they provoke and challenge horizons. So beliefs and opinions matter in this way too. Thus the relationship between the subject matter of visualizations and participants’ beliefs and opinions was important.

 

Non-UK born census populations 1951-2011
 
Figure 3: Non-U.K. born census populations 1951–2011 (U.K. Office for National Statistics).

 

Time

Engaging with visualizations can be seen as work, or laborious, by people for whom doing so does not come easily. Because of this, having time available is crucial in determining whether people are willing to do this ‘work’. Most participants who said they lacked time to look at visualizations were women (seven out of nine), and they put their lack of time down to family and home commitments. Manini (female, 39, Indian, working in education project support) is a working mother who talked about how her combined paid and domestic labour were so tiring that when she finished her day, she did not want to look at news, on TV or elsewhere, and that included looking at visualizations. Such activities felt like ‘work’ to her, and she was too tired to undertake them at the end of her busy day. Arguably, time to look at visualizations and to engage in data discussions is a gendered issue: in Europe the greater burden of housework falls to women (European Social Survey, 2013) and Manini is one example of this. That three female participants needed to leave the same focus group early in order to collect children from school is a further example of how domestic labour impacts women’s ability and willingness to spend time looking at visualizations.

As Chris’ comments above suggest, educational background and class are also factors that affect engagement, and they affect time available to engage with visualizations too. J.C. was an agricultural worker who told us in an e-mail that his working hours were very long and this impacted on his ability to keep a month-long diary of engagements with visualizations:

Because I don’t have a lot of time to like read things and what have you, so if it’s kept simple and easy to read, then I’m more likely to be interested in it and reading it all and, and you know, to look at it, have a good look at it really. (J.C.)

Time also affected the variety of sources that our participants looked at. J.C.’s extended diary contained visualizations that only came from the Daily Mail, the newspaper that he regularly read, or publications associated with his work (a National Farmers Union report, for example). This contrasted sharply with those that Horace (male, 27, white British, working for a civil society group) discussed in his extended diary: these were much more numerous and came from a greater variety of locations. J.C. valued simple visualizations because he did not have time to dedicate to looking in detail. But when time is set aside to engage with visualizations, as in our focus groups, almost all participants found the experience enjoyable and fruitful, and were disappointed when the time to look at visualizations came to an end. Marty lamented being asked to stop looking at the visualization comparing the quality of life across countries, shown in Figure 4:

I could probably spend an hour on it (Marty, male, 38, German, working in biotech research) [14].

 

Better Life Index
 
Figure 4: Better Life Index (at http://www.oecdbetterlifeindex.org), Organisation for Economic Co-operation and Development (OECD).

 

Emotions

Strong emotions may arise when considering a visualization, and these also affect engagement. They can emerge as first impressions which play a role in determining whether people decide to commit time to looking at a visualization and interrogating the data within it. We found that some participants felt immediately confused when they first looked at some visualizations, and this put them off exploring them further — this happened to Chris in relation to all of the visualizations. Sally and Horace, both mentioned above, used strong language to describe their feelings when they looked at the visualizations of migration in the U.K. shown in Figures 2 and 5. The data caused them to reflect on how it must feel to be a migrant who comes to the U.K. and encounters the anti-immigration headlines of the media. They described themselves as feeling ‘guilty’ (Sally) and ‘ashamed’ to be British (Horace).

 

Migration In The Census, produced for the Migration Observatory, University of Oxford
 
Figure 5: Migration In The Census (http://www.compas.ox.ac.uk/migrationinthenews/), produced for the Migration Observatory(http://www.migrationobservatory.ox.ac.uk), University of Oxford.

 

Other participants had strong emotional responses to the visual style of some visualizations. The visualization of film box office receipts by the New York Times shown in Figure 6 divided participants, with some drawn to its aesthetic and some put off by it. Noon and Mark wrote in their focus group notes:

It was a pleasure to look at this visual presentation because of the co-ordination between the image and the message it carries (Noon).

Frustrated. It was an ugly representation to start with, difficult to see clearly, no information, just a mess (Mark, male, in his 30s, white British, local government data quality officer).

Emotions derive from a number of the factors discussed here: the subject matter of a visualization, the data within it, its source or media location, users’ beliefs and opinions, their mood at the time of engagement, or their responses to visual aspects of the visualization. Wherever they derive from, emotions play a role in how visualizations are experienced.

 

The Ebb and Flow of Box Office Receipts, 1986-2008
 
Figure 6: The Ebb & Flow of Box Office Receipts, 1986–2008 (at http://www.oecdbetterlifeindex.org), New York Times.

 

Confidence and skills

Users need to feel that they have the necessary skills to decode visualizations, and some participants indicated a lack of confidence in this regard. Sara, 45, a part-time careers advisor, said of one visualization: ‘It was all these circles and colours and I thought, that looks like a bit of hard work; don’t know if I understand’. Many of our participants expressed concern about their lack of skills, or they demonstrated that they did not have the required skills. In the first half of the quote from Sara, the words ‘all these circles and colours’ point to the importance of visual literacy skills. Three participants, Chanda (female, 37, Pakistani Asian, housewife), Haleema (female, 34, Pakistani Asian, unemployed) and Rania (female, 43, British Asian, housewife), all born in Pakistan, struggled to make sense of some visualizations because English was their second language. This demonstrates the importance of language skills in visualization engagement. After they left the focus group early due to caring commitments, two remaining women whose first language was English, Z (46, female, British Asian, careers advisor) and Sara (mentioned earlier), both British of Pakistani heritage, said that we researchers should take into account those women’s English language skills, arguing that if we had been speaking to them in their first language, we would have got a different picture of their engagement with visualizations. Nationality and attendant linguistic skills are therefore important factors that affect how people understand visualizations.

Mathematical and statistical skills, as well as the ability to read particular chart types, are also needed. This was the case with the visualization of global fresh water consumption shown in Figure 7, which uses a Sankey Diagram. This type of visualization was new to many of our participants who therefore did not like it. However, Robert said, ‘I didn’t hate it because it made me want to try and put a little bit of effort into navigate those lines’ (Robert, male, 46, white British, business data analyst). Although Robert saw the new visualization type as a challenge, Sara’s comment — ‘don’t know if I understand’ — captures the lack of confidence that some participants felt when they encountered unfamiliar visualizations. She also commented on the need for literacy and numeracy skills:

I think to understand some of these, not all of them, I think you would have had to be at a certain level of literacy and numeracy to be able to understand some of this information [...] How would you know what that is and what all this is unless you’ve got a certain level of maths skills or English skills as well? (Sara)

 

Top Ten Freshwater Consumers
 
Figure 7: Top Ten Freshwater Consumers (at https://www.scientificamerican.com/article/water-in-water-out/), Scientific American.

 

Participants who kept a visualization diary for a month identified that critical thinking skills are also helpful in making sense of visualizations, such as asking what has been left out, or what point of view has been prioritised. In her month-long diary, Sally thought about these things in relation to a visualization about the amount of people currently in slavery:

I did wonder how they got this data and how reliable it is — there could be a far higher number of people in slavery who are not being counted. The [visualization] itself does not say where or how the data was collected although this info was presumably in the full report (Sally).

Educational background is important in relation to critical thinking skills: some participants identified that higher education helped them develop such skills. Horace made the connection like this:

A Master’s degree is actually learning to look at an argument and take everything with a pinch of salt. And go ‘okay fine, um it’s probably true.’ [...] Okay so they’ve made a conclusion and you’re going to draw that conclusion in your thesis but where did I draw that conclusion from? Was it from a valid source? (Horace)

J.C.’s confidence with simple chart types demonstrates how his school education provided a baseline level of visualization literacy. Meanwhile, Horace was more confident in his abilities to understand and assess the data visualizations than Chris because he felt he had the training to do so. However, as noted above, demographic factors like gender and class also influence personal characteristics like confidence.

In total, we argue that accounting for the factors discussed here is vital to fully understand how users engage with visualizations. Such factors play a role because so many of the activities in which we engage, including engaging with visualizations, always take place within culture and within the social. They also have implications for definitions of effectiveness in relation to data visualizations.

 

++++++++++

Implications of findings for definitions of effectiveness

Existing HCI visualization research understands effectiveness as relating to things like memorability or speed of comprehension (Borkin, et al., 2013; Anderson, et al., 2011), or to specific visual characteristics like low data-ink ratio (Borkin, et al., 2013), grouping of elements (Haroz and Whitney, 2012) or the use of visual metaphors (Borkin, et al., 2013). Many studies do not explicitly define how the term effectiveness is understood. We argue that the bottom-up, grounded approach that we adopted in our research made it possible to identify a broader range of understandings and potential definitions of effectiveness than that which currently appears in HCI literature. We did this not by explicitly asking participants if they thought any of the visualizations that they saw in the focus groups or in their daily lives were ‘effective’. Instead, we asked them what they felt about the visualizations, whether they liked them, what, if anything, they learned from them, and what they thought about them. Taken together, participants’ responses tell us something about what constitutes an effective visualization and how effectiveness might be defined in this context. Based on our findings, we argue that definitions of effectiveness need to take into account the fact that people do not always look at visualizations with the aim of accessing specific information quickly and remembering it indefinitely. Visualizations in the media that are targeted at non-specialists might aim to persuade, for example. They all need to attract attention in order to draw people in and convince them to to commit time to finding out about the data on which the visualization is based. Visualizations might stimulate particular emotions, which inspire people to look longer, deeper or further. They might provoke interest, or indeed prompt disinterest.

Considering our findings, effectiveness could be defined in a number of different ways, as listed in Table 2 below:

 

Table 2: Ways of defining effectiveness.
Possible definition of effectivenessExamples from research
Provoking questions and the desire to engage in discussions with othersIshmael’s reaction to the Migration in the Census visualization and Angela’s engagement with the Dr. Who visualization
Creating empathy for other humans in the dataSally and Horace’s reactions to the Migration in the News visualization
Generating enough curiosity to draw the user inJason’s reaction to the Migration in the Census visualization
Reinforcing or backing up existing knowledgeSally’s reaction to the Migration in the News visualization
Provoking surpriseJason’s reaction to the Non-U.K. Born Census Populations visualization
Persuading or changing mindsJason’s reaction to the Non-U.K. Born Census Populations visualization
Learning something newJason’s reaction to the Non-U.K. Born Census Populations visualization
Acquiring new confidence in reading visualizations or dataRobert’s response to Freshwater Consumers
Finding the data useful for one’s own purposesTheresa describing herself as having a ‘professional interest’
Enabling an informed or critical engagement with a topicSally’s reaction to the visualization about numbers of people in slavery
Having a pleasurable experience or being entertainedMarty enjoyed the Better Life Index so much that he wishes he could look for longer
Enjoying the visualNoon’s reaction to the Ebb and Flow of Box Office Receipts visualizations
Provoking a strong emotional responseAll participants’ reactions to the Ebb and Flow of Box Office Receipts visualizations

 

Enabling the quick extraction of accurate information is only one of several possible ways of defining what constitutes an effective visualization. We saw in our focus groups that spending time looking at and asking questions of visualizations was essential for extracting information (although this was not the case for every person looking at every visualization, and some visualizations require more time than others). A definition of effectiveness that focuses on speed fails to recognise that having and dedicating time is an important factor in visualization engagement. The definitions listed above, which emerge from our research, are quite distinct from definitions that emphasise speedy and accurate information extraction. Notably, they are context-specific. This means that what constitutes effectiveness is fluid — no single definition will apply across all engagements with visualizations. For example, being entertained by a visualization is relevant in some contexts, but not others. Visualizations have various objectives: to communicate research findings; to inform a general audience; to influence decision-making; to enable exploration and analysis of data; to surprise and affect behaviour. Given this, we suggest that the factors that affect engagement which we identified in our research should be seen as dimensions of effectiveness, which carry different weight in relation to different visualizations, contexts and purposes. Returning to concepts introduced earlier (Hall, 1973), many of these factors lie outside of the control of visualisers because they relate to the decoding, not the encoding, of visualization texts. In other words, whether a visualization is effective depends in large part on how, by whom, when and where it is decoded.

 

++++++++++

Conclusion

In this paper, we have set out an approach to studying the factors that affect non-experts’ engagements with data visualizations that builds on methods and insights from media audience research. In so doing, we proposed bridging HCI and media and communication studies paradigms, in order to develop understanding of how people engage with data visualizations. Such an approach gives attention to the role played by a range of social, cultural and other extra-textual and contextual factors in the ways in which people experience media and communication artefacts like visualizations. Social and cultural context matter — visualizations are always encountered in such contexts, so we need to understand their role. Turning to media audience research is necessary because most research with visualization users has not acknowledged the role played by these factors in their engagements with visualizations.

Using such an approach, we identified six factors that affect engagement with data visualizations. These factors influenced users’ desire to stay with a visualization and explore the data within it. They relate to the user, the visualization text, and the context of engagement. In each engagement with each visualization, a unique convergence of these elements takes place. This has important implications for what we understand an effective visualization to be: attentive to viewers’ needs and abilities, acknowledging of differing viewing contexts and alert to the different weightings that these six factors carry in different user/text/context convergences. Our findings do not lead to a simple checklist which guarantees the production of universally effective visualizations. Rather, they point towards a challenge for the future, for visualization producers: to think with and through these socio-cultural factors in order to generate data visualizations that reflect the highly contextualised and mediated character of everyday engagements with them.

 

Your Olympic Athlete Body Match
 
Figure 8: Your Olympic Athlete Body Match, BBC online, July 2014.

 

 

Clicks Don't Lie
 
Figure 9: The Clicks Don’t Lie, the Metro newspaper, July 2014.

  End of article

 

About the authors

Helen Kennedy is Professor of Digital Society at the University of Sheffield. Her research has traversed digital media landscapes, covering a range of topics and she has published widely in this field. She is currently interested in critical approaches to big data and data visualisations, how to make data more accessible to ordinary citizens, and whether data matter to people.
E-mail: h [dot] kennedy [at] sheffield [dot] ac [dot] uk

Rosemary Lucy Hill is a Lecturer in Sociology at the University of Leeds. She researches gender, data visualisations and popular music. She has published on the metal media, the moral panic around emo, subcultural theory and semiotics. Current interests centre around the use of data visualisations in media activism. She has appeared on BBC Radio 4’s Thinking Allowed on the subject of women fans, metal and subcultures.
E-mail: r [dot] l [dot] hill [at] leeds [dot] ac [dot] uk

William Allen is a Research Officer at the Migration Observatory, in the Centre on Migration, Policy, and Society (COMPAS) at the University of Oxford. His research and published work examines the links among media, policy-making, and public perceptions about migration issues. He is also interested in how non-academic groups, particularly those in civil society, engage with migration data, research, and statistics.
E-mail: william [dot] allen [at] compas [dot] ox [dot] ac [dot] uk

Andy Kirk is a U.K.-based freelance data visualization specialist and editor of the award-winning Web site visualisingdata.com. He is author of two books about data visualization. The latest, Data visualisation: A handbook for data driven design was published by Sage in 2016. He is a visiting lecturer teaching data visualisation modules on Master’s programmes at the Maryland Institute College of Art (MICA) and Imperial College, London. Andy offers design consultancy and delivers training workshops around the world.
E-mail: andy [at] visualisingdata [dot] com

 

Acknowledgements

Seeing Data (http://seeingdata.org/) was supported by U.K. Arts and Humanities Research Council (AHRC) Digital Transformations Large Grant (Grant No. AH/L009986/1). With thanks to the AHRC for this support.

 

Notes

1. Gitelman and Jackson, 2013, p. 12.

2. See Sluijs, 2008, at http://firstmonday.org/article/view/2306/2061, and Hochman and Manovich, 2013, at http://firstmonday.org/article/view/4711/3698, for examples published just in this journal.

3. The work of Dadzie, et al. (2009) is an exception.

4. Chin, et al., 2009, p. 212.

5. In another paper, some of us examine the visualization text to assess the semiotic resources being used and the work that they do (Kennedy, et al., 2016).

6. Dadzie, et al., 2009, p. 207.

7. For example Kosara, et al., 2003.

8. Hall, 1973, p. 130.

9. Ibid.

10. In the field of design, Desmet (2002) and Norman (2004) have also addressed the importance of emotions.

11. Konijn and ten Holt 2011, p. 49.

12. For example Jaggar (1989) and Ahmed (2004).

13. This point is contradicted by the previous comment from J.C., in which he demonstrates a preference for one of two publications which arguably share similar world views. This point notwithstanding, there was evidence of the importance of beliefs and opinions in our research.

14. The emergence of time as an important factor in our research points to the related significance of perceptions of time and attitudes to managing time, which are beyond the scope of the discussion here.

 

References

Sara Ahmed, 2004. The cultural politics of emotion. London: Routledge.

Erik W. Anderson, Kristin C. Potter, Laura E. Matzen, Jason F. Shepherd, Gilbert A. Preston and Cláudio T. Silva, 2011. “A user study of visualization effectiveness using EEG and cognitive load,” Computer Graphics Forum, volume 30, number 3, pp. 791–800.
doi: http://doi.org/10.1111/j.1467-8659.2011.01928.x, accessed 6 October 2016.

Michelle A. Borkin, Azalea A. Vo, Zoya Bylinskii, Phillip Isola, Shashank Sunkavalli, Aude Oliva and Hanspeter Pfister, 2013. “What makes a visualization memorable?” IEEE Transactions on Visualization and Computer Graphics, volume 19, number 12, pp. 2,306–2,315.
doi: http://doi.org/10.1109/TVCG.2013.234, accessed 6 October 2016.

Pierre Bourdieu, 2010. Distinction: A social critique of the judgement of taste. Translated by Richard Nice. London: Routledge.

George Chin, Mudita Singhal, Grant Nakamura, Vidhya Gurumoorthi and Natalie Freeman-Cadoret, 2009. “Visual analysis of dynamic data streams,” Information Visualization, volume 8, number 3, pp. 212–229.
doi: http://doi.org/10.1057/ivs.2009.18, accessed 6 October 2016.

Aba-Sah Dadzie, Vitaveska Lanfranchi and Daniela Petrelli, 2009. “Seeing is believing: Linking data with knowledge,” Information Visualization, volume 8, number 3, pp. 197–211.
doi: http://doi.org/10.1057/ivs.2009.11, accessed 6 October 2016.

Pieter Desmet, 2002. Designing emotions. Delft: Delft University of Technology, Department of Industrial Design.

Katrin Döveling, Christian von Scheve and Elly A. Konijn (editors), 2011. Routledge handbook of emotions and mass media. London: Routledge.

European Social Survey, 2013. “Exploring public attitudes, informing public policy. Selected findings from the first five rounds,” at http://www.europeansocialsurvey.org/docs/findings/ESS1_5_select_findings.pdf, accessed 6 October 2016.

Joseph P. Forgas and Rebekah East, 2008. “On being happy and gullible: Mood effects on skepticism and the detection of deception,” Journal of Experimental Social Psychology, volume 44, number 5, pp. 1,362–1,367.
doi: http://dx.doi.org/10.1016/j.jesp.2008.04.010, accessed 6 October 2016.

Anita Gibbs, 1997. “Focus groups,” Social Research Update, number 19, pp. 1–7, and at http://sru.soc.surrey.ac.uk/SRU19.html, accessed 6 October 2016.

Lisa Gitelman and Virginia Jackson, 2013. “Introduction,” In: Lisa Gitelman (editor). “Raw data” is an oxymoron. Cambridge, Mass.: MIT Press, pp. 1–14.

Stuart Hall, 1973. Encoding and decoding in the television discourse. Birmingham: Centre for Cultural Studies, University of Birmingham.

Steve Haroz and David Whitney, 2012. “How capacity limits of attention influence information visualization effectiveness,” IEEE Transactions on Visualization and Computer Graphics, volume 18, number 12, pp. 2,402–2,410.
doi: http://dx.doi.org/10.1109/TVCG.2012.233, accessed 6 October 2016.

Nadav Hochman and Lev Manovich, 2013. “Zooming into an Instagram City: Reading the local through social media,” First Monday, volume 18, number 7, at http://firstmonday.org/article/view/4711/3698, accessed 6 October 2016.
doi: http://dx.doi.org/10.5210/fm.v18i7.4711, accessed 6 October 2016.

Weidong Huang, Peter Eades and Seok-Hee Hong, 2009. “Measuring effectiveness of graph visualizations: A cognitive load perspective,” Information Visualization, volume 8, number 3, pp. 139–152.
doi: http://dx.doi.org/10.1057/ivs.2009.10, accessed 6 October 2016.

Alison M. Jaggar, 1989. “Love and knowledge: Emotion in feminist epistemology,” Inquiry, volume 32, number 2, pp. 151–176.
doi: http://dx.doi.org/10.1080/00201748908602185, accessed 6 October 2016.

Helen Kennedy, Rosemary Lucy Hill, Giorgia Aiello and William Allen, 2016. “The work that visualisation conventions do,” Information, Communication & Society, volume 19, number 6, pp. 715–735.
doi: http://dx.doi.org/10.1080/1369118X.2016.1153126, accessed 6 October 2016.

Rob Kitchin, 2014. The data revolution: Big data, open data, data infrastructures & their consequences. London: Sage.

Elly A. Konijn and Jelte M. ten Holt, 2011. “From noise to nucleus: Emotion as key construct in processing media messages,” In: Katrin Döveling, Christian von Scheve and Elly A. Konijn (editors), 2011. Routledge handbook of emotions and mass media. London: Routledge, pp. 37–59.

Robert Kosara, Christopher G. Healey, Victoria Interrante, David H. Laidlaw and Colin Ware, 2003. “User studies: Why, how, and when?” IEEE Computer Graphics and Applications, volume 23, number 4, pp. 20–25.
doi: http://dx.doi.org/10.1109/MCG.2003.1210860, accessed 6 October 2016.

Viktor Mayer-Schönberger and Kenneth Cukier, 2013. Big data: A revolution that will transform how we live, work, and think. Boston, Mass: Houghton Mifflin Harcourt.

Donald A. Norman, 2004. Emotional design: Why we love (or hate) everyday things. New York: Basic Books.

Lynn M. Sanders, 1997. “Against deliberation,” Political Theory, volume 25, number 3, pp. 347–376.
doi: http://dx.doi.org/10.1177/0090591797025003002, accessed 6 October 2016.

Priti Shah and James Hoeffner, 2002. “Review of graph comprehension research: Implications for instruction,” Educational Psychology Review, volume 14, number 1, pp. 47–69.
doi: http://dx.doi.org/10.1023/A:1013180410169, accessed 6 October 2016.

Jasper P. Sluijs, 2008. “Sense.Us: towards a more social ‘social visualization’,” First Monday, volume 13, number 12, at http://firstmonday.org/article/view/2306/2061, accessed 6 October 2016.

Caroline Ziemkiewicz and Robert Kosara, 2009. “Preconceptions and individual differences in understanding visual metaphors,” Computer Graphics Forum, volume 28, number 3, pp. 911–918.
doi: http://dx.doi.org/10.1111/j.1467-8659.2009.01442.x, accessed 6 October 2016.

 


Editorial history

Received 18 January 2016; revised 22 September 2016; revised 31 October 2016; accepted 1 November 2016.


Creative Commons License
This paper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Engaging with (big) data visualizations: Factors that affect engagement and resulting new definitions of effectiveness
by Helen Kennedy, Rosemary Lucy Hill, William Allen, and Andy Kirk.
First Monday, Volume 21, Number 11 - 7 November 2016
http://www.firstmonday.dk/ojs/index.php/fm/article/view/6389/5652
doi: http://dx.doi.org/10.5210/fm.v21i11.6389





A Great Cities Initiative of the University of Illinois at Chicago University Library.

© First Monday, 1995-2017. ISSN 1396-0466.