Prioritizing Unread Emails: An Experiment Showing that People Send Urgent Responses Before Important or Short Ones

Anna Cox, Jon Bird, Duncan P Brumby, Marta E Cecchinato, Sandy JJ Gould

This is the author version of the paper. Please cite the published version of the paper that appears in Human-Computer Interaction, the Taylor-Francis Journal:

Cox, A. L., Bird, J., Brumby, D.P., Cecchinato, M.E., & Gould, S. J.J. (2020). Prioritizing unread e-mails: People send urgent responses before important or short ones. Human–Computer Interaction, 1–24. doi: 10.1080/07370024.2020.1835481

Abstract

People are overwhelmed by the volume of email that they receive. To ensure their emails are read, senders sometimes use explicit inbox-level cues in an attempt to garner the receiver’s attention. We report the results of a field experiment that investigates whether and how such cues influence recipients’ email processing behavior. Forty-five participants were sent 360 emails each over a three-week period. Inbox-level cues were given to indicate: (1) the urgency of responding, (2) the time that would be required to work on a response, (3) the importance of responding, (4) and the salience of that importance. Results show that email prioritization is influenced by an interaction between these cues. When emails were not time-sensitive, participants sensibly prioritized responses to messages that were most important and required the least effort to respond to. This rational triaging strategy faltered when emails required a time-sensitive response; urgent messages were responded to quickly regardless of other cues. The results are discussed with reference to Kahneman’s dual-process theory of judgment and decision making.

1. Introduction

Although email interfaces have not changed much in the past decade, how we manage our inbox has changed a lot, as a result of the growing number of messages we receive (The Radicati Group, 2015) across a growing collection of accounts (Cecchinato et al., 2015). This growth has increased the workload associated with tasks necessary to filter and manage the inbox and makes for an overwhelming experience known as email overload (Dabbish & Kraut, 2006; Eppler & Mengis, 2004). Email triage --“the process of going through unhandled email and deciding what to do with it” (Neustaedter et al., 2005, p. 1997)-- is particularly challenging because, while some emails are critical, the majority are either irrelevant or do not require immediate attention (Buthpitiya et al., 2009). Thus, researchers have been arguing for more work that helps users reduce the burden of decision making associated with managing emails (Grandhi & Lanagan-Leitzel, 2016).

Whittaker et al. (2005) argued that people need help in identifying important messages and called for "systems that support users in detecting and processing messages associated with important tasks" (p. 6). However despite current solutions that rely on machine learning algorithms to cluster messages into priority/non-priority groups (e.g. Microsoft Outlook’s Clutter and Gmail’s Priority Inbox), users may still be receiving large numbers of messages within each of those clusters and thus may still need to “[sift] through multiple messages attempting to determine how each message might relate to their outstanding tasks” (Whittaker et al., 2005, p. 6).

Email etiquette often requires using clear and directive subject lines. Wainer et al. (2011) demonstrated that people use the subject line as a filter mechanism to help them decide which messages to prioritize. However, there are several ways in which the subject line can influence the decision process. For example, although previous work (Porter & Whitcomb, 2005; Sappleton & Lourenço, 2016; Wainer et al., 2011) has found that leaving a blank subject line or omitting information can help feed the receiver’s curiosity to react to the message, Sappleton and Laurenço (2016) found that leaving a subject line blank was not always enough though to warrant a response.

To date, very little work has looked at how various email factors, such as importance, urgency and cost of reply, aid users in sifting through their emails and making appropriate decisions. Moreover, when these factors have been investigated it has often been done in an isolated manner, in the lab using simulated inboxes. The reality of managing one’s inbox is more complicated and messier than is simulated in such investigations. It is therefore timely to explore how these factors interact to influence how people prioritize their emails.

To address our lack of understanding about how different factors in a subject line interact to affect the decision-making process of email replies (especially in a situated context), in this paper we report on a study with 45 participants, who collectively over a three-week period replied from their own inboxes to 16,200 emails we sent them. For each email we manipulated four email subject line cues (urgency, cost, importance, salience of importance) and measured the number of responses and response times. Our findings show that people prioritized responses to important messages, except when emails required a time-sensitive response; urgent messages were responded to quickly regardless of other cues. We make three novel contributions. First, we provide empirical evidence of how responses are prioritized based on explicit subject line cues. Second, we demonstrate how our field experiment can be used as a valid method to investigate the daily triage of emails in a situated context. Finally, we elaborate on implications for design that can inform how future email management systems should be designed.

2. Related Work

2.1. Current Email Systems

Over the past few years, a variety of email support tools have been developed to help people quickly and easily identify important unread messages in their inbox e.g., Gmail’s Priority Inbox, VIP Lists on iOS Mail, and Microsoft Outlook’s Clutter. These approaches try to identify important emails based on user-defined information about the importance of the sender or system-based assumptions about a sender’s importance based on the user’s prior responsiveness to messages from that sender (indicated by whether or not a response was given and the speed of response). VIP Lists on iOS identifies emails coming from people that are important to the user (e.g., a boss, a close colleague, or family member), so that these messages can be given attention first. Gmail’s Priority Inbox extends this concept by using automated machine learning techniques to assign greater importance to emails that users have responded to quickly in the past. Microsoft Outlook’s Clutter moves emails that are usually ignored or not responded to into a separate inbox. These new tools have moved beyond threading or clustering of related messages to making judgements about what is important to the receiver.

Whilst these tools are a step forward compared to those advocated by Whittaker et al. (2005), they are as yet unable to differentiate between messages from the same sender that relate to tasks of different importance. As a consequence, when people send an email, they often try to give explicit cues about the urgency, cost and importance of replying to their message in order to influence the responsiveness of the sender. For example, using a subject line, “URGENT: Can we meet today?” is giving a clear and explicit signal that the receiver can pick up on as they triage their unread messages, hopefully eliciting a timely response. Most email clients also allow senders to use and set explicit Priority Flags (e.g., “!! High Priority” vs. “- Low Priority”). A third but perhaps less frequently used cue is an indication of the type of response required: subject lines such as “For Information Only” and “NNTR” (No Need To Respond) indicate that the message can be read and then simply filed or deleted. Other cues such as “Quick short response required” are sometimes included in email subject lines with the aim of indicating that the response required will only take a very short time to compose. All of these cues are explicitly created by the sender of an email in an attempt to influence how the receiver prioritizes their responses to the many unread emails in their inbox.

Many factors can potentially determine if and when people decide whether to file, defer, delete, or respond to a message. In the email literature there are four particularly important factors that influence this triaging behavior (Wainer et al., 2011): the perceived urgency of giving a timely response; the cost of composing a response; the perceived importance of responding to an email; and the salience of that importance to a recipient. In the following subsections we discuss each of these in turn and argue that it is necessary to develop a detailed understanding of how these factors interact for the development of more efficient email management practices and support tools.

2.2 The Urgency of a Message

Prior research suggests that people prioritize emails that are time-sensitive and require an urgent response (Tyler & Tang, 2003). Receivers likely use information such as the identity of the sender in order to determine the urgency required (Siu et al., 2006). For example, an email from a senior colleague might elicit an urgent response. Indeed, this assumption underpins many email support tools, which mark messages from particular people as important. For instance, Google Priority Inbox ranks messages from people that are usually responded to as important, whereas VIP lists on iOS Mail allow the user to explicitly define ‘important people’. However, a study by Karagiannis and Vojnovic (2009) found very little evidence of email response times being influenced by organizational seniority. This would suggest that when prioritizing responses to emails, people have a more nuanced strategy than to simply respond to their senior colleagues quickly. Instead, other features of the message must be used to determine the urgency of response required.

People sometimes give clear and explicit inbox-level cues about the urgency of response required to their message. For example, by using the subject line “URGENT: Please review final draft of CHI paper (attached) prior to submission deadline”, the sender makes it clear to the receiver that they require a prompt and timely response. While urgency and importance often align, there are many cases when these two features are orthogonal. For instance, one might be cc’d on a message marked as “URGENT”. While an urgent response is required by the primary recipient, the message does not require a timely response from those listed in the ‘cc’ line.

The question of how urgency cues affect email response behavior has not been given sufficient attention in the HCI literature. Is it that people assess both the urgency and the importance of a message when deciding whether to respond, or do urgent but less important messages garner undue attention? Research on human decision-making would suggest that decisions made in haste are instinctive and automatic, whereas decisions that are made slowly are rational and deliberative (Kahneman, 2011). Similarly, research on how people multitask has found that ‘urgent’ events are often prioritized over less urgent but more important ones (Bogunovich & Salvucci, 2011; Kerstholt, 1994). This suggests that urgency trumps importance. However, it is not clear whether or how this cognitive bias extends to how people triage email; do people prioritize responses to messages that a sender has indicated require an urgent response? The results of the study presented here provide a detailed, empirically grounded understanding of how people trade-off explicit inbox-level cues about the expressed urgency and importance of responding to an email.

2.2. The Cost of Making a Response

Not all emails demand a response (Di Castro et al., 2016), but for those that do, responses can vary considerably. Some require only a one-word response, while others require a lengthy and engaged answer. This difference can then be compounded by the fact that email is also triaged on a variety of devices (Cecchinato et al., 2016; Collins et al., 2015). Writing a one-word response on a tiny smartphone keyboard is easy, but writing a long and complex response on the same device is difficult and time consuming. Matthews et al. (2009) show that the difficulty of composing messages on small mobile keyboards means that users often defer writing longer replies. Cecchinato et al. (2015) found that people tended to use their smartphones to reply to messages that were urgent and required only a short, quick response. Similar behaviors have been observed by (Kooti et al., 2015), who found that email responses from smartphones were the quickest and shortest, followed by tablets and then laptops/PCs.

In addition to the physical costs of typing replies, people consider the amount of work required in order to be able to respond to a given message. Karagiannis and Vojnovic (2009) found that people responded particularly slowly to emails that had an attachment, suggesting that people are sensitive to variations in the time costs of reading emails. Sinking a lot of time into responding to one complex email necessarily takes away from time that could be given over to responding to many messages that require only a brief and simple response. Human decision-making research shows that people can make these kinds of trade-offs about how to allocate effort over time in isolation (Jarvstad et al., 2012). We investigate here how people trade-off the cost of responding against multiple other critical factors (i.e., response urgency and importance).

2.3. The Importance of a Message

People seem to prioritize emails that have been designated as ‘important’ by the sender (Wainer et al., 2011). A sender can use inbox-level cues to try and signal the relative importance of a message, for instance, by communicating this in the subject line of the email or by using importance indicators found in many email clients (e.g., Priority Flags). In an experiment by Wainer et al. (2011), participants had to process emails to organize a fictional event. Messages that were marked as important by the sender always contained important information that was relevant to the organization of the event. Their study showed that people were more likely to respond to messages if they included this reliable indicator of importance.

However, importance flags can be misused. What is important to the sender of an email may not always be important to the recipient of that message. In another lab study, Kraut et al. (2002) found that sender-determined indicators of importance tended not to be used by the receiver to prioritize attention to messages. A possible explanation for this finding is that there is often considerable variability in whether and how people use importance indicators when sending email. As a result, it can be difficult to know whether an email that is flagged as important really is of importance to the recipient of that message, or whether it is an indicator that the sender thinks the email is important, or whether, in fact, the sender is trying to indicate that an urgent response is required. This prior research would therefore suggest that people can and do use importance indicators when prioritizing the unread emails that they receive, but only when these flags are used in a consistent and meaningful way by senders.

2.4. The Salience of a Message’s Importance

The importance of a message is sometimes obvious to a recipient from the subject line. Sometimes, though, more engagement (such as reading the body of the message) is required before the importance of a message becomes clear. The relationship between the salience of a message’s importance and people’s processing is not always as intuitive as one might expect. When sending email, many strive to give accurate and explicit information in the subject line about the importance and urgency of their message. Paradoxically, Porter and Whitcomb (2005) found that blank subject lines actually yielded the highest response rates. This finding was corroborated by Wainer et al. (2011), whose participants received messages with ambiguous subject lines. The ‘information gap’ created by the ambiguous subject lines in their study influenced how emails were prioritized: people attended to messages with ambiguous subject lines more quickly. Receiving a handful of emails with ambiguous or empty subject lines might effectively pique one’s curiosity. But would this still be the case if many emails were received that had ambiguous or blank subject lines? We are interested in further investigating how people respond to how these different kinds of inbox level cues are used to indicate the importance of an email.

Whilst factors such as importance, urgency and length of response have been previously studied in independently, in real life settings these cues are often used in combination. However, no work to date has looked at how these factors interact with each other to influence email response behavior. Our research question is therefore: To what extent do people respond to competing subject line cues about importance, urgency, and cost of replying when managing their situated daily email?

3. An investigation of the influence of multiple inbox level cues on email response behavior

In our review of the literature we identified four different types of inbox-level cue that people use when deciding how to prioritize responding to unread email: (1) the urgency of responding, (2) the cost of responding, (3) the importance of the message, and (4) the salience of that importance. These factors have often been studied separately and often in a lab setting, with participants doing an artificial email task that involves processing messages that have no personal meaning. However, we want to know how these factors interact with one another and their impact on situated email behaviour. This is important because the senders of emails often use multiple cues in combination to try and garner a fast response to the emails that they send. Taking inspiration from Wainer et al.‘s (2011) lab-based experiment on email behavior, we developed and deployed a field experiment in which we sent people emails to their primary existing email addresses to find out which combination of inbox-level cues resulted in the most responsive behaviour, measured by whether or not they responded and how fast they responded. To do this, we sent 45 participants 360 emails each over a three-week period and recorded their email response patterns. These experimenter-generated emails were sent to participants’ main existing email accounts so that these emails would sit amongst the usual variety and quantity of email that our participants received. The external pressures on participants’ lives limited their ability to respond to our emails and we therefore expected their responses to be selective, focused on those emails that captured their attention.

Our primary concern in conducting this field experiment was to answer our research question and learn to what extent people respond to competing subject line cues about urgency, cost of replying and importance when managing their situated daily email. Given the variety of factors that might influence people’s decision making around inbox-management, and the need to understand these factors individually and collectively, we first make the following hypotheses about how these individual factors are likely to impact response rate (number of responses) and response time based on previous literature:

Urgency:

H1a– participants will respond to more emails where a less urgent response is required than those that require fast responses. This is because participants are more likely to miss the opportunity to respond, on time, to incoming emails that require fast responses.

H1b – participants will respond more quickly to incoming emails that require fast responses than those that have a longer response time.

Cost:

H2a– participants are more likely to respond to incoming emails that require a low cost response (i.e. require less effort to respond to) than those where the cost of making a response is higher.

H2b – participants will respond more quickly to incoming emails that require a lower cost response than those that require a high cost response.

Importance:

H3a – participants are more likely to respond to incoming emails that are marked as important than those marked as low importance.

H3b – participants will respond more quickly to incoming emails with a higher marked importance than those with a lower marked importance.

Salience:

H4a – participants are more likely to respond to incoming emails without an indicator of importance in the subject line than those with an indicator of importance due to the curiosity to discover how many points the message is worth.

H4b - similarly, participants will respond more quickly to incoming emails without an indicator of importance.

As we have already noted, these individual factors do not operate in isolation. Understanding the way that these factors interact with one another is therefore critical if we are to understand both the relative importance of each factor and the way that factors interact to amplify or diminish one another. We therefore pay close attention to the interactions in our analyses.

Our field experiment was designed to assess the relative influence of inbox-level cues that can be used by a sender in the subject line to communicate the importance, urgency, and cost of responding to their email. To do this, we needed an objective and simple way to operationalize each of these variables for the purposes of conducting an experiment. Sappleton and Laurenço (2016) suggested using incentives alongside blank subject lines to investigate response rates. We arrived at an approach in which participants received points for sending on-time responses to our emails. The benefit of using a points-based scheme is that it can be used to explicitly define, and then systematically vary, the relative importance of responding to different types of email. The purpose of our method was not to create a game per se, but simply to use points to indicate importance to receivers. We therefore did not include any other gamification elements. There is strong precedent for using point-based rewards in experimental HCI and psychology research to operationalize the manipulation of value (Farmer et al., 2018; Gould et al., 2016; Schumacher et al., 1999). These studies have shown that points can be explicitly communicated to participants, and that participants adapt their behavior to maximize these rewards. Using this approach, responding promptly to emails that had high importance (as defined by the sender/experimenter) earned participants more points, whereas responding promptly to emails that were of low importance earned fewer points. Using the subject line of the email we were then also able to manipulate the salience of importance by either explicitly stating how many points were on offer, or by leaving this information unspecified, and thereby creating an information gap (Wainer et al., 2011). The subject line of the email was also used to communicate how quickly a response was required in order to earn these points (urgency), whereas the effort required to make give a response (cost) was only evident in the email body.

3.1. Method

Participants

Forty-five participants (31 male) with a mean age of 28 (SD = 5.81) were recruited via an online recruitment advertisement. Thirty of the participants were in full-time employment, nine were part-time workers, and six were full-time students. All participants were self-assessed ‘high’ users of email (60% receive more than 25 emails a day), and stated they check their inbox at least once every few hours.

Participants were motivated to take part in the study by the chance to receive one of three rewards. Two £50 rewards were allocated based on the highest number of points obtained through responding to the emails in the study. The third £50 reward was allocated at random to one participant who completed the study, to avoid drop-outs and maintain engagement.

Design

A 3×3×2×2 entirely within-subjects design was used, with the variables of: Urgency (20 minutes, 3 hours, and 24 hours), cost (low and high), importance (10 points, 30 points, and 100 points), and the salience of importance (low and high). The dependent variables were the number of on-time responses (response rate) and the response time.

Materials

We sent participants email, and they earned points for giving on-time responses to these messages. Emails contained information in the subject line as well as in the body of the message (Figures 1 and 2). The information included in the email subject line varied in three ways: urgency, importance, and the salience of importance.

Figure 1. Example subject lines An image showing email subject lines in an email client.

Figure 2. Example email sent to a participant by the system An image of an email sent to a participant as viewed through a plain-text email client

Urgency.* Karagiannis and Vojnovic (2009) demonstrated that the majority of emails are responded to within 24 hours. The median response time is one hour. And 20% of emails are responded to within 5 minutes. In our experiment, three levels of urgency were specified in the subject lines, indicating how quickly the sender required a response. The highest level showed three crosses in the subject line “[+++]”, which meant participants had 20 minutes to reply to the email in order to receive the points assigned to that message. The medium level showed two crosses “[++]”, indicating a 3-hour window to reply, and the low level showed one cross “[+]”, indicating a 24-hour window to reply. An on-time response meant that a participant replied to the email within the time window specified by the level of urgency and thereby received the points indicated in the subject line.

Cost of responding.* The cost of responding was indicated in the body of the email rather than the subject line. Unless specified, it is generally hard for users to know the cost of responding until a message is opened. In the low-cost condition, responding to an email involved the participant opening the email and copying and pasting a unique random code from the email body (Figure 2) into the subject line of a reply email. In the high-cost condition, participants were required to click on a link contained within the message (Figure 3). This took them to a website where they were required to rate the emotional content of a series of text messages (Figures 4 and 5). This task lasted two minutes. Only when this task was completed were participants provided with a unique random code, which then had to be copied and pasted into the subject line of a response email. Participants again earned the points on offer if this response email was received on-time.

Figure 3. Example of high-cost email An example of an email from the high-cost condition as viewed through an email client.

Figure 4. Screenshot of instructions for the high-cost emails A screenshot of the instructions that participants were given once they clicked the link in a high-cost messages.

Figure 5. Screenshot of the rating task given to participants when in the high-cost condition. (Question asks whether the recipient is traveling by car or by train.) A screenshot of the task interface participants were given after they had read the high-cost condition instructions.

Importance. To manipulate importance, each email in our study was assigned one of three levels of points that would be earned by an on-time response. The highest level meant that an on-time response was worth 100 points, the medium level was worth 30 points, and the lowest level was worth 10 points. The use of points was to operationalize the importance of responding to an email. To earn the points on offer, the response email had to be received within the specified time window (20 minutes, 3 hours or 24 hours, depending on the level of urgency). In our experiment, points also served as a way to keep participants engaged throughout the three-week study as two of the £50 rewards were given to participants who earned the most points during the experiment.

The salience of importance. There were two levels of the salience of importance. When the salience of importance was high, the importance of an email was included in the subject line (see the second item in Figure 1) and the number of points for an on-time response was shown. When the salience of importance was low, the importance of an email did not appear in the subject line (see the first item in Figure 1). For both levels, the number of points a participant could collect by responding always appeared in the main body of the message (Figure 3).

Procedure

Each participant was sent demographic questionnaires to complete on the first day of the study. This was done to gauge their inbox size and their email management style. Questions included how many emails they had received that day and how many they had sent, along with a Likert scale to score how representative their answers were of their normal daily email workload.

Participants received emails in their existing main email account, every day between 9am and 9pm for three weeks, excluding weekends, from a consistent sender. There were 36 types of emails due to the 3 (urgency) × 3 (importance) × 2 (the salience of importance) × 2 (cost of responding) design of the experiment. All participants received 10 emails of each type, meaning 360 emails in total, over the 15-day study. This averaged out to 24 emails per day, with 28 emails being the highest number sent on any one-day and 20 the lowest. Participants responded to our emails using whichever strategies and devices they usually used for handling their email.

3.2. Results

From the initial survey about their usual email behavior, 66.7% of participants reported receiving on average between 10 and 49 emails per day, with 26.6% receiving more than 50 emails a day, and 6.7% reporting less than 10 emails a day. Participants reported that they often replied from their laptop (60.5%) or their smartphone (48.9%), followed by desktop PC (35%) and tablets (24.2%). We also asked participants to indicate which factors they considered when deciding to read an email: 57.8% stated that the sender was very important, and subject, date-received, and flagged-as-important were considered important by 61.4%, 44.2%, and 52.3% of participants, respectively.

Participation in the Study

We sent 16,200 emails to 45 participants, and participants responded to 65% of these emails (10,551 of 16,200). The total number of points available per participant was 16,800. The mean number of points participants earned was 14,073 (SD = 1,465). The top-10 scoring participants were all within 1,590 points of the leader, who scored 16,730.

To gain a better understanding of how each individual participant was engaging with the study, we considered participants’ response rate over the duration of the study. These data are shown in Figure 6 (range: 0-100%). It can be seen in the figure that while many participants engaged with the study and responded to the majority of emails that were sent to them, some did not. On closer inspection, we found that 16 participants failed to respond to at least 50% of the emails that were sent to them. Moreover, these same participants also failed to respond to at least one email from each of the experimental conditions. This latter point is particularly problematic as it results in missing cells for the statistical analysis of data.

Excluded participants responded to far fewer emails (M = 28%) than those that were included in the main analysis (M = 85%). Here we quantify the level of non-responsiveness of these excluded participants. The experimental design has 36 cells (3×3×2×2 entirely within-subjects) and data from 16 participants was excluded – this makes 576 cells in total. Of these 576 cells, 205 (36%) had zero responses to the emails that were sent. One participant did not respond to any emails at all, and four participants had zero data in half of all cells in the experimental design. In other words, the scale of non-responsiveness amongst excluded participants was extremely high, making it impractical to exclude cases pairwise or impute data. Hence, we chose to exclude these 16 participants from all subsequent analysis of data. We return to this point in the discussion.

Having excluded 16 (of 45) participants, it is possible that the generalisability of the results might be affected if, for example, the participants who were excluded show entirely different patterns of response behavior. To allay this concern, at the end of the results section we report a descriptive analysis of email responses from participants that were excluded. The analysis necessarily focuses on reporting mean performance values; it is not possible to conduct a thorough statistical analysis due to the number of missing data points from these participants.

Figure 6. Histogram showing the distribution of participants’ mean email response rate across all conditions. A histogram showing the rate of participants responses to emails across all conditions.

Of the 29 remaining participants, the mean response rate was 85% (SD = 10%, range: 60-99%). The mean of participants’ average response times (i.e., the mean of means) was 61 minutes (SD = 36 min). Our fastest participant took, on average, 18 minutes to respond to a message. Our slowest participant took an average of 3hrs 1min to respond. Of the 8,777 responses we received, only 228 (2.6%) were made within 60 seconds. It is therefore evident that emails were responded to during both work and non-work time throughout the period over which they were sent (9am to 9pm, Monday to Friday, for a period of three weeks). For these participants we investigate the effects of condition on the response rate and response time to emails.

Response Rate

We used the R statistical programming environment to perform a repeated measures ANOVA with a significance level of .05 to compare the main effects of urgency, cost, importance and salience of importance on number of responses and to understand the interaction of these main effects. We provide a summary of this analysis in Table 1.

Table 1. Summary of ANOVA results for response rate and response time to emails. Effect sizes are reported for only significant results. Significant interactions are presented in Figures 7–11.

Response rate Response time
Source F ηp2 F ηp2
Urgency (U) 57.10b *** 0.67 45.80b *** 0.62
Cost (C) 21.41a*** 0.43 37.56*** 0.57
Importance (I) 10.79b *** 0.28 5.27b ** 0.16
Salience (S) 2.55a 0.02a
U × C 4.22b * 0.13 29.61b *** 0.51
U × I 0.24c 3.12c * 0.1
U × S 3.22b * 0.1 0.44b
C × I 6.11b ** 0.18 0.02b
C × S 0.08a 0.27a
I × S 1.10b 0.30b
U × C × I 0.87c 0.08c
U × C × S 0.76b 0.13b
U × I × S 0.92c 1.18c
C × I × S 0.13b 0.30b
U × C × I × S 0.09c 0.12c

adf = 1, 28, bdf = 2, 56, cdf = 4, 112
*p < .05, **p < .01, ***p < .001

First, we consider the effect of response urgency (i.e., how long participants had to respond to an email in order to gain points from responding to it). As expected (H1a), participants were significantly more likely to miss the opportunity to respond on time to incoming emails that require fast responses than those where a less urgent response is required (20 minute: M = 74%, SD = 22%; 3 hours: M = 88%, SD = 17%; 24 hours: M = 93%, SD = 14%), F(2, 56) = 57.10, p < .001, ηp2 = .67. This presumably reflects the fact that once the response window to earn points from an email has been missed, there is simply no point in responding to it at all.

Second, we consider the effect of the cost (H2a) of response (i.e., the amount of effort and time that is required to respond to each email). We found that participants were more likely to respond to incoming emails that require a low cost response(M = 91%, SD = 14%) than to emails that had a high response cost, (M = 79%, SD = 22%) F(1, 28) = 21.41, p < .001, ηp2 = .43.

Third, we consider the effect of the importance (H3a; i.e., the number of points that participants earned) for responding to an email on-time on response rate. As expected, participants were generally more likely to respond to emails that are marked as important than those marked as low importance. Reflecting this, there was a significant main effect of importance on response rate, F(2, 56) = 10.79, p < .001, ηp2 = .28.

Fourth, we consider the effect of the salience (H4a) of importance on response rate (i.e., whether inbox-level cues revealed the importance of the email). Results showed that there was no significant main effect of the salience of importance on response rate. The hypothesis was therefore not supported.

We now turn our attention to the interactions between the variables. A significant urgency × cost interaction was found, F(2, 56) = 4.22, p < .05, ηp2 = .13. This interaction is shown in Figure 7. To investigate this more thoroughly we conducted tests of the simple main effect of response cost across each of the different levels of urgency (applying Bonferroni corrections). It was found that the effect of response cost was robust across the manipulation of urgency: Participants were significantly more likely to respond to emails that had a low-cost than a high-cost, and this effect occurred when the response window was either 24-hours, F(1, 28) = 13.91, p < .001, ηp2 = .33, 3-hours, F(1, 28) = 19.46, p < .001, ηp2 = .41, or 20-minutes, F(1, 28) = 20.29, p < .001, ηp2 = .42. In other words, the effect of cost of responding is robust across the manipulation of urgency.

Figure 7. Urgency by cost (U × C) interaction on response rate. Errors bars represent standard errors of the mean. There was a significant effect of cost on response rates, and this effect occurred at all levels of urgency: 24-hours, 3-hours, and 20-minutes. A graph of the interaction between the cost of an email and the response rate to messages.

There was also a significant importance × cost interaction, F(2, 56) = 6.11, p < .01, ηp2 = .18. This interaction can be seen in Figure 8. It shows that the effect of importance on response rates is moderated by the cost of responding. To investigate this interaction, we report the results of a simple main effects test, with Bonferroni corrections. Results show a significant simple effect of importance on response rates to emails that had a high response cost, F(2, 27) = 8.31, p < .01, ηp2 = .38; participants were more likely to respond to high-cost emails that were worth more points. In contrast, there was no such simple effect of importance on response rate to emails that had a low response cost, F(2, 27) = 3.21, p = .056, ηp2 = .19; participants tended to respond to most emails that were easy to respond to regardless of how many points were on offer.

Figure 8. Cost by importance (C × I) interaction on response rate. Errors bars represent standard errors of the mean. There was a significant effect of importance on response rates in the low-cost condition; the effect of importance was not significant in the high high-cost condition. A graph of the interaction of cost of an email and the rate of interaction with the message.

There was a significant urgency × salience interaction, F(2, 56) = 3.22, p < .05, ηp2 = .10. This interaction can be seen in Figure 9. A simple main effects analysis of this interaction with Bonferroni corrections showed that when participants had a brief 20-minute window to respond to an email, then responses were more likely when there was of low-salience (i.e. no inbox-level cues indicating an email’s importance), compared to when this information was of high-salience (i.e., clear inbox-level cues were present), F(1, 28) = 4.88, p < .05, ηp2 = .15. However, when participants had longer to respond (3- or 24-hours), there was no effect of the salience of importance on response rate, all F’s < 1.

All other interactions were not significant.

Figure 9. Urgency by salience (U × S) interaction on response rate. Errors bars represent standard errors of the mean. There was a significant effect of salience on response rates in the 20-minute condition; the effect of salience was not significant in either the 24-hour or 3-hour condition. A graph of the interaction between the urgency of an email and its inbox-level salience.

Response Time

Having analyzed the number of emails participants responded to, we next consider how quickly participants responded to messages. We used the R statistical programming environment to perform a repeated measures ANOVA with a significance level of .05 to compare the effects of urgency, cost, importance and salience of importance on response time. Table 1 provides a summary of this analysis.

First, in line with our prediction (H1b) we found that the urgency of an email affected how quickly participants responded to it. Responses to emails were significantly faster when there was a shorter response window (20 minute: M = 20 min, SD = 42 min; 3 hours: M = 38 min, SD = 38 min; 24 hours: M = 125 min, SD = 133 min), F(2, 56) = 45.80, p < .001, ηp2 = .62.

Second, we found that participants were faster at responding to emails that had a low response cost (H2b; M = 37 min, SD = 56 min) than to emails that had a high response cost (M = 85 min, SD = 118 min), F(1, 28) = 37.56, p < .001, ηp2 = .57.

Third, participants were faster at responding to emails that had greater importance (H3b). This was operationalized in the study by varying the number of points that were earned for responding to the email within the required time frame. We found that participants were significantly faster at responding to emails that were worth more points (100-points: M = 57 min, SD = 88 min; 30-points: M = 60 min, SD = 93 min; 10-points: M = 66 min, SD = 104 min), F(2, 56) = 5.27, p < .01, ηp2 = .16.

Fourth, there was no effect of the salience (H4b) of importance on performance. Response times were similar regardless of whether the importance was missing (M = 61 min, SD = 97 min) or visible in the subject line of the email (M = 61 min, SD = 94 min), F < 1.

We now turn our attention to the interactions between this variables. It can be seen in Figure 10 that the effect of response cost was moderated by the urgency of response required. Indeed, statistical analysis found a significant urgency × cost interaction, F(2, 56) = 29.61, p < .001, ηp2 = .51. A simple main effects analysis of this interaction with Bonferroni corrections show that when participants had at least 3 hours to respond to an email, there was an effect of response cost. That is, participants were faster at responding to emails that had a low-cost response than those that were high-cost in the 3 hour condition, F(1, 28) = 26.31, p < .001, ηp2 = .48, and the 24 hour condition, F(1, 28) = 37.40, p < .001, ηp2 = .57. However, when participants had a relatively brief 20-minute response window they were equally quick to respond to both the high- and low-cost messages. In other words, there is no significant simple effect of response cost when a time sensitive response was required.

Figure 10. Urgency by cost (U × C) interaction on time to reply to an email. Errors bars represent standard errors of the mean. There was a significant effect of cost on response rates in both the 24-hour and 3-hour condition; the effect of cost was not significant 20-minute condition. A graph of the interaction between the cost of an email and the time it took for participants to reply.

It can be seen in Figure 11 that the effect of importance was moderated by the urgency of response required. Statistical analysis found a significant urgency × importance interaction, F(4, 112) = 3.12, p < . 05, ηp2 = .10. A simple main effects analysis of this interaction with Bonferroni corrections show that when emails were less urgent and could be deferred for up to 24-hours, there was a significant effect of importance on response times, F(2, 27) = 3.98, p < . 05, ηp2 = .29. As can be seen in Figure 11, participants were strategic and responded faster to emails that gave more points. In contrast, when there was a relatively short response window (20 minutes or 3 hours), the number of points earned from responding to an email had no effect on response times.

Figure 11. Urgency by importance (U × I) interaction on time to reply to an email. Errors bars represent standard errors of the mean. There was a significant effect of importance on response rates in the 24-hour condition; the effect of importance was not significant in either the 3-hour or the 20-minute condition.