“Okay, whatever”: An Evaluation of Cookie Consent Interfaces

Hana Habib, Carnegie Mellon University, United States, htq@cs.cmu.edu
Megan Li, Harvey Mudd College, United States, mlli@hmc.edu
Ellie Young, New College of Florida, United States, eleanor.young18@ncf.edu
Lorrie Cranor, Carnegie Mellon University, United States, lorrie@cmu.edu

Many websites have added cookie consent interfaces to meet regulatory consent requirements. While prior work has demonstrated that they often use dark patterns — design techniques that lead users to less privacy-protective options — other usability aspects of these interfaces have been less explored. This study contributes a comprehensive, two-stage usability assessment of cookie consent interfaces. We first inspected 191 consent interfaces against five dark pattern heuristics and identified design choices that may impact usability. We then conducted a 1,109-participant online between-subjects experiment exploring the usability impact of seven design parameters. Participants were exposed to one of 12 consent interface variants during a shopping task on a prototype e-commerce website and answered a survey about their experience. Our findings suggest that a fully-blocking consent interface with in-line cookie options accompanied by a persistent button enabling users to later change their consent decision best meets several design objectives.

CCS Concepts:Security and privacy → Usability in security and privacy; • Social and professional topics → Computing / technology policy; • Human-centered computing~Empirical studies in HCI;

Keywords: cookie consent, GDPR, privacy choice, privacy notice

ACM Reference Format:
Hana Habib, Megan Li, Ellie Young, and Lorrie Cranor. 2022. “Okay, whatever”: An Evaluation of Cookie Consent Interfaces. In CHI Conference on Human Factors in Computing Systems (CHI '22), April 29-May 5, 2022, New Orleans, LA, USA. ACM, New York, NY, USA 27 Pages. https://doi.org/10.1145/3491102.3501985

1 INTRODUCTION

When consumers visit a website for the first time, their experience is often interrupted with an interface related to the use of cookies. These interfaces, used to meet legal requirements for notice and consent to data collection and processing under the EU's ePrivacy Directive (EPD) and General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) [7, 13, 32], contribute to the considerable user burden involved in managing digital privacy. While existing privacy regulation stipulates that cookie consent interfaces must be usable, there are no existing standards for usable consent interfaces. As a result, organizations use a wide range of design practices in their implementations, some of which have been highlighted in prior work as dark patterns [10, 29]. Dark patterns within cookie consent interfaces are design practices that nudge users toward less privacy-protective options and could lead to users unknowingly consenting to data collection or failing to exercise their preferred privacy choices. Beyond dark patterns, it is important to consider other usability aspects of cookie consent interfaces, such as user awareness and comprehension of choices, as interfaces with poor usability could cause privacy fatigue in users, described as “the tendency of consumers to disclose greater information over time when using more complex and less-usable privacy controls” [17].

We conducted a comprehensive two-stage evaluation of cookie consent interfaces to more holistically assess the usability impact of common interface designs beyond the presence of dark patterns. Our study contributes a comprehensive definition of usability for the context of consent interfaces and provides a better understanding of design choices that improve the consent experience. Our results can be leveraged by websites, self-regulatory organizations, and Consent Management Platforms (CMPs) to improve consent interfaces, as well as by legislators and regulators who define and enforce usable consent interface requirements.

We based our evaluation of cookie consent interfaces on those implemented through CMPs. These services have emerged to help organizations manage consent flows on their websites and apps. Many CMPs rely on the Transparency and Consent Framework (TCF) developed by IAB Europe, a self-regulatory group for the advertising industry, to ensure that the design choices available to their clients for their consent interface implementations are in compliance with regulatory requirements and follow industry standards [14]. According to a report by the ad-tech company Kevel, approximately 52% of the top 10,000 US websites that serve ads have a CMP-implemented cookie consent interface, with six CMPs found frequently on IAB member websites in the third quarter of 2021 and a single CMP capturing the majority of the market share [18]. While some design aspects of the consent interface are standardized by the TCF for each CMP, there are others that organizations can choose to customize for their particular website or app. Considering the prevalence of CMPs and consolidation of the space into a handful of services, improvements in the usability of CMP-implemented cookie consent interfaces would have widespread impact.

Our research builds on prior research in this space, which primarily identified dark patterns within cookie consent interfaces and explored their impact on user decision-making [10, 29, 39, 42]. We expand on this work by investigating additional design parameters for consent interfaces and more deeply exploring users’ perceptions of different consent interface designs. First, we conducted an inspection-based evaluation of 191 cookie consent interfaces implemented through five major CMP services, using an approach informed by three standard HCI methods: heuristic evaluation, cognitive walkthrough, and independent expert review [45]. We evaluated each interface for several dark pattern heuristics identified in prior work [24, 29, 39] as well as other potential usability barriers. Our inspection-based evaluation yielded a list of design parameters that appear to be customizable through CMPs. In the second stage of our evaluation, we drew on our findings from our inspection-based evaluation to further investigate seven design parameters (listed in section 4.1) that may impact the usability of consent interfaces. We conducted a between-subjects online experiment with 1,109 participants to evaluate the usability of 12 consent interface design variants. Participants in the experiment were asked to complete a shopping task on a prototype of a fictitious retail website where they encountered one of the consent interface design variants. Following task completion, participants answered survey questions related to the usability of the consent interface.

Our analysis of participants’ survey responses and interactions with the prototype website highlighted significant usability differences between the design variants tested. We found that prominence of the consent interface on the website impacted participants’ awareness of available choices and their ability to make their preferred consent decision. Our results also indicate that the absence of in-line cookie options within the initial screen of the interface appeared to have led participants to consent to all cookies and reduced participants’ investment in their consent decision. These results highlight design choices that organizations could make, or that could be incorporated into a standardization effort, that would improve the usability of cookie consent interfaces.

2 Background & Related Work

In this section we provide an overview of the regulatory requirements that are applicable to cookie consent interfaces. We then describe prior work that explored the prevalence and impact of dark patterns within such interfaces. Next, we identify seven components of usability that pertain to cookie consent interfaces. Last, we describe alternative mechanisms for capturing and enforcing users’ consent decisions.

2.1 Regulatory Requirements

Cookie consent interfaces started appearing on websites after a 2009 amendment to the EU's ePrivacy Directive (EPD) which aimed to address issues related to the confidentiality of digital communication and tracking on the internet [13]. Consent interfaces became more widespread after the European Union's General Data Protection Regulation (GDPR), a comprehensive privacy legislation having global impact, went into effect in May 2018 [5]. Under GDPR, organizations must request consumers’ consent to process personal data beyond what is required for fulfilling a legitimate business interest. Under Article 4, consumers must provide their consent through a clear and affirmative action [13]. Agreement cannot be incorporated into default settings or pre-checked boxes [30]. Furthermore, Article 12 specifies that consent must be requested through an easily accessible form with simple and clear language. Moreover, Article 7 stipulates that consumers must be able to withdraw their consent at any time [13]. In the wake of its enactment, the GDPR has inspired several other national privacy laws, including those in Canada, Japan, South Korea, Colombia, Argentina, and South Africa [38].

The GDPR also laid the groundwork for the California Consumer Privacy Act (CCPA), which went into effect in 2020. The California state law requires certain companies to provide notice to consumers related to data collection. Among other privacy rights, it grants California residents the right to opt out of having their personal data sold to third parties, for example, for marketing purposes [32]. The California Privacy Rights and Enforcement Act (CPRA), which will go into effect in 2023, builds upon the CCPA. The law provides additional privacy rights to California consumers, including a right to opt out of a business using sensitive personal information and to opt out of the sharing of information with third parties (in addition to selling). Furthermore, the CPRA explicitly prohibits the use of dark design patterns in consent interfaces [31].

While it may be too soon to evaluate organizations’ response to CCPA requirements, there is evidence of mixed compliance with the GDPR's consent requirements. A measurement study by Degeling et al. found a 16% increase in the display of cookie consent interfaces among the 6,579 websites evaluated after the GDPR went into effect in May 2018 [5]. However, past research has identified several issues with how these interfaces are implemented. In analysis of nearly 30k websites, Bollinger et al. found that 36.4% incorrectly categorized at least one cookie and 21.3% placed cookies when the user explicitly opted out [2]. Furthermore, Matte et al. observed that even websites with CMP-implemented consent interfaces following the IAB's Transparency and Consent Framework could mishandle users’ consent decisions [25].

2.2 Dark Patterns in Consent Interfaces

Several studies evaluating consent interfaces for dark patterns have found that current implementations may also be violating the intent of the GDPR and other regulation. Utz et al. conducted a field study exploring the impact of four design variables, finding that position of the interface, choices offered, nudging patterns, and language used in the interface text impact users’ interactions with the interface [42]. Soe et al. manually evaluated cookie consent notices on 300 online news outlets based on 13 heuristics, providing a reference for several types of common dark patterns specific to consent notices [39]. Drawing from existing literature in design, law, and privacy, Gray et al. performed an interaction criticism of consent banners from four perspectives: the designer's intent, designed UI, end-user, and potential societal impact. By reviewing recordings from over 50 websites, they identified different stages of the consent task flow and common design choices that raised ethical dilemmas warranting additional dialogue [10]. Nouwens et al. quantified the impact of consent interface design choices through an online experiment, finding that the display of granular options within an initial cookie consent prompt decreased the probability of a user consenting to all cookies, while removing a “reject all” button increased the probability of consent [29].

The impact of other types of dark patterns have not yet been empirically explored in prior work. One such pattern has been termed “confirmshaming’ which is wording that guilts or shames users about a decision presented to them [3]. Implementations of confirmshaming could exploit a cognitive bias called loss aversion. This well-studied bias explains how people prefer to avoid a loss compared to gaining something equivalent [37]. In the context of cookie consent, loss aversion framed text could highlight negative outcomes of not accepting optional cookies, which may create a nudging effect towards accepting all cookies and impact other aspects of usability.

A small number of other studies have focused on usability aspects of interfaces related to cookies that extend beyond dark patterns. Kulyk et al. found that the text of cookie disclaimers had little impact on participants’ sentiment and decision to continue using the website [20]. Additionally, Machuletz and Böhme explored the impact of the number of consent options present on the effort required to use the interface and participants’ sentiments, finding minimal difference between one and three options [23].

This study complements the growing body of research related to cookie consent interfaces. Our inspection-based evaluation builds on Soe et al.’s [39] review of consent notices on news websites by applying a similar set of dark pattern heuristics to cookie consent notices implemented through CMPs. Complementing prior user studies [20, 23, 29, 42], our online experiment explores the impact of additional design parameters, specifically the formatting of interface text, presence of loss aversion, text used in button options, layout of cookie options presented beyond the initial screen of the consent interface, and process for changing consent decisions. More significantly, our study evaluates the impact of design parameters against a comprehensive definition of usability that includes aspects that have not been deeply explored in prior work in the context of consent interfaces.

2.3 Defining Usability for Consent Interfaces

A limitation of prior work related to the consent interface design is that usability is primarily framed through the lens of dark patterns. However, the privacy and usable design literature defines usability through a variety of other aspects related to how people interact with and perceive an interface. Here we provide an overview of different definitions of usability, primarily selected from textbooks in HCI and privacy. We observed that components of these definitions could be organized into seven groups, providing a structure for our usability assessment of consent interfaces.

Feng et al. propose a definition of usability specific to usable privacy choice interactions and describe the concept of meaningful privacy choices that “extend beyond traditional usability considerations to include several facets that are more specifically tied to supporting users in making privacy decisions that capture their true privacy preferences” [8]. Schaub and Cranor emphasize that meeting regulatory requirements is not enough and describe four components they consider required for effective privacy interfaces: findability, understandability, usability, and usefulness [35]. Other usability definitions are applicable to interfaces beyond those related to consent. The International Organization for Standardization (ISO) 9241 considers usability as the effectiveness, efficiency, and satisfaction with which users achieve specified goals in particular environments [16]. In her definition, Quesenbery describes the “5 Es” of a usable interface by extending ISO's definition of usability to include error tolerance and ease of learning [33]. Nielsen defines usability through five “quality components” that assess how easy interfaces are to use [28]. Morville's UX Honeycomb is commonly referred to in web design and explains six qualities of the user experience that must be addressed [27].

Table 1 highlights the overlap between these six usability definitions, as well as where they differ. In grouping the components of these definitions we identified seven aspects pertaining to the usability of consent interfaces. We define these usability aspects as:

  1. User needs: whether a privacy choice interface addresses the intended users’ privacy needs in a particular privacy choice context. Also includes accuracy and completeness of the interface in addressing these needs.
  2. User ability & effort: whether a privacy choice interface allows the intended users to accomplish a particular privacy goal and with minimal effort.
  3. User awareness: whether the intended users are aware that a particular privacy choice exists within a privacy choice interface, and if they are able to find it.
  4. User comprehension: whether the intended users understand what a particular privacy choice does and the implications of their decisions.
  5. User sentiment: whether the intended users are satisfied with a privacy choice interface and options it provides. This includes whether users have faith that the privacy choice will be honored.
  6. Decision reversal: whether a privacy choice interface allows the intended users to correct an error or change their decision. This also includes the effort required to do so.
  7. Nudging patterns: whether the design of a privacy choice interface leads the intended users to select certain choices in the interface over others (including dark patterns that lead users to less privacy-protective options).
Table 1: Components of six usability definitions grouped according to different usability aspects.
Related to user needs Related to ability & effort Related to awareness Related to comprehension Related to sentiment Related to decision reversal Related to nudging patterns
Feng et al. [8] Effectiveness Efficiency User awareness Comprehensiveness Neutrality
Schaub & Cranor [35] Usefulness Usablity Findability Understandability
ISO [16] Effectiveness Efficiency Satisfaction
Quesenbery [33] Effectiveness Efficiency Easy to learn (initial use) Easy to learn (continued used) Engaging Error tolerant
Nielsen [28] Efficiency Satisfaction Error tolerant
Morville UX Honeycomb [27] Useful Usable, Accessible Findable Desirable, Credible

2.4 Automated Consent Mechanisms

As a reaction to the ubiquity of consent interfaces on websites and their typical poor usability, automated consent mechanisms have been proposed to ease the user burden involved in making consent decisions. For example, browser extensions are now available that block consent interfaces from appearing on websites and automatically respond to interface prompts on behalf of the user (e.g., [19, 26, 29]). A proposed scheme by the UK Information Commissioner's Office (ICO) would allow users to set their cookie preferences in their web browsers and have those preferences be automatically communicated to the websites they visit [40], similar to the Do Not Track (DNT) standard [44]. Client-side consent mechanisms that directly manage cookies, rather than relying on websites’ handling of consent decisions, can mitigate issues related to how websites handle consent decisions. For example, the CookieBlock browser extension categorizes cookies and automatically removes those that the user has not consented to [2].

While automated consent mechanisms have the potential to address both the user burden as well as implementation issues associated with consent interfaces, we expect that consent interfaces requiring user interaction will continue to proliferate for some time to come. Furthermore, as there is evidence to support that users’ decisions related to cookies are largely impacted by the perceived trustworthiness of a website [20], there may be people who would want to allow certain categories of cookies on some websites they visit but not on others. While automated mechanisms could allow for custom rules, we expect some users may fall back to using the website's consent interface manually when they want to make an exception to their rules. This study contributes a better understanding of the design choices that impact the usability of consent interfaces, which may continue to persist even as automated consent mechanisms become more widespread.

3 INSPECTION-BASED EVALUATION OF COOKIE CONSENT INTERFACES

We used the seven usability aspects identified in section 2.3 to guide our evaluation of design choices in CMP-implemented cookie consent interfaces. As an initial step in our evaluation, we conducted an inspection-based evaluation of such interfaces across a wide range of websites. We developed a standardized procedure for our evaluation, informed by independent expert review, cognitive walkthrough, and heuristic evaluation approaches [45]. Utilizing the results of this expert review, we identified design parameters for consent interfaces that seem to be customizable through CMPs and may have an impact on usability.

3.1 Inspection Procedure

To conduct our inspection-based evaluation of CMP-implemented interfaces, we first identified five services that are in widespread use through a review of prior work in this space [12, 29]: Cookiebot, Crownpeak, OneTrust, QuantCast, and TrustArc. We compiled a diverse set of websites that we believed would have consent interfaces that are implemented through these CMPs. As a starting point, we used a subset of 603 websites that were found to contain interfaces implemented by these five CMPs from Nouwens et al.’s dataset of popular UK websites [29]. However, as the majority of websites in this dataset contained a consent interface implemented through either Quantcast or OneTrust, we aimed to diversify our set of websites by identifying additional websites utilizing less popular CMPs and with those based outside of the UK. We developed a web scraper using webXray, a tool for analyzing webpage traffic [22], which looked for domain requests to any of the five CMPs. We ran our scraper on 1,000 websites evenly sampled across Tranco's list of top 10,000 global websites [21] (as of June 21, 2021). This yielded an additional 207 unique websites for our dataset of websites potentially containing an interface implemented by one of the five CMP providers. The distribution of CMPs in our final dataset of 810 websites was 369 (46%) Quantcast, 202 (25%) OneTrust, 87 (11%) CookieBot, 78 (10%) TrustArc, and 74 (9.1%) CrownPeak.

 We evaluated 191 consent interfaces on websites drawn from our dataset of 810 websites potentially containing CMP-implemented interfaces.1 We evaluated at least ten interfaces implemented through each CMP and attempted to identify distinct interface designs within the group of websites using each service, particularly which cookie options were provided, where and how they were presented to users, and the content of the interface text. We continued our review until we observed that consent interfaces on additional websites were very similar or identical to ones already reviewed (i.e., we had reached saturation in the data). Thus our sample includes a wide variety of interfaces but is not representative of the frequency with which each type of interface appears.

Two members of the research team jointly examined 15 websites to establish the evaluation protocol. The remaining websites in our sample were inspected by one of these researchers. For each website, the researcher visited the desktop version of the website from a computer with a US-based IP address. They visited each website in private browsing mode, which mitigated the impact of existing browser cookies and simplified the inspection procedure compared to creating new browser profiles or virtual machines for each website. The researcher first assessed how difficult it was to see the consent interface and noted if it changed or was dismissed when clicking or scrolling on the website. They then evaluated the interface based on a set of dark pattern heuristics identified in prior work, including design patterns that may lead to unintentional data disclosure or be considered illegal under the GDPR or CCPA [24, 29, 39]. To evaluate for these patterns, the researcher would attempt to select the most privacy-protective option available (as dark patterns tend to appear as roadblocks in this process). Specifically they identified:

  • Unequal paths: The interface had unequal interaction paths for the most and least privacy-protective options (see 1a ).
  • “Bad” defaults: The interface had default options that were not privacy-protective (see 1b ).
  • Confusing buttons: The interface had unintuitive placement of buttons for confirming users’ cookie preferences and allowing all cookies (see 1c ).
  • No choices: The interface did not provide choices related to the use of cookies (see 1d ).
  • Confirmshaming: The interface used “confirmshaming’ which is wording that guilts or shames users to influence their decision [3] (see 1e ).
Figure 1
Figure 1: Examples of cookie consent interfaces found during our inspection-based evaluation for each dark pattern heuristic.

In conjunction with evaluating the interface against these heuristics, the researcher conducted a cognitive walkthrough of the cookie consent interface. To guide their walkthrough, they approached the interface as a user who wants to select privacy-protective options but has minimal knowledge about digital privacy and cookies and little motivation to read the privacy information provided by the website. The researcher considered the visual appearance of the interface, the interface text, and available options, specifically:

  1. Are there visual aspects of the interface that may aid or be detrimental to using selecting privacy-protective options (i.e., options other than allowing all cookies)?
  2. What is the most privacy-protective option available in the interface?
  3. What is the process for selecting the most privacy-protective option (i.e., allowing the fewest cookies)?

Finally, based on their knowledge of usability and dark patterns, the researcher made any additional notes about aspects of the consent interface that may confuse users. The researcher's observations were recorded in a database, along with screenshots or screen recordings of the analyzed cookie consent interface.2

3.2 Inspection Evaluation Results

We reviewed the results of our data collection and found that dark pattern heuristics and other usability issues were prevalent in the CMP-implemented consent interfaces we evaluated. We then identified a list of design parameters that appear to be customizable through CMPs and may impact the usability of consent interfaces.

3.2.1 Summary of Dark Pattern Heuristics & Usability Barriers. As shown in Table 2, the vast majority of the consent interfaces (88.0%) reviewed exhibited a dark pattern heuristic, including some that may be in violation of GDPR requirements [7]. The most prevalent, observed on 150 (78.5%) websites, was having a simpler interaction path for less privacy-protective cookie options (i.e.,“accept all cookies”) than for more privacy-protective options. Forty-nine (25.4%) consent interfaces in our sample also had pre-selected or default options that were less protective of users’ privacy than other available options. This dark pattern heuristic occurred relatively more frequently in consent interfaces implemented through OneTrust or TrustArc, but as our sample is not representative, this may or may not reflect trends across all websites using these CMPs.

Some consent interfaces also exhibited usability barriers, beyond potential dark patterns, that were uncovered during a cognitive walkthrough of the interfaces. One example was a consent interface that contained an “Options” button on the cookie options page that did not appear to do anything but dismiss the consent interface.3 This would likely impact users’ comprehension of the interface, as well as sentiment towards the company. Another interface contained a “Confirm My Choices” button within the cookie options page but no choices were actually present on the page.4 The absence of choices on an interface where users would expect them to be present is highly likely to impact users’ ability to effectively make decisions related to cookies on the website. In contrast, some interfaces included options to enable or disable different categories of cookies on the initial screen of the interface,5 which prior work has shown to impact users’ engagement with consent options [29, 42].

Table 2: Counts of the dark pattern heuristics and other usability barriers identified during our inspection-based evaluation of consent interfaces implemented through five CMP services. (n = number of consent interfaces evaluated for a particular CMP).
CMP (n) Unequal paths “Bad” defaults Confusing buttons No choices Confirmshaming Other barriers None
OneTrust (70) 60 33 21 4 1 5 6
Quantcast (69) 55 0 0 7 1 1 7
CookieBot (20) 9 5 0 2 0 5 8
TrustArc (19) 14 9 0 2 1 0 2
CrownPeak (13) 12 2 1 2 0 1 0
Sum: (191) 150 49 22 17 3 12 23

3.2.2 Design Parameters for CMP-Implemented interfaces. We used our recorded observations to compare consent interfaces implemented through the same CMP, observing that interfaces primarily differed in how they were initially presented, the interactions and graphical elements involved in making a consent decision, and what information was provided about the cookies used on the website. When reviewing our notes about the evaluated consent interfaces, we highlighted any aspects of the interface that could be particularly beneficial or harmful to the usability of any privacy-protective options available and for which design choices could be enumerated. In doing so, we identified design parameters that we hypothesized would have an impact on the usability of the consent interface and that appear to be customizable through CMPs, as well as possible implementations of the parameter that we observed were being used in practice (listed in Table 3).

Table 3: List of design parameters that appear to be customizable through the five CMPs investigated, possible implementations for each (in order of the least to best option for usability based on our expert knowledge), and the corresponding usability objectives that we hypothesized could be impacted. The two parameters not explored in our user study are italicized.
Design Parameter Example Possible Implementations Usability Objective(s)
Prominence of the consent interface persistent “Cookie Preferences” button, non-blocking banner, consent wall User awareness
Path to a cookie options interface (where options to allow/deny cookies are presented) link embedded in text, equally weighted interface button, in-line options in initial screen User awareness
How/whether the notice described the presence of choices loss aversion text present, text mentions that options are available User awareness, User sentiment
Readability of the notice fonts, colors, contrasts, text layout (bulleted vs. paragraph) User comprehension
Text within button options generic (“Okay,” “Submit”), detailed (“Allow selected cookies,” “Allow all cookies”) User comprehension
Layout of cookie options page choices separated in multiple tabs, all choices on same page User ability & effort
Process for changing or revoking a consent decision none (clear browser cookies), link in cookie policy, persistent “Cookie Preferences” button Decision reversal
Placement of button options “Allow all” option shifts with user actions, “Allow all” remains in place User ability & effort
The granularity of choices offered cookie-level, category-level User ability & effort

4 USER STUDY EVALUATION OF CONSENT INTERFACE DESIGNS

To further investigate the usability impact of design choices that organizations can make when implementing their cookie consent interfaces, we conducted a large-scale online user study in which 1,109 participants were each randomly assigned to visit and interact with a fictitious e-commerce website implementing one of 12 cookie consent design variants. This online experiment builds on our inspection-based evaluation, as well as prior user studies of consent interfaces, by evaluating the impact of different design parameters in consent interfaces against a comprehensive definition of usability rather than primarily through the lens of dark patterns. We frame our findings around the other six usability aspects identified in section 2.3: user needs, user ability & effort, user awareness, user comprehension, user sentiment, and decision reversal.

4.1 User Study Design

We enumerated possible design choices for the design parameters we identified (listed in Table 3) based on the practices we observed in our inspection-based evaluation. As it was infeasible to study all of the possible design choices, we ranked the design parameters according to what we believed was likely to have the most impact on usability and prioritized design choices for which there has not yet been much research or established best practice in UX design. Along these criteria, we decided not to explore the placement of button options within the consent interface (for which there are established best practices [34]) and the granularity of choices offered (for which there has been prior research that shows that users may be overwhelmed by having too many choices [1]). Our study also did not explore accessibility issues, such as those related to color contrast and size of button components within the interface, which also have established guidelines [43]. The remaining seven design parameters corresponded to variables in our study.

Figure 2
Figure 2: Two consent interface design variants that demonstrate the design choices for each parameter explored in our user study.
Figure 3
Figure 3: The two styles of the“Cookie Preferences” linked through the cookie consent interface design variants explored in our study. The consent interface was dismissed upon clicking the “Save prefences” button.

We developed 12 design variants of cookie consent interfaces (provided in section A) based on the practices we observed in our inspection of CMP-implemented consent interfaces. Table 4 provides an overview of these twelve variants and their values for the seven study variables: prominence of the consent interface, path to a cookie options interface, presence of loss aversion text, layout of the interface text, specificity of the button text, layout of the cookie options page, and process for changing or revoking a consent decision. So that we could isolate the effect of each design choice, one design variant was composed of what we considered as “best practices”: what we hypothesized as the most privacy-protective or usable options for each study variable (see 2a ). Ten of the design variants manipulated just one study variable such that they differed from the “best practices” baseline in only one aspect of the interface design in a way that we believed to be less usable or less privacy-protective. Another design variant (prominence-cornerButton) was based off of a OneTrust-implemented consent interface design that presented visitors with consent options through a button that persisted on the bottom right corner of the webpage.6 A twelfth design variant explored the combination of design choices that we considered were the least privacy-protective or usable, which we refer to as “worst practices” (see 2b ). The design variants included a link to a single-layer “Cookie Preferences” page (shown in 3a ) or a multi-layer version of the page (3b ), which included information about cookies and four different cookie categories defined by the International Chamber of Commerce (ICC) UK (strictly necessary, performance, functional, and targeting) [15] as well as toggles to enable/disable the later three categories. Table 4 provides an overview of the design variants explored in our study and their values for the seven study variables.

Table 4: Overview of the 12 cookie consent interface design variants and their values for the seven design parameters explored in our online experiment. Design choices that differ from best-practices are bolded.
Condition Name Prominence Options Path Loss Aversion Text Layout Button Text Choices Layout Decision Reversal
best-practices fully-blocking in-line absent bulleted detailed single-page persistent button
prominence-cornerButton non-blocking button n/a n/a n/a n/a single-page persistent button
prominence-banner non-blocking banner in-line absent bulleted detailed single-page persistent button
options-embeddedLink fully-blocking embedded link absent bulleted detailed single-page persistent button
options-interfaceButton fully-blocking interface button absent bulleted detailed single-page persistent button
text-lossAversion fully-blocking in-line present bulleted detailed single-page persistent button
text-layoutParagraph fully-blocking in-line absent paragraph detailed single-page persistent button
button-generic fully-blocking in-line absent bulleted generic single-page persistent button
layout-multilayer fully-blocking interface button absent bulleted detailed multi-page persistent button
reversal-noInstructions fully-blocking in-line absent bulleted detailed single-page no instructions (button present)
reversal-cookiePolicy fully-blocking in-line absent bulleted detailed single-page cookie policy
worst-practices non-blocking banner embedded link present paragraph generic multi-page no instructions (cookie policy)

4.2 User Study Data Collection & Analysis

4.2.1 Experimental Protocol. We conducted an online experiment utilizing a between-subjects protocol. To explore the impact of the different design parameters in a realistic setting, we presented our consent notice designs in the context of a fictitious e-commerce website that sold cups, mugs, and other drinkware. We used Adobe XD to implement the parts of an e-commerce website relevant to the cookie consent experience or basic shopping functionality, including a cookie consent interface (varied per condition), privacy policy, cookie policy, product catalog, and product detail pages. We implemented the prototypes only in a desktop version of a website to maximize the chances of participants being able to read and interact with the consent notice. In order to capture participants’ interactions with the website as well as timing data, we utilized a usability testing platform called Useberry. After completing the study consent form and verifying their eligibility, participants in our study were assigned one of the study conditions at random and directed to Useberry. To prevent participants from overly fixating on the consent notice, participants were give a distraction task: add a product from the store catalog to their cart. Participants were instructed to interact with the prototype as they would a real website and perform whatever action they would take the first time they visited a real e-commerce website. After the initial instruction screens, participants were exposed to a cookie consent interface design according to their assigned condition. Once participants completed the study task, or indicated that they give up on the task through a button provided by Useberry, they were directed to a follow-up survey implemented on Qualtrics.

The survey (provided in section B) included questions for evaluating the different aspects of usability described in section 2.3. Participants first answered questions related to user awareness and unfocused comprehension based on their recall of the consent notice. After completing this portion of the survey, participants were provided an opportunity to refer back to the consent interface and prototype of the e-commerce website as they answered additional questions.

Our protocol was approved by Carnegie Mellon University's Institutional Review Board. While participants consented to their interactions with the prototype website being captured, we did not collect any personal information from participants.

4.2.2 Participant Recruitment. To prevent priming potential participants, we described the study as a study requesting feedback about an e-commerce website. Due to the language competencies of the research team and the consideration that cookie consent interfaces and users’ experiences with them may differ across legal jurisdictions, we only recruited US-based participants. Additionally, participants were required to be over 18 years old, fluent in English, and have access to a tablet or computer to complete the study (to properly render the prototypes). Median completion time for our study was 15 minutes and 48 seconds, and participants were compensated $5.00.

 We piloted our study setup with 45 participants prior to launching the full study. Based on a power analysis for our planned statistical tests, at least 66 participants per condition (786 participants total) would be needed to detect a moderate effect size with at least 80% power. In total, 1,316 participants from Prolific completed our study between July 28 and July 30, 2021.

4.2.3 Data Analysis. Our analysis includes data from 1,109 participants. We did not include responses from 127 participants who were inadvertently exposed to two different versions of our consent notice due to a technical issue with Useberry prior to completing the survey.7 We also removed responses from 42 participants who were detected using a mobile device by Useberry, as our prototypes were designed for tablet or desktop viewing. Last, we removed 38 participant responses for which a valid Useberry session (sessions in which we could confirm participants saw a consent notice either through successful task completion or reviewing their interaction data) was not recorded. A few participants completed the study twice, so we retained only their first submission.

We analyzed user interaction and timing metrics collected via Useberry, as well as participants’ survey responses. User flow diagrams generated by Useberry provided aggregated metrics about which pages of the consent interface and website participants visited during the study task, providing insight into whether participants engaged with the consent interface beyond making a consent decision. Useberry also provided metrics with regards to how long participants spent on the study task overall and the number of clicks they made prior to task completion. However, Useberry could not be configured to record individual participants’ exact consent decision in a format appropriate for such large-scale analysis; therefore we analyzed participants’ self-reported consent decision from the survey. Participants first indicated which cookie options they selected in the recall portion of the survey. To mitigate recall bias, participants answered the question again after reviewing the consent interface. A researcher reviewed a recording of a participant's interactions with the prototype captured by Useberry to verify their consent decision if there was a discrepancy in their response to these two questions or if they indicated selecting an unavailable option (i.e., “Allow social media cookies” or “Allow no cookies”). Approximately 20% of participants’ consent decisions were reviewed in this manner. Participants’ self-reported data was used to summarize the consent decisions made by users of each of the design variants.

In our reporting of findings, we highlight cases where we observed significant differences between the best-practices design variant and another variant, presenting both p-values and an effect size for the appropriate statistical test. Since our study was powered to detect at least moderate effect sizes with at least 80% power for our planned analysis, we note any significant results for which a smaller effect size was observed. P-values from any post-hoc pairwise comparisons were adjusted with a Bonferroni correction to be able to correct for additional comparisons with categorical data.

We conducted a thematic analysis of the responses to the qualitative survey questions asking participants about their expectations and goals related to their consent decision, any additional cookie options they desired, and methods for changing their initial consent decision. One member of the research team developed an initial codebook based on a subset of 10% of responses drawn at random. Two researchers then independently coded another random subset of 20% of the data, achieving a Cohen's κ inter-rater agreement of 0.84 (averaged over all questions), which is considered high agreement [9]. Any conflicts in the coding were resolved and the codebook was accordingly modified in collaboration. The remaining survey responses were coded by a single researcher using the modified codebook.

4.3 Participant Demographics

Table 5 provides a demographics summary of our study population. While our participant sample was diverse, it was not representative of the US population, skewing more female, white, and younger than the general population [41]. It is likely that our study was impacted by an influx of new registrations on Prolific by young women that occurred in July 2021 due to a viral video on TikTok [4]. We report on the impact of age and gender in our analysis of participants’ consent decision, awareness of available cookie options, comprehension of the interface, and investment in decision-making. The vast majority of our participants (85.8%) reported shopping online at least once a month, and only four participants indicated that they never shop online. This suggests that participants in our sample likely had prior experiences with websites similar to our prototype that may have influenced their interactions during our study.

Table 5: Summary of participant demographics. Participants were allowed to select multiple options for race/ethnicity so percentages are greater than 100. Those who reported having a formal education or work experience in a computer-related field were counted as technical experts.
Gender Age (Years) Race/Ethnicity Education Income Tech Expertise
Agender 0.45% 18-24 64.9% Am. Indian/Alaska Native 1.0% High school or less 15.0% < $10k 8.6% Yes 17.0%
Female 79.8% 25-34 26.3% Asian 8.7% Some college 30.7% $10k to $49,999 31.2% No 83.0%
Male 15.1% 35-44 5.6% Black 5.1% Associates/Bachelors 40.7% $50k to $99,999 29.5%
Non-binary/Genderqueer 4.1% 45-54 2.4% Hispanic/Latinx 3.2% Graduate/Professional 13.6% $100k to $149,999 14.5%
Self-described 0.36% 55-64 0.63% Hawaiian/Pacific Islander 0.26% No response 0.09% $150k 9.8%
No response 0.27% > 65 0.0% White 79.7% No response 6.4%
No response 0.45% Self-described 1.1%
No response 1.1%

4.4 User Study Results

Our study results, summarized in Table 6, highlight that several design parameters that we explored significantly impacted the usability of consent interfaces. We found that the absence of in-line options within the initial screen of the interface impacted participants’ consent decision, comprehension of available cookie options, as well as sentiment toward the consent interface. Additionally, we observed that awareness of available cookie options was impacted by the prominence of the consent interface. Furthermore, a persistent “Cookie Preferences” button improved participants’ ability to change their initial consent decision.

Table 6: Summary of how study conditions significantly differed from best-practices when considering the six usability aspects explored in the user study.
Usability Aspect Significant Conditions Summary
User Needs options-embeddedLink, options-interfaceButton, layout-multilayer, worst-practices Participants in these conditions were significantly more likely to consent to all cookies, compared those in best-practices.
User Ability & Effort prominence-cornerButton Participants in prominence-cornerButton were significantly less likely to select their preferred consent decision, compared to those in best-practices.
User Awareness prominence-cornerButton, worst-practices Participants in these conditions were significantly less aware of a privacy decision, compared to those in best-practices.
User Comprehension options-embeddedLink, options-interfaceButton, layout-multilayer Participants in these conditions answered more comprehension questions correctly after reviewing the consent interface
User Sentiment options-embededLink, layout-multilayer, worst-practices Participants in options-embededLink and layout-multilayer were significantly more likely to report choosing the “easiest option”; those in worst-practices were more likely to report skipping over the interface text, compared to those in best-practices.
Decision Reversal reversal-cookiePolicy Participants in reversal-cookiePolicy were significantly less likely to recognize a correct method to change their initial cookie consent decision, compared to those in best-practices.

4.4.1 User Needs. The majority (72.7%) of our participants who reported making a consent decision selected the “Allow all cookies” option in the interface, 24.4% selected “Allow only strictly necessary cookies,” and another 2.9% allowed some custom combination of strictly necessary, performance, functional, or targeting cookies. A Pearson's chi-squared test found that participants’ reported cookie consent decision (shown in Figure 4) significantly differed across conditions (p < 0.001, Cramer's V = 0.29). Participants in four conditions that did not include in-line options (options-embeddedLink, options-interfaceButton, layout-multilayer, and worst-practices) were significantly more likely to consent to all cookies, compared those in best-practices. We did not observe significant impact of age or gender on participants’ consent decision.

Figure 4
Figure 4: Participants’ reported consent decisions in their interactions with the prototype website where “custom” refers to any combination of strictly necessary, performance, functional, or targeting cookies. Three participants who saw blocking consent interface (in the reversal-cookiePolicy, reversal-noInstructions, and button-generic conditions) bypassed making a consent decision by clicking on other links within the consent interface, which dismissed the interface in the prototype.

About half of participants who selected “Allow all cookies” (50.2%) described that their goal was to dismiss the consent interface (e.g., “I just wanted to get to the website and thought, ‘Okay, whatever”’). suggesting that participants may have become habituated into clicking this option when available. Others who allowed all cookies described more specific goals, such as enabling specific features of the website (e.g.,“Ease of use when I return to the website in remembering my information”), allowing for full functionality of the website (e.g., “To gain full access to the website and all its features”), or improving the performance of the website (e.g.,“For the website to run as smooth as possible”). In contrast, the majority of those who only allowed strictly necessary cookies (57.9%) described privacy-related goals, including limiting the amount of personal data that is collected (e.g., “Bare minimum private information collected”) or web tracking that may occur which could lead to targeted ads (“I don't want my actions to be tracked unnecessarily, especially for targeting ads.”). Some participants who selected this option expressed that they wanted to limit the number of cookies because of an incomplete understanding of web cookies (e.g.,“I do not really understand cookies, but I think that they clog up your computer so I wanted to avoid this.”). These results highlight the importance of providing cookie options that align with specific goals.

In assessing user needs related to the consent interface, we also asked participants to describe what, if any, additional options related to cookies they would like to have. While the majority of participants did not articulate any additional choices they would like to have, 157 participants suggested providing an option for denying all cookies (which would be infeasible for an e-commerce website given current web technology). Additionally, 48 described other suggestions, including providing “cookie options” for other privacy or security-related features (e.g., “Cookies that will help keep passwords and logins safe.”), or an option for cookies not to persist beyond the browsing session (e.g., “Option to clear cookies when done browsing”). In lieu of additional options, 36 participants desired additional information, such as definitions for the term cookies and different cookie categories or how the website would behave if not all cookies were allowed.

4.4.2 User Ability & Effort. In the survey, participants were provided an opportunity to review the consent interface again and were explicitly asked to select what their preferred consent decision would be for the website. In their response, 40.1% indicated they would want to allow all cookies, 29.7% preferred to allow only strictly necessary cookies, 25.2% indicated a custom combination of cookie categories, and 5.1% preferred that the website not use any cookies at all. Excluding participants who reported that they would prefer not to allow any cookies (a preference that could not be selected in any condition), less than half of participants (45.3%) actually selected their preferred consent decision during their interactions with the website. A Pearson's chi-squared test found that this significantly differed across conditions (p < 0.001, V = 0.34). As no participants in prominence-cornerButton made a consent decision, post-hoc comparisons found that participants in that condition were significantly less likely to select their preferred consent decision compared to those in best-practices. However, the majority of participants (74.0%) felt that it was very easy or somewhat easy to make their preferred consent decision, which did not significantly differ across conditions.  Taken together, these results suggest that while the majority of participants in the prominence-cornerButton condition did not find it difficult to use the consent interface, the absence of a banner or pop-up consent interface severely impacted participants’ ability to provide their cookie preferences.

Participants spent an average of 1 minute and 28 seconds with 9.1 clicks to complete the study task (i.e., adding a product to the shopping cart) which was not found to significantly differ across conditions. This suggests that the effort required to complete a consent decision was similar across conditions. In our analysis of participants’ interactions with the prototype website, we observed that 24.0% of participants in worst-practices and 19.8% of participants in prominence-nonblockingBanner went directly to the catalog without making a consent decision. No participants in prominence-cornerButton were observed indicating their cookie preferences at any point during their interactions with the website. This implies that a substantial portion of users are likely not to indicate their cookie preferences if not blocked from using other parts of the website.

Beyond making a consent decision with a button option, we observed 99 additional interactions with other components of the cookie consent interface, seven interactions with one of the links to the website's privacy policy (located within the consent interface or in the footer of the website), and no interactions with the website's cookie policy. Figure 5 provides a summary of participants’ engagement with cookie-related options. The most common interactions were changing in-line options in the initial consent interface, clicking the persistent “Cookie Preferences” button, clicking the link or button leading to options from the initial consent interface, and changing the consent option toggles in the “Cookie Preferences” interface.

Figure 5
Figure 5: A summary of participants’ engagement with the cookie consent interface beyond selecting one of the button options. Specifically, we noted (if applicable to the study condition) whether participants changed any of the in-line options in the interface, clicked on the link or button leading to the cookie choices interface, clicked the persistent cookie preferences button, or changed any toggles within the cookie choices interface. Interactions not applicable to a condition are marked with a ‘X’.

4.4.3 User Awareness. While only 66.6% of participants initially recalled making a privacy-related decision, when we asked them specifically about cookie-related decisions, 86.8% of participants reported making a consent decision during their interaction with the prototype website. This suggests that some participants may have been habituated to consent interfaces and did not pay enough attention to the cookie consent interface to understand that it was related to privacy. In their recall of consent options, participants correctly selected between three and four cookie categories out of seven listed (two of which were not actually available on the website). We found that both awareness of a privacy decision (Fisher's exact test, p < 0.001, V = 0.48), as well as available options related to cookie categories (Kruskal-Wallis test, p < 0.001, η = 0.053 [small effect]) significantly differed across conditions. In follow-up pairwise comparisons, participants in the prominence-cornerButton and worst-practices conditions reported significantly less awareness compared to those assigned to best-practices . Three-quarters of participants in best-practices recalled making a privacy decision and, on average, participants in this condition correctly recalled 3.5 cookie options. In comparison, half of worst-practices and only 2.9% of prominence-cornerButton participants recalled a privacy decision. On average, participants in worst-practices correctly selected 2.8 options and those assigned to prominence-cornerButton recalled 2.7 options. The relatively poor awareness of participants in these two conditions is unsurprising given that less than three quarters of worst-practices participants and no prominence-cornerButton participants made a cookie consent selection (see Figure 4) and thus were unlikely to even view the options; the options they recalled correctly are likely based on their past experience with cookie consent decisions.

Kruskal-Wallis tests found that recall of cookie options was also significantly impacted by age (p = 0.005, η =.006 [small effect]) and gender (p < 0.001, η = 0.010 [small effect]). Those aged 35 and older had better recall of available options (3.6 correct) than those younger than 35 (3.2 correct). Compared to women (3.2 correct), men were found to have significantly better recall (3.6 correct). However it is unclear whether participants with better observed recall truly were better at recalling information, or if they had simply reviewed their cookie options more carefully.

Our analysis of interactions with the website prototype (reported in Figure 5) also provides evidence that user awareness of cookie options was impacted by the prominence of the initial consent interface. While the “Cookie Preferences” button in the bottom corner of the webpage was used in all of the other design variants that included it, it seemed to go ignored in prominence-cornerButton. This suggests that a fully-blocking or banner-style consent interface led to greater awareness of available cookie choices.

4.4.4 User Comprehension. To gauge participants’ comprehension of their cookie-related choices, the survey included five multiple-choice questions in which participants were asked to select the correct definitions for the term “cookies” (in the context of the internet) and each of the four cookie categories included in the interface. On average, participants correctly answered between two and three questions, based on their recall of the website and consent interface when their attention likely was not focused on available cookie choices. Less than half of participants (47.6%) selected the correct definition for “performance cookies” and only 16.0% selected the correct answer for “functional cookies,” suggesting that these two labels for cookie categories defined by the ICC UK are not very intuitive. Most commonly, participants thought functional cookies were those that were needed for the website to work properly; this is actually the correct definition for “strictly necessary cookies.”

A Friedman test found a significant improvement in comprehension (p < 0.001, Kendall's W = 0.59) by about one question when participants answered the same five comprehension questions again after being able to review the consent interface. Unlike participants’ unfocused comprehension, there was a significant difference across conditions in focused comprehension (p < 0.001, η = 0.08). Compared to those in best-practices, participants in options-embeddedLink, options-interfaceButton, and layout-multilayer answered more of the comprehension questions correctly after reviewing the consent interface. This may be because participants in these conditions were not exposed to the different cookie category terms through in-line options and instead saw them on the Cookie Preferences page where they were defined, making them less prone to guessing than participants who saw the cookie categories on the initial screen. When asked which aspects of the consent interface they referred to when answering the survey questions, a larger percentage of participants in these conditions reported referring to the Cookie Preferences page, compared to those in best-practices.

Kruskal-Wallis tests found significant differences in unfocused comprehension by age (p = 0.01, η = 0.005 [small effect]), as well as gender (p < 0.001, η = 0.01 [small effect]). Those younger than 35 correctly answered 2.7 questions, compared to 3.1 questions for those 35 and older, while women answered 2.6 questions correctly on average, compared to 3.1 questions for men. These results are aligned with findings related to awareness of cookie options.

After reviewing the consent interface, participants were asked how easy or difficult they thought the consent interface was to understand. Over two-thirds (68.0%) reported that it was somewhat easy or very easy to understand, which was was not significantly different across conditions. The survey also asked participants about their comprehension of which cookie consent option was being recommended by the interface, reported in Figure 6. While none of the design variants explicitly recommended a consent option, a Pearson's chi-squared test found that participants’ interpretations of what was being recommended significantly differed across conditions (p < 0.001, V = 0.14). The majority of participants in worst-practices (60.0%), text-lossAversion (51.7%), and options-embeddedLink (50.0%) thought that the interface was recommending to allow all cookies, though post-hoc comparisons did not find this to significantly differ from best-practices.

Figure 6
Figure 6: Participants’ comprehension of what (if any) cookie consent options the website seemed to be recommending.

Participants were also asked to indicate the likelihood of five different scenarios if a cookie consent decision was not made on the website. The most common expectations were “all cookies would be allowed and the entire website would still work” and “no cookies would be allowed but some parts of the website would still work,” rated as “probably yes” or “definitely yes” by 68.3% and 56.0% of participants respectively. This highlights that ambiguity that exists in current implementations of cookie consent interfaces, as both of these scenarios are technically feasible, although their legality may depend on regulations in the user's location.

4.4.5 User Sentiment. To gauge participants’ level of investment in making a cookie consent decision, we asked participants who indicated that they made a consent decision on the prototype website two multiple-choice questions and one Likert scale question related to their decision-making process. Pearson chi-squared tests found that there was a significant different across conditions in participants’ strategies for selecting their cookie preferences (p < 0.001, V = 0.18), as well as their engagement with the interface text (p < 0.001, V = 0.23). Similarly, a Kruskal-Wallis test found that participants’ ratings for how carefully they made their consent decision also differed across conditions (p < 0.001, η = 0.051 [small effect). Significantly more participants in options-embededLink (83.3%, p.adj = 0.006) reported choosing the “easiest option” when making their consent decision and were more likely to report that they made their decision “not at all carefully” (73.4%, p.adj = 0.04), compared to participants in best-practices (55.7% and 46.6% respectively reported the same). Similarly, significantly more participants in worst-practices than best-practices made their decision “not at all carefully” (75.6%, p.adj = 0.01) and reported skipping over the interface text (59.0%, p.adj = 0.001). Those in layout-multilayer were also significantly more likely to report choosing the “easiest option” (80.7%, p.adj = 0.04). This suggests that the absence of in-line options within the initial screen of the consent interface may have reduced participants’ investment in their consent decision.

Pearson chi-squared tests and a Kruskal-Wallis test comparing responses to these three questions also revealed significant differences in decision-making investment with age and gender. Compared to those 35 years or older, younger participants were more likely to report choosing the “easiest option” (p < 0.001, V = 0.16), skipping over the interface text (p = 0.001, V = 0.11), and making their decision “not at all carefully” (p <.001, η = 0.051 [small effect]). Relative to men, women reported less investment in their decision-making, being more likely to choose the “easiest option” (p < 0.001, V = 0.20), skipping over the interface text (p = 0.002, V = 0.10), and making their decision “not at all carefully” (p < 0.001, η =.065).

4.4.6 Decision Reversal. Two design variants (reversal-noInstructions and reversal-cookiePolicy) directly explored participants’ ability to revoke or change their consent decision. A Pearson's chi-squared test found that participants in the best-practice condition that contained a persistent “Cookie Preferences” button in the bottom right corner of the page were significantly more likely than those in reversal-cookiePolicy which did not contain this button to recognize a correct method to change their initial cookie consent decision (p = 0.001, V = 0.28). The vast majority (81.8%) of best-practices participants stated that they would use this button to change their decision, while 45.3% of participants in reversal-cookiePolicy stated they would visit the website's cookie policy (as instructed in the interface text). We found that the presence of reversal instructions did not have a significant impact on participants’ ability to reverse their initial consent decision.

When asked how they would reverse their preferences if there was no “Cookie Preferences” button, only 16.1% of participants in the conditions that contained this button described an effective alternative method for revising their consent decision for the website, such as the website's privacy or cookie policy, deleting browser cookies, using a different browser or device, or visiting the website in private browsing mode. This suggests that after being exposed to the “Cookie Preferences” button its absence had a much greater impact than if participants had not seen it at all. Along these lines, 42.2% said that they would give up trying to change their consent preferences or just leave the website. Over a fifth (22.9%) described other strategies that could potentially lead them to a correct decision reversal path, such as changing browser settings, looking through the settings or other parts of the website, contacting the website, or searching for instructions using a search engine. A small portion of participants (10.4%) described an incorrect strategy such as refreshing the page or revisiting it in another tab, and another 6.3% were not sure what they would do to reverse their consent decision.

5 DISCUSSION

In this section, we first describe limitations of our evaluation of cookie consent interfaces. We then review the implications of our findings on the design of such interfaces.

5.1 Limitations

While our study provides valuable insights into the usability of consent interfaces, it is not without its limitations.

5.1.1 Inspection-based Evaluation. The procedure used to evaluate consent interfaces likely introduced some bias. First, we compiled our list of websites to evaluate from multiple sources and not through random sampling. In addition, the consent interfaces that we observed from a US-based IP address could differ from those displayed to visitors in the EU or elsewhere. We developed our list of 10 design parameters based on our inspection and evaluation of consent interfaces implemented by CMPs. It is possible that cookie consent interfaces that are not implemented through CMPs incorporate other design parameters that were not uncovered in our inspection-based evaluation. Furthermore, our user study only explored a subset of the identified design parameters and implementations corresponding to these parameters. Though prior research and best practices exist with regard to the two parameters we did not include (placement of button options and granularity of the choices offered), these should be further explored in the context of cookie consent interfaces. In addition, our study examined cookie consent interfaces on computers and not on mobile devices. Our study also did not evaluate the accessibility of cookie consent interfaces, which should be implemented according to standardized accessibility guidelines to ensure that they are usable by a larger population of internet users [43].

5.1.2 User Study. Though our user study evaluated our cookie consent interface designs in a realistic context, participants were aware that they were interacting with a prototype website through Useberry, which may have impacted their interactions and impressions of the consent interface. Additionally, while Useberry allowed us to capture data related to the time and number of clicks participants spent on the study task overall, we were unable to analyze these metrics specifically for the consent interface. Considering participants’ interactions with the consent interface, it appears that none of the conditions required significantly more effort, with the exception of the prominence-cornerButton variant. As none of the participants in that condition attempted to make a consent decision, we cannot draw conclusions about the amount of effort required. However the lack of interaction with “Cookie Preferences” button in that condition raises questions about whether participants even noticed that it was there. Furthermore, we were only able to summarize participants’ consent decisions for each variant through their survey responses, which could have been inaccurately reported. However, manual inspection of the session recordings of our pilot participants found misreporting to be rare.

Our study also explored cookie consent interfaces in the context of a single website. As such, we cannot provide insight into usability aspects that may be impacted when users encounter such interfaces across multiple websites or apps. Future work could more deeply explore usability issues, such as decision fatigue, as well as the impact of trust in a particular brand in the context of users’ cookie consent decision.

Due to time constraints, we aimed for a diverse, though not necessarily representative, sample for our user study. However, our results may be impacted by the relatively poor gender and age diversity of our final study population. While we did not find that gender or age significantly impacted participants’ consent decisions, we did observe differences in user awareness, comprehension, and sentiment. Female-identifying participants and those under the age of 35 had less awareness and comprehension of available cookie options and were less invested in their decision-making, on average, compared to male-identifying participants or those older than 35. Technical literacy more generally is likely to differ with gender and age, as 10.5% of women under 35 in our study sample reported having a degree or working in a computer-related field, compared to 78.7% of men older than 35. Given that our sample was dominated by participants with less investment in in their decision-making and lower comprehension of available cookie options, we expect we may have failed to detect some differences in conditions that might be detectable in a more representative study. Future work evaluating the usability of consent interfaces should be conducted with a study population that is more representative of the internet population overall.

5.2 Design Implications

We found that several of the design parameters we explored had a significant impact on the usability of the consent interface. Table 6 provides a summary of our findings with regards to how they impacted the six usability aspects studied.Among the seven design parameters we explored, we find that the prominence of the consent interface, presence of in-line options within the initial screen of the interface, and presence of a persistent “Cookie Preferences” button for enabling changes to the initial consent decision had the greatest impact on usability. These results are in line with prior work which suggest that more salient privacy information and options yield better usability outcomes (e.g., [6, 42]).

5.2.1 Prominence of the interface. Our results related to user awareness in section 4.4.3 suggest that less conspicuous consent interfaces may not be effective in communicating the presence of privacy choices. Additionally, as highlighted in section 4.4.2, without a fully-blocking “consent wall” interface, many participants proceeded to use the website without indicating a consent decision. Participants had varying expectations as to how this inaction would be interpreted by the website, as reported in section 4.4.4, suggesting a need for non-blocking consent interfaces to clarify the privacy implications of choosing not to engage. However, fully-blocking interfaces may come with some usability trade-offs, particularly for users who may not know what their consent preference might be for a particular website or app before they have interacted with it. It is important for users to be able to get enough information about the context of this decision, which may be through details available within the consent interface itself, or from other parts of the website that are still visible or accessible with the consent interface overlay.

Ultimately, our findings on interface prominence highlight a fundamental problem with applying the consent interface paradigm to such frequently-encountered decisions as cookie consent decisions. Cookie consent interfaces need to be prominent in order to promote user engagement and informed decisions, but such prominence necessarily introduces friction into a user's web browsing experience, decreasing usability. Users who are not privacy conscious are still forced to take some action and users who do desire privacy-protective options are burdened with taking the time needed to actually make an informed decision. As long as the cookie consent interface paradigm persists, our results suggest that interface prominence may indeed be necessary for achieving informed consent. However, as we discuss below in section 5.4, automated consent mechanisms are likely to offer a more usable solution.

5.2.2 Presence of in-line options. Our findings provide further support to prior recommendations that cookie consent interfaces should include in-line options to better enable user decision-making [29, 42], but add a new observation that users tend to guess (often incorrectly) at the meaning of common cookie categories when they are presented in-line without accompanying definitions. As described in section 4.4.1, without these options available participants were more likely to allow all cookies. Along these lines, an additional button option that more clearly corresponds to allowing only strictly necessary cookies may be useful to users, considering that over a quarter of participants reported this as their preferred consent decision. However, we did find a negative impact of providing in-line options on participants’ comprehension of choices when explicitly instructed to revisit the consent interface. Participants in these conditions seemed to be more likely to guess at the definitions of the available cookie options rather than review the definitions on the “Cookie Preferences” page. This suggests that providing definitions of cookie categories within the in-line options, such as through a tooltip or “accordion” interface, may help with comprehension of choices and better enable user decision-making. However, both the addition of a third button option and tooltip or accordion definitions for cookie categories should be tested for their usability impact.

5.2.3 Enabling decision reversal. While the “Cookie Preferences” button on its own was ineffective for communicating the presence of available cookie options, our results in section 4.4.6 revealed that it did help participants identify a means to change their initial consent decision. Furthermore, our results suggest that the presence of this button following an interruptive banner or fully-blocking interface is enough on its own to facilitate decision reversal, as we found no significant differences between the reversal-noInstructions and best-practices variants. Therefore text instructions in the consent interface related to the “Cookie Preferences” button for decision reversal could be removed, which would reduce the reading effort required for the consent interface.

5.2.4 Recommendations for CMPs. While individual websites may not have the incentives or resources to create more privacy-friendly consent interfaces, CMPs may be in a position to change website practices, although they may need to be pressured to do so. Our inspection-evaluation of CMP-implemented consent interfaces, as well as prior work [29], make it clear that CMPs are enabling companies to implement consent interfaces with known dark patterns. Such design options include unequal paths for the most and least privacy protective options, privacy-invasive default cookie options, confusing placement of button options, and consent interfaces without any explicit cookie options. As such, CMPs should ensure that the design parameters they make available to organizations do not facilitate such dark patterns. Design choices included in our best-practices variant could guide CMPs in the design options they provide, particularly our recommendations related to the prominence of the interface, presence of in-line options, and enabling decision reversal. Furthermore, CMPs should provide guidance with regards to the text that is included in the interface, such as avoiding loss aversion framing of choices and suggesting more intuitive language to use instead of “performance” and “functional” cookies. Last, CMPs should conduct usability assessments similar to this study, to identify and address potential negative impacts of the design options they provide to organizations for their consent interfaces. Some CMPs already offer templates for compliance with particular laws, e.g., switching to opt-in rather than opt-out for GDPR compliance. Regulators may be able to put pressure on these CMPs to do more, requiring CMPs who claim compliance to remove or warn against options that would result in dark patterns or usability challenges.

5.3 Incentives

In the absence of legal requirements and enforcement related to usability, websites may not have incentives to make their cookie consent interfaces more usable or privacy friendly. This is especially true for websites that serve visitors from the US, which was the focus of our study. In the US, where by default websites can usually set cookies without restriction, more engagement with cookie consent interfaces may lead to more users opting out of more cookies. The situation is somewhat different under regulatory regimes such as GDPR where opt-in consent is required [13]. To comply with GDPR websites must refrain from setting unnecessary cookies until users have opted in. Thus these websites have an incentive to engage more users in their consent interface so that more of them will opt-in to additional cookies. Some have even blogged about A/B testing their consent interface to increase engagement [36].

As privacy legislation is written to require usability and prohibit dark patterns, there are opportunities for regulators to leverage these laws to force websites to revise their cookie consent interfaces. Our findings can guide regulators in identifying potentially problematic interfaces. Furthermore, our study methods and survey questions could be used by regulators in their evaluations. Besides going after individual websites, we suggest that regulators work with CMPs to offer guidance and default options for their customers that result in more privacy-friendly and usable consent interfaces.

5.4 Reducing the Burden of User Consent

While more usable cookie consent interfaces would better enable users to exercise their consent preferences, this model of consent still poses a considerable burden on users. Considering that users must make consent decisions on each website or app they use, aggregated together the cost of reading cookie consent interfaces, comprehending available options, and making a decision is not trivial. It is likely that users have formed coping strategies to manage the burden of cookie consent decisions, as over half of participants in our best-practices condition reported selecting the “easiest option” when making their consent decision. Therefore, usable consent interfaces are only part of the solution for enabling effective privacy protection on websites.

Technology could play a role in further reducing the burden of user consent. As described in section 2.4, browser-based consent mechanisms could automate users’ consent decisions so that they do not have to make consent decisions on every website they visit. However, such approaches face their own challenges. These automated mechanisms may prove to be more cumbersome to users who want to make website-specific consent decisions. Browser extensions must be installed and managed by users who may be unwilling or lacking the technical skills to do so. Additionally, the scheme proposed by the UK's ICO could suffer the same fate as the DNT standard, which ultimately failed due to lack of industry support [11]. However, given existing regulatory requirements related to capturing user consent for data collection, it is possible that this proposal may have more success; a browser-based consent mechanism may be more attractive to industry stakeholders than consent interfaces that interrupt the user experience. Until alternative consent mechanisms become widely adopted, it is necessary to ensure that current mechanisms are usable to facilitate effective privacy protection.

6 CONCLUSION

We conducted a two-part study of cookie consent interfaces, finding that the design of these interfaces significantly impact several aspects of usability. We first conducted an inspection-based evaluation of consent interfaces implemented through consent management platforms (CMPs) which identified design parameters that organizations can customize for their websites or apps. To explore which design choices for these parameters result in better usability, we conducted a large-scale between-subjects experiment on Prolific evaluating 12 cookie consent design variants. We find that several design choices, such as a “consent wall” implementation of the consent interface, in-line options corresponding to cookie categories, and a persistent “Cookie Preferences” button enabling decision reversal yielded significantly better usability outcomes. Our comprehensive usability assessment of cookie consent interfaces complements prior work in this space by providing actionable design implications that consider a more holistic perspective of usability, beyond considerations related to dark patterns.

ACKNOWLEDGMENTS

This research was supported in part by the National Science Foundation under grant CCF-1852260, Carnegie Corporation of New York, Innovators Network Foundation, and Facebook. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or other funders.

A CONSENT INTERFACE DESIGN VARIANTS

best-practices (on landing page)

All fully-blocking variants were the same dimensions on the landing page as best-practices.

prominence-cornerButton

Clicking the button led to the single-layer “Cookie Preferences” page.

prominence-banner

options-embeddedLink

options-interfaceButton

text-lossAversion

text-layoutParagraph

button-generic

layout-multilayer

The “Edit cookie preferences” button links to the multi-layer version of “Cookie Preferences” page.

reversal-noInstructions

reversal-cookiePolicy

worst-practices (on landing page)

B SURVEY QUESTIONS

Questions with standard bullets were single-select, while those with squares allowed for multiple selections.

  1. Were you able to successfully complete the task?
    • Yes, I added a product to my cart
    • No, I skipped the task
    • No, I had a technical problem (please describe)

[Logic for Q2: If “Were you able to successfully complete the task?” = “Yes, I added a product to my cart” (participants who answered “No” were directed to Q3)]

  1. Which product did you select?
    • Grandma's Diner Special
    • Adult Sippy Cup
    • Delicate Irish Coffee
    • Afternoon Tea
    • The Minalimist
    • Stemware Essentials
    • Bamboo Tea Set
    • Insulated Espresso Cup
    • Café Classic Set
    • I don't remember
  2. How easy or difficult was it to shop on this website? (5-point Likert scale from “Very easy” to “Very difficult”)
  3. Do you recall making any privacy-related decisions during your interaction with the cups n’ such website?
    • Yes
    • No
    • Not sure

[Logic for Q5: If “Do you recall making any privacy-related decisions during your interaction with the cups n’ such...” = “Yes”]

  1. What was this decision about?
    • The use of cookies on the website
    • The creation of a username and password for the website
    • The visibility of credit card info on the website
    • The use of location data while shopping on the website
    • Other (please describe)
  2. When visiting cups n’ such's website, you might have seen an interface related to the use of cookies. Which option(s) do you remember selecting? (Select all that apply)
    • Allow all cookies
    • Allow only strictly necessary cookies
    • Allow social media cookies
    • Allow performance cookies
    • Allow functional cookies
    • Allow targeting cookies
    • Don't allow any cookies
    • I don't remember
    • I didn't select any options related to the use of cookies

[Logic for Q7 and Q8: “If When visiting cups n’ such's website, you might have seen an interface related to the use of cook...” != “I don't remember” And “When visiting cups n’ such's website, you might have seen an interface related to the use of cook...” != “I didn't select any options related to the use of cookies”]

  1. What do you expect to happen since you selected [consent decision in Q6]? (open-ended question)
  2. What were you trying to achieve when you selected [consent decision in Q6]? (open-ended question)

[Logic for Q9-Q11: “If When visiting cups n’ such's website, you might have seen an interface related to the use of cook...” != “I didn't select any options related to the use of cookies”]

  1. Which of the following best describes how you made your decision related to the use of cookies on the cups n’ such website?
    • I picked an option based on my actual cookie preferences
    • I picked whichever option seemed easiest so the consent interface would go away
    • I picked an option randomly
    • Other (please describe)
  2. How carefully did you consider the options related to cookies on the cups n’ such website?
    • Not at all carefully
    • Moderately carefully
    • Extremely carefully
  3. The cookie notice interface included some text. What did you do when you saw it?
    • Skipped over it
    • Skimmed it
    • Read it carefully
  4. What options related to cookies do you recall being available to you on this website? (each option rated as “Definitely not available,” “Probably not available,” “Not sure if available,” “Probably available,” “Definitely available”)
    • Allow all cookies
    • Allow only strictly necessary cookies
    • Don't allow any cookies
    • Allow social media cookies
    • Allow performance cookies
    • Allow functional cookies
    • Allow targeting cookies
    Instructions: Please select the definition that fits best for each of the following terms.
  5. In the context of the web, what is a cookie?
    • A security token for two-factor authentication
    • A small piece of data stored on a computer to keep track of information such as logins or websites the user has visited previously
    • A memorized secret used to confirm the identity of a user
    • A unique string of numbers separated by periods that identifies each computer using the Internet Protocol to communicate over a network
    • I don't know
  6. What are strictly necessary cookies?
    • Cookies that are needed for the website to work properly
    • Cookies that are needed for collecting certain metrics
    • Cookies that are needed for determining your location
    • I don't know
  7. What are performance cookies?
    • Cookies that help measure and improve website features
    • Cookies that are given priority over other cookies on the website
    • Cookies that make the website run faster
    • I don't know
  8. What are functional cookies?
    • Cookies that are needed for the website to work properly
    • Cookies that help personalize the website's services for you
    • Cookies that are given lower priority than other cookies on the website
    • I don't know
  9. What are targeting cookies?
    • Cookies that are used for delivering personalized advertisements
    • Cookies that help users navigate the website
    • Cookies that are needed for determining your location
    • I don't know
  10. How likely do you think the following scenarios are to happen if you don't make a selection regarding the use of cookies on this website? (each option rated as “Definitely not,” “Probably not,” “Not sure,” “Probably yes,” “Definitely yes”)
    • I would be blocked from using the website entirely
    • No cookies would be allowed so the website would not work at all
    • No cookies would be allowed but the entire website would still work
    • All cookies would be allowed and the entire website would still work
    • No cookies would be allowed but some parts of the website would still work

Instructions: Open the prototype again in a new tab by clicking the link below and keep it open for the remainder of the survey. Please answer the following questions after you review your options related to cookies. [Link to prototype ]

  1. You may have seen several cookie options on the prototype website. What additional options related to cookies would you like to have available to you, if any? (open-ended question)

Instructions: Next, we are going to ask some of questions again with your previous answers marked. After reviewing the information provided about the use of cookies on the website, please edit your answers if you need to.

[Questions 6, 7, 13-18 are asked again]

Instructions: Please answer the following questions, referring to the prototype if necessary.

  1. What option related to cookies do you think the website is recommending?
    • Allow all cookies
    • Allow selected cookies
    • Strictly necessary
    • Other (please describe)
    • The website isn't recommending any options (all options are presented equally)
  2. How easy or difficult do you find the cookie consent interface to understand? (6-point Likert scale from “Very easy” to “Very impossible”)
  3. What would be your preferred cookie consent decision for this website? (Select all that apply)
    • Allow all cookies
    • Allow only strictly necessary cookies
    • Allow social media cookies
    • Allow performance cookies
    • Allow functional cookies
    • Allow targeting cookies
    • Don't allow any cookies
    • Other (please describe)
  4. How easy or difficult would it be for you to make your preferred cookie consent decision? (6-point Likert scale from “Very easy” to “Very impossible”)
  5. To what extent do you feel... (each option rated as “Not at all,” “Moderately,”“Extremely,”“Not sure”)
    • Informed about the data being collected by cookies on this website?
    • That this cookie consent interface provides the choices you want related to the use of your data?
    • Informed about your choices related to cookies on this website?
    • Capable of making a decision related to cookies on this website?

[Logic for Q25: “If When visiting cups n’ such's website, you might have seen an interface related to the use of cook...” != “I didn't select any options related to the use of cookies”]

Instructions: The following questions refer to “your cookie consent decision” which refers to the decision you made about the use of cookies on cups n’ such the first time you encountered the cookie consent interface.

  1. To what extent do you feel... (each option rated as “Not at all,” “Moderately,”“Extremely,”“Not sure”)
    • Confident that your cookie consent decision was the best option for you?
    • Comfortable about how data associated with cookies will be used on this website, given your cookie consent decision?
    • That your cookie consent decision will be honored by the website?

Instructions: Please refer to the screenshot below for the following questions. (Screenshot of landing page with or without “Cookie Preferences” button depending on the condition)

  1. Suppose you already made a decision about how cookies can be used on this website. What would you do if you wanted to change your cookie consent decision, or make a decision if you didn't when first visiting the website? (open-ended question)
  2. What would you do if what you described in your previous answer was not available on the website? (open-ended question)
  3. Which parts of the website did you click on or look at when reviewing your options related to cookies to answer the survey questions? (Select all that apply)
    • Cookie policy
    • Privacy policy
    • Cookie consent interface (banner or popup that appeared when opening the website)
    • Cookie preferences button (button on bottom right corner of the page)
    • Cookie preferences page (page with toggles next to cookie categories)
    • Other (please describe)
  4. How frequently do you shop online?
    • Never
    • Less than once a month
    • A few times a month
    • A few times a week
    • Almost every day
  5. What is your age in years? Enter “0” if you prefer not to respond.
  6. How do you describe your gender identity?
    • Male
    • Female
    • Non-binary
    • Agender
    • Genderqueer
    • Prefer to self-describe
    • Prefer not to respond
  7. How do you describe your race or ethnic identity? (You may select more than one option.)
    • American Indian or Alaska Native
    • Asian
    • Black
    • Native Hawaiian or Pacific Islander
    • White
    • Prefer to self-describe
    • Prefer not to respond
  8. What is the highest level of school you have completed or the highest degree you have received?
    • Less than high school degree
    • High school graduate (high school diploma or equivalent including GED)
    • Some college but no degree
    • Associate degree in college (2-year)
    • Bachelor's degree in college (4-year)
    • Master's degree
    • Doctoral degree
    • Professional degree (JD, MD)
    • Prefer not to respond
  9. What was your approximate household income in 2020? Please answer based on your entire household's income in 2020, before taxes.
    • Less than $10,000
    • $10,000 to $19,999
    • $20,000 to $29,999
    • $30,000 to $39,999
    • $40,000 to $49,999
    • $50,000 to $59,999
    • $60,000 to $69,999
    • $70,000 to $79,999
    • $80,000 to $89,999
    • $90,000 to $99,999
    • $100,000 to $149,999
    • $150,000 or more
    • Prefer not to respond
  10. Do you have a formal education in a computer-related field, such as computer science or IT? (”Formal education” could mean a completed degree or certificate, or classes or trainings you took towards a degree or certificate.)
    • Yes
    • No
  11. Do you have work experience in a computer-related field, such as computer science or IT?
    • Yes
    • No
  12. Please enter your Prolific ID again.
  13. If you have any feedback on the survey or cookie consent interface you saw, please leave it here. (open-ended question)

REFERENCES

  • Dirk Bollen, Bart P Knijnenburg, Martijn C Willemsen, and Mark Graus. 2010. Understanding Choice Overload in Recommender Systems. In Proceedings of the Conference on Recommender Systems (RecSys). ACM, 63–70.
  • Dino Bollinger, Karel Kubicek, Carlos Cotrini, and David Basin. 2022. Automating Cookie Consent and GDPR Violation Detection. In USENIX Security Symposium. USENIX.
  • Harry Brignull. [n. d.]. Types of Dark Patterns: Confirmshaming. https://www.darkpatterns.org/types-of-dark-pattern/confirmshaming.
  • Nick Charalambides. 2021. We Recently Went Viral on TikTok - Here's What We Learned. https://blog.prolific.co/we-recently-went-viral-on-tiktok-heres-what-we-learned/.
  • Martin Degeling, Christine Utz, Christopher Lentzsch, Henry Hosseini, Florian Schaub, and Thorsten Holz. 2019. We Value Your Privacy...Now Take Some Cookies: Measuring the GDPR's Impact on Web Privacy. In Proceedings of Network and Distributed System Security Symposium (NDSS). Internet Society.
  • Nico Ebert, Kurt Alexander Ackermann, and Björn Scheppler. 2021. Bolder is Better: Raising User Awareness through Salient and Concise Privacy Notices. In Proceedings of the Conference on Human Factors in Computing Systems (CHI). ACM.
  • European Parliament. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R0679.
  • Yuanyuan Feng, Yaxing Yao, and Norman Sadeh. 2021. A Design Space for Privacy Choices: Towards Meaningful Privacy Control in the Internet of Things. In Proceedings of the Conference on Human Factors in Computing Systems (CHI). ACM.
  • Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. 2013. Statistical Methods for Rates and Proportions. John Wiley & Sons.
  • Colin M Gray, Cristiana Santos, Nataliia Bielova, Michael Toth, and Damian Clifford. 2021. Dark Patterns and the Legal Requirements of Consent Banners: An Interaction Criticism Perspective. In Proceedings of the Conference on Human Factors in Computing Systems (CHI). ACM.
  • Kashmir Hill. 2018. ’Do Not Track,’ the Privacy Tool Used by Millions of People, Doesn't Do Anything. Gizmodo (October 2018). https://gizmodo.com/do-not-track-the-privacy-tool-used-by-millions-of-peop-1828868324.
  • Maximilian Hils, Daniel W Woods, and Rainer Böhme. 2020. Measuring the Emergence of Consent Management on the Web. In Proceedings of the Internet Measurement Conference (IMC). ACM, 317–332.
  • Horizon 2020 Framework Programme of the European Union. 2021. Cookies, the GDPR, and the ePrivacy Directive. https://gdpr.eu/cookies/.
  • IAB Europe. 2021. Transparency & Consent Framework. https://iabeurope.eu/transparency-consent-framework/.
  • International Chamber of Commerce UK. 2012. ICC UK Cookie Guide. https://www.cookielaw.org/wp-content/uploads/2019/12/icc_uk_cookiesguide_revnov.pdf.
  • ISO Technical Committee 159. 2018. Ergonomics of Human-System Interaction. https://www.iso.org/standard/63500.html.
  • Mark J Keith, Courtenay Maynes, Paul Benjamin Lowry, and Jeffry Babb. 2014. Privacy Fatigue: The Effect of Privacy Control Complexity on Consumer Electronic Information Disclosure. In Proceedings of the International Conference on Information Systems (ICIS). Association for Information Systems.
  • Kevel. 2021. Consent Management Platform (CMP) 2021 Tracker. https://www.kevel.co/cmp/.
  • Daniel Kladnik. 2021. I Don't Care About Cookies. https://www.i-dont-care-about-cookies.eu/.
  • Oksana Kulyk, Annika Hilt, Nina Gerber, and Melanie Volkamer. 2018. “This Website Uses Cookies”: Users’ Perceptions and Reactions to the Cookie Disclaimer. In European Workshop on Usable Security (EuroUSEC).
  • Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczyński, and Wouter Joosen. 2019. Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation. In Proceedings of Network and Distributed System Security Symposium (NDSS). Internet Society.
  • Timothy Patrick Libert. 2017. Track The Planet: A Web-Scale Analysis Of How Online Behavioral Advertising Violates Social Norms. Ph. D. Dissertation. University of Pennsylvania, Philadelphia, PA.
  • Dominique Machuletz and Rainer Böhme. 2020. Multiple Purposes, Multiple Problems: A User Study of Consent Dialogs after GDPR. Proceedings on Privacy Enhancing Technologies (PoPeTs) 2 (2020), 481–498.
  • Arunesh Mathur, Jonathan Mayer, and Mihir Kshirsagar. 2021. What Makes a Dark Pattern...Dark? Design Attributes, Normative Considerations, and Measurement Methods. In Proceedings of the Conference on Human Factors in Computing Systems (CHI). ACM.
  • Célestin Matte, Nataliia Bielova, and Cristiana Santos. 2020. Do Cookie Banners Respect My Choice?: Measuring Legal Compliance of Banners from IAB Europe's Transparency and Consent Framework. In IEEE Symposium on Security and Privacy (SP). IEEE, 791–809.
  • Minimal Consent.2020. https://www.minimal-consent.com/.
  • Peter Morville. 2004. User Experience Design. http://semanticstudios.com/user_experience_design/.
  • Jakob Nielsen. 2012. Usability 101: Introduction to Usability. https://www.nngroup.com/articles/usability-101-introduction-to-usability/.
  • Midas Nouwens, Ilaria Liccardi, Michael Veale, David Karger, and Lalana Kagal. 2020. Dark Patterns after the GDPR: Scraping Consent Pop-Ups and Demonstrating Their Influence. In Proceedings of the Conference on Human Factors in Computing Systems (CHI). ACM.
  • European Court of Justice. 2018. Judgment of the Court (Grand Chamber) of 1 October 2019: Bundesverband der Verbraucherzentralen und Verbraucherverbände - Verbraucherzentrale Bundesverband e.V. v Planet49 GmbH. Official Journal of the European Union (March 2018). https://eur-lex.europa.eu/legal-content/en/TXT/PDF/?uri=uriserv%3AOJ.C_.2018.112.01.0009.01.ENG.
  • Office of the California Attorney General. 2019. The California Privacy Rights and Enforcement Act of 2020. https://oag.ca.gov/system/files/initiatives/pdfs/19-0017%20%28Consumer%20Privacy%20%29.pdf.
  • Office of the California Attorney General. 2020. California Consumer Privacy Act (CCPA): Final Text of Proposed Regulations. https://oag.ca.gov/sites/all/files/agweb/pdfs/privacy/oal-sub-final-text-of-regs.pdf.
  • Whitney Quesenbery. 2004. Balancing the 5Es: Usability. Cutter IT Journal (February 2004). http://whitneyquesenbery.com/articles/5es-citj0204.pdf.
  • Maksim Say-fo. 2020. Buttons Alignment Policy. https://uxplanet.org/buttons-alignment-policy-a26de4ce0c70.
  • Florian Schaub and Lorrie Faith Cranor. 2020. Usable and Useful Privacy Interfaces. In An Introduction to Privacy for Technology Professionals, Travis Breaux (Ed.). IAPP, 176–299.
  • Christopher Schepelle. 2020. Despite GDPR: Up to 70% Analytics Opt-in rates – Why Extensive Testing is Worth Every Minute of Effort. https://analytics.dhl.com/despite-gdpr-up-to-70-analytics-opt-in-rates-why-extensive-testing-is-worth-ervery-minute-of-effort/.
  • Ulrich Schmidt and Horst Zank. 2005. What is Loss Aversion?Journal of Risk and Uncertainty 30, 2 (2005), 157–167.
  • Mark Scott and Laurens Cerulus. 2018. Europe's New Data Protection Rules Export Privacy Standards Worldwide. Politico (January 2018). https://www.politico.eu/article/europe-data-protection-privacy-standards-gdpr-general-protection-data-regulation/.
  • Than Htut Soe, Oda Elise Nordberg, Frode Guribye, and Marija Slavkovik. 2020. Circumvention by Design - Dark Patterns in Cookie Consent for Online News Outlets. In Proceedings of the Nordic Conference on Human-Computer Interaction (NordiCHI).
  • UK Information Commissioner's Office. 2021. ICO to Call on G7 Countries to Tackle Cookie Pop-ups Challenge. https://ico.org.uk/about-the-ico/news-and-events/news-and-blogs/2021/09/ico-to-call-on-g7-countries-to-tackle-cookie-pop-ups-challenge/.
  • United States Census Bureau. 2020. 2020 Demographic Analysis Estimates Press Kit. https://www.census.gov/newsroom/press-kits/2020/2020-demographic-analysis.html.
  • Christine Utz, Martin Degeling, Sascha Fahl, Florian Schaub, and Thorsten Holz. 2019. (Un)informed Consent: Studying GDPR Consent Notices in the Field. In Proceedings of the Conference on Computer and Communications Security (CCS). ACM, 973–990.
  • W3C Web Accessibility Initiative. 2018. Web Content Accessibility Guidelines (WCAG) 2.1. https://www.w3.org/TR/WCAG21/.
  • W3C Working Group. 2019. Tracking Preference Expression (DNT). https://www.w3.org/TR/tracking-dnt/.
  • Chauncey Wilson. 2013. User Interface Inspection Methods: A User-Centered Design Method. Newnes.

FOOTNOTE

1We did not observe consent interfaces on all websites in our compiled dataset likely due to websites making domain requests to use other services provided by the five CMP companies, or because our scraper was run with a US IP address.

2The database of our observations is available at https://airtable.com/shrnbTJ0ZIPl9OMm6.

3This particular consent interface was on friday-ad.co.uk and implemented through Quantcast.

4This particular consent interface was on sketchup.com and implemented through OneTrust.

5An example of such a consent interface is on acm.org implemented through CookieBot.

6Bitnami.com's website has an example of such an interface.

7Due to the same technical issue with Useberry, another 342 Prolific workers attempted to participate in our study but were unable to complete it. These participants were compensated $1.00 for their time.

CC-BY license image
This work is licensed under a Creative Commons Attribution International 4.0 License.

CHI '22, April 29–May 05, 2022, New Orleans, LA, USA

© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9157-3/22/04.
DOI: https://doi.org/10.1145/3491102.3501985