This is a pre-print HTML author version of the paper. A PDF author version is available. Please cite the work as:

Sandy J.J. Gould, Duncan P. Brumby, and Anna L. Cox. 2024. ChatTL;DR – You Really Ought to Check What the LLM Said on Your Behalf. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24), May 11–16, 2024, Honolulu, HI, USA. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3613905.3644062

Abstract

Interactive large language models (LLMs) are so hot right now, and are probably going to be hot for a while. There are lots of problems exciting challenges created by mass use of LLMs. These include the reinscription of biases, ‘hallucinations’, and bomb-making instructions. Our concern here is more prosaic: assuming that in the near term it’s just not machines talking to machines all the way down, how do we get people to check the output of LLMs before they copy and paste it to friends, colleagues, course tutors? We propose borrowing an innovation from the crowdsourcing literature: attention checks. These checks (e.g., “Ignore the instruction in the next question and write parsnips as the answer.”) are inserted into tasks to weed-out inattentive workers who are often paid a pittance while they try to do a dozen things at the same time. We propose ChatTL;DR¹, an interactive LLM that inserts attention checks into its outputs. We believe that, given the nature of these checks, the certain, catastrophic consequences of failing them will ensure that users carefully examine all LLM outputs before they use them.

Introduction

Do you know how many papers on Large Language Models (LLMs) were submitted to the Papers track of CHI 2024? We don’t. But you know it will be a lot. A lot². We wanted to experience CHI à la mode, so we have submitted this paper for the consideration of alt.chi reviewers. We are sufficiently confident of our contribution to have put our names on the paper and put it forward for public scrutiny³.

Anyway. Have you received correspondence from someone that started with, for some reason, a précis of what you’d sent them, followed by three verbose paragraphs of something that felt like a simulacrum of response? Did it contain ‘[insert company name]’ or ‘[insert your name]’ or something like that? If so, you’ve been ChatGPT’d – someone has asked ChatGPT to reply to you, but they’ve not even had the common courtesy to check it before they sent it⁴. Of course, this is normally only going to reach the level of mildly irritating, but what if an inattentive miscommunication has a catastrophic result, like denying Luis Díaz a crucial leveller against Spurs (Fisher and MacInnes 2023)? OpenAI are not oblivious to the potential for erroneous output: ChatGPT comes with a warning, which is shown in Figure 1.

Fig. 1. ChatGPT includes a subtle warning about the fact that it can make mistakes, underneath the text field. The warning seems to have hallucinations in mind, rather than straight-up errors and reads: ChatGPT can make mistakes. Consider checking important information’’}. Also, some of us have used ChatGPT quite a bit without ever realising this was even there – so we should probably be sceptical about this having any potential to influence user behaviour. This image isn’t a picture of the warning, because dealing with fair use doctrine seemed too onerous. Here’s a picture of the warning on the OpenAI forums, though: http://tinyurl.com/t6vkk533

GhatGPT includes a subtle warning about the fact that it can make mistakes, underneath the text field. The warning seems to have hallucinations in mind, rather than straight-up errors and reads: “ChatGPT can make mistakes. Consider checking important information”. Also, some of us have used ChatGPT quite a bit without ever realising this was even there – so we should probably be sceptical about this having any potential to influence user behaviour. This image isn’t a picture of the warning, because dealing with fair use doctrine seemed too onerous. Here’s a picture of the warning on the OpenAI forums, though: http://tinyurl.com/t6vkk533

The problem with all of this stuff is that it can have irreversible consequences. As Rossmy et al. (Rossmy et al. 2023) put it so elegantly in their CHI 2023 paper, we are so used to being able to hit undo to and have our foolishness be erased, we become inured to it. We end up forgetting we’re not just trying to remotely control the teeny tiny switches inside a computer somewhere at the end of the line. We forget there are real consequences that arise from the stuff we do on computers. This is partly because of the immateriality of working with computers, but it’s also because habituation is part of the human cognitive architecture and complacency is part of the human condition.

What can we do about this in the context of LLMs? There already lots of work happening (e.g. (Lehmann 2023; Dang et al. 2023)) to understand how LLM interfaces influence the behaviour of their users. But satisficing –getting as much return for as little effort when completing tasks– is a really fundamental cognitive strategy (Gray and Fu 2001) rather than a particular affordance of interactive LLMs. Therefore, it’s worth considering how people have approached the problem of human beings in other contexts. Let’s have a look at a few.

attention checks and Instructional Manipulation Checks

Crowdsourcing platforms invite remote, distributed workers to complete small piecework tasks for money (Alkhatib, Bernstein, and Levi 2017). Folks working on these platforms are busy completing multiple tasks (Gould, Cox, and Brumby 2016) while they also undertake the ‘metawork’ required to find new work (Toxtli, Suri, and Savage 2021). Given that these workers are poorly paid and have an imperative to work as quickly as possible, they are often rushing. This, appallingly, can lead to aboidable errors in their wrok. One solution to the fallibility of the folk trying to eke out a living on these platforms is to put tricksy questions or steps into tasks, so that you can see who is paying enough attention and who isn’t (Abbey and Meloy 2017; Kapelner and Chandler 2010). These are called attention checks.

In the psychology literature, attention checks are known as Instructional Manipulation Checks (IMCs) (Oppenheimer, Meyvis, and Davidenko 2009). The idea is so similar as to be indistinguishable: where your experiment relies on instructing some participants to, say, prioritise speed over accuracy and other participants to prioritise accuracy over speed, you need to know that participants have read and processed these instructions. Otherwise, your experiment will be fatally lacking in internal validity. So you stick an attention check somewhere in your experiment and if folks fail it, you can assume they didn’t read your instructions to go faster or go slower and drop their data.

Whether attention checks ‘work’ depends on your definition of ‘work’. They can certainly be effective for moving risk from those requesting work onto those doing it (though AI bots are able to pass quite a few of the common types these days (Pei et al. 2020)). If you ‘fail’ an attention check, either because you were not paying attention, because it was asking you to do something weird to which you didn’t know how to respond, or simply because you made a sensorimotor error because you’re human, then you don’t get paid for your work. So they can work, in a way, for those ‘downstream’ of errors, but don’t do much for the person making them.

There is some evidence that attention checks and IMCs cause participants to realise that the experimenter is attempting to monitor their attentiveness, and that this increases ‘systematic’ thinking – i.e., it makes them think harder (Hauser and Schwarz 2015). However, there is also evidence that people subjected to them can learn to detect them and complete them more effectively (Hauser and Schwarz 2016), so habituation (along with the bots) is certainly something that needs to be considered if one decides to make use of these kinds of interventions.

Lockouts

If attention checks provide a post hoc way of seeing if someone has been inattentive, can we be a bit more proactive and catch the proverbial before it hits the fan? Perhaps give folks a chance to see they’ve made a mess of something before they’re committed and can no longer undo? One that could work ex ante? One solution might be lockouts that encourage people to take a moment to consider whether they are ready to proceed and plan their next step (O’Hara and Payne 1999).

We have studied lockouts ourselves over the years (Brumby and Seyedi 2012; Back, Brumby, and Cox 2010; Gould, Cox, and Brumby 2015). In our CHI 2016 paper (Gould et al. 2016), we discussed the Welwyn Winder. This was a device was part of the signal box situated on part of a railway line that was particularly risky from a signaller’s perspective. Signallers had to wind the box using a handle for a set amount of time. The winding achieved nothing at all in terms of setting the signals. Its only purpose was to require the signaller to do something that took an extended period of time. They could use this time to consider whether what they were about to do was a grand idea. The idea is that you are prevented from doing something foolish while you’re on autopilot (or System 1, as psychologists call it).

Stopping people from doing the thing they want to do sounds like a great way to get them to undertake calm, deliberative thought, doesn’t it? The challenge of implementing lockouts –beyond the fact that they make people want to destroy their device and then seek you out in order to exact retribution– is that their period has to be precisely calibrated. Too short, and people stick in the autopilot and watch the pretty colours for a moment. Too long and they will simply go and do something else, rather than thinking about the step they are going to take. Nevertheless, there is something to be said for trying to help people prevent cock-ups before they happen, rather than simply berating (or not paying) them after the horse has bolted, after the egg has been scrambled, or after the ship has sailed. No can undo.

That thing in your car that erroneously believes (‘believes’) you’re drifting out of your lane and tries to steer for you

We see interventions to make people pay attention everywhere. In the UK in 2022, 1,695 people were killed on roads⁵. As Volkswagen, which manufactured nearly nine million road vehicles in the same year⁶ sagely notes, “on monotonous journeys, risky situations can arise.”⁷ Their solution, ‘Lane Assist’ is, they state, able to interpret the intent of a driver (!) It will actively take control of the steering wheel to stop you drifting out of your lane (or it’ll get confused and fail to recognise your ‘intent’ or that you’re leaving a lane).

These lane-control features are a ‘just-in-time’ solution for inattentiveness, which is different to how attention checks (largely post hoc interventions) or lockouts (ex ante interventions before ‘point of no undo’) work. This kind of intervention has the advantage that it occurs at the moment of inattentiveness, rather than before errors are committed (like lockouts, which unnecessarily interfere with ‘normal’ performance) or after they are committed and nothing can be corrected (like attention checks, which move costs around, but don’t necessarily fix the issue).

Of course, such a just-in-time system needs context awareness in order to work out ‘time’ so that it can ‘just-in’ it. Being able to detect intent feels obviously impossible, unless you think pre-crime is a thing. Even if we leave aside the question of intent, making sense of environments is a huge challenge. There’s a reason why we’ve been about to get fully autonomous vehicles any minute now for several million minutes. If you have no real idea what you’re doing on a day-to-day basis, or why you’re doing it, how can you reasonably expect a machine to step in and help you out?