Brief Title: Project 4 Example: Mixed Hybrid Checklist

This experiment is one member of a 'bundle' of BESH experiments that were registered as part of the grant application process. It is not possible or meaningful to register a bundle of experiments in a single CT.gov filing so "Mixed Hybrid Checklist" is being registered as an example of this bundle.

This is the text of the pre-registration of Mixed Hybrid Checklist that was originally placed on the Open Science Framework: https://osf.io/j6h4s

NOTE: Since the pronoun comment is advisory, we will leave this for later. The website is, indeed, a functioning OSF page.

========

What's the main question being asked or hypothesis being tested in this study?

In real world search (and, notably, in radiology), it can be important to keep "an eye out" for items of interest that are not the primary subject of the search. In radiology these are known as "incidental findings," and they are a significant issue because clinicians are expected to detect items of clinical importance even they are answering a different question (e.g. you need to detect pneumonia even if your task was to screen for lung cancer). We have developed a "model system," in the form of a 'mixed hybrid search' task that can be used to investigate the fundamental cognitive processes that lie behind incidental finding errors. In this mixed hybrid search task, observers search for any one of multiple specific targets (e.g., this rabbit, this truck, and this spoon) and multiple categorical targets (e.g. any masks, furniture, and plants). In the all of the 'mixed hybrid search' experiments conducted thus far, the categorical targets are missed at a much higher rate than the specific targets. The categorical targets, thus, become the analog of the incidental error and our goal is to reduce those errors in our model system in the hope that this will provide guidance as to how to reduce errors in real-world tasks.

We have tested two strategies intended to reduce the number of missed categorical targets. In Experiment 5, we reminded observers about the categorical targets via non-search trials in which categorical targets were clearly marked. In Experiment 6, we forced observers to respond twice on each trial: Once to confirm the presence or absence of specific targets, and once to confirm the presence or absence of categorical targets. Both of these interventions failed.

This 'Hybrid Checklist' experiment is another attempt to develop a strategy that helps us reduce these categorical errors. Like the previous experiments, observers begin by memorizing three specific targets and three categorical targets. After viewing the targets, observers must pass an old/new recognition test with 100% correct responses to move to the search phase. Each memory test consists of 2×N items where N is the memory set size, and 50% of the items are old (targets) and 50% are new. Having passed the memory test, observers search for their targets in displays with 4, 8, or 12 items. Observers must indicate whether each target in the memory set is present or absent in the current display by clicking with a mouse. There are 6 targets (3 specific, 3 categorical), so observers make 6 responses on each trial. Error and reaction time will be recorded for each click, and for each trial.

Hypothesis (H1):

• We hypothesize that this checklist procedure will improve accuracy and reduce miss errors. However, given that there are 6 responses on each trial, the time per trial (or time per case for radiologists) is likely to increase significantly.

Describe the key dependent variable(s) specifying how they will be measured.

Miss errors and reaction times are the key dependent variables. These will be averaged across trials of the same condition (specific target, categorical target, and absent trials) and visual set size (4, 8, 12). Note that these measures will be compared to results from previous experiments rather than a within-observer design.

How many and which conditions will participants be assigned to?

In this experiment, participants are assigned to a single, 'checklist' condition. All observers memorize a set of three specific and three categorical targets. They are then required to pass a memory test before moving on to perform 20 practice trials and 300 experimental trials of the visual search task. Of these 300 trials:

70% are target-present trials 45% of target-present trials will have one target

• 80% of one-target trials have one specific target
• 20% of one-target trials have one categorical target 25% of target-present trials will have two targets
• 65% of two-target trials have two specific targets
• 5% of two-target trials have two categorical targets
• 30% of two-target trials have one specific and one categorical target The remaining 30% of trials will have no targets

Specify exactly which analyses you will conduct to examine the main question/hypothesis.

Reaction time and error rate will be compared to those in the previous mixed hybrid search experiment - a between-subjects design (Wolfe, Soce, and Schill, 2017). Analyses will be performed on both reaction time and miss error rate. For reaction time, a repeated measures ANOVA will be run with two factors: condition (specific target, categorical target, and absent) and visual set size. For the miss error rate, statistical analysis of errors will be performed on arcsine transformed error rates (Hogg \& Craig, 1995).

Any secondary analyses?

We will explore the data for other patterns of interest: E.g.

• Order effects: Does a categorical target on trial N influence the probability of detecting such a target on N+1.

How many observations will be collected or what will determine sample size? No need to justify decision, but be precise about exactly how the number will be determined.

To be consistent with the previous 'Mixed Hybrid Search' experiments, we will collect data from 12 observers. Participants are excluded if they are not between the ages of 18 and 55, if their visual acuity is worse than 20/25, or if they do not pass the Ishihara Color Test.

Anything else you would like to pre-register? (e.g., data exclusions, variables collected for exploratory purposes, unusual analyses planned?)

N/A