Steven Penrod is a professor of psychology at John Jay College of Criminal Justice, City University of New York. He has written a multitude of publications exploring the nexus between social psychology and the law.

As of January 2003, the Cardozo Law School Innocence Project’s website reported that 123 individuals who had been convicted of crimes had subsequently been exonerated because of DNA testing. (See www.innocenceproject.org.) Of the first 82 exonerations, 60 of them—or nearly 75 percent—involved mistaken eyewitness identification. This will surprise no one familiar with the research on erroneous convictions generally or on eyewitness reliability more specifically. Studies on erroneous conviction date back 70 years, and the 1987 conclusions of Huff confirm much of the research that preceded him. (See C. Ronald Huff, Wrongful Conviction: Societal Tolerance of Injustice, 4 Res. in Soc. Probs. & Pub. Pol’y 99 (1987).) Based on 500 cases of erroneous conviction, he observed that the leading cause of mistaken conviction was faulty eyewitness identification of defendants.

Convictions arising from mistaken identifications have prompted research psychologists to explore a wide variety of factors that may contribute to mistaken identifications and convictions, including characteristics of witnesses, crimes, and police procedures that can influence identification accuracy. The research has also looked at jury decision making in eyewitness cases and examined the impact of eyewitness testimony and witness confidence, the efficacy of traditional legal safeguards such as cross-examination, and even the effect of expert testimony about eyewitness research on the quality of jury decisions.

Rather than summarize this vast body of eyewitness research, I instead focus on police identification procedures and attempt to combine data from a number of recent research papers in order to offer tentative answers to several questions that have not been adequately addressed in prior inquiries, including:

1. How often do witnesses make mistaken identifications of innocent suspects?

2. How often do the witnesses merely guess?

3. How effective are the police in assembling lineups that contain perpetrators rather than innocent suspects?

4. How accurate are the witnesses, i.e., how effectively do they identify perpetrators and how often do they correctly reject lineups that do not contain perpetrators?

5. How often are witnesses merely guessing when they identify suspects?

Answers to these questions may well raise further queries:

1. Can the number of mistaken identifications be reduced?

2. Are police and prosecutors employing identification procedures that ensure the reliability of eyewitness identifications?

3. Are eyewitness identifications, as they are currently collected in the United States, of sufficiently dubious reliability that should not be admitted into evidence as part of criminal prosecutions?

4. Should procedures that research by psychologists has shown can significantly enhance the reliability of eyewitness identifications be mandated for the police?

5. Should eyewitness identifications not be admitted as evidence if police and prosecutors are unable to demonstrate that they employ identification procedures that ensure the reliability of eyewitness identifications?

6. Should jurors be informed by experts about weaknesses in the methods of lineup construction and presentation, and the tendencies of eyewitnesses to guess and make errors?

I focus on the first group of questions here, but in pondering the full set of questions, it is useful to consider—though this is rarely done—that eyewitness evidence is actually very similar to other forms of forensic evidence collected, maintained, and tested by the police. That is, eyewitness evidence is not far different from fingerprint evidence, bloodstains, DNA, and the like. As with those techniques, it is logically appropriate to hold police and their identification procedures to standards that maximize the reliability and probative value of eyewitness evidence. Although eyewitness identification evidence traditionally has been admissible in the United States, courts also have been willing to scrutinize identification procedures to ensure minimum levels of fairness. In fact, one can argue that courts—and police and prosecutors—can do more.

Let us begin our inquiry by asking what is known about the accuracy of eyewitness identifications and the effectiveness of current police procedures. With respect to the former, two primary sources of information exist: (1) experimental studies of eyewitness performance in staged crime and other situations, and (2) archival studies of eyewitness performance in actual cases.

Three noteworthy features of this research are evident. First, virtually all studies have been conducted by research psychologists in academic settings rather than by police or by prosecutors. Indeed, very little has even been funded by agencies that have traditionally supported police and prosecution-oriented research.

Second, we must recognize that prosecutors clearly rely upon the products of identification procedures conducted by the police. In theory, then, prosecutors ought to know something about, and be in a position to demonstrate, the reliability of the identification evidence these procedures generate. They do not, and are not. Causes for this are numerous, but one item to note is that the relevant empirical work has not been fully completed. Indeed, archival research indicates that police records are so incomplete that it is difficult and often impossible to extract information essential to assessments of eyewitness reliability. Frankly, if eyewitness identification was now being offered to courts for the first time, and prosecutors had to prove the reliability of that evidence—just as they might have to prove the reliability of some new forensic test—they would be hard-pressed to demonstrate its reliability successfully.

Finally, and most importantly, archival and experimental research conducted by psychologists raises serious questions about the reliability of witness performance.

Experimental studies. Experimental research has some important advantages over archival research. For one thing, experimental researchers know when witnesses have made correct and incorrect judgments. Where research psychologists stage crimes or crime-like events, or expose witnesses to videotapes of such situations, they are able to assemble photo arrays or lineups in which they know whether the perpetrator (or "target") is present or absent. This makes it possible to assess the likelihood that witnesses will correctly identify targets when present. In contrast, archival research is plagued by the impossibility of authoritatively determining, in most instances, whether identifications are accurate or not. Further, experimental researchers can systematically explore the impact of a wide variety of witness and witnessing factors, identification procedures, and other factors on witness performance. Though it is sometimes possible to examine the relationships among some of these factors and actual witness behavior performance in archival studies, the lack of experimental control in archival data makes it difficult to reach firm causal conclusions about any observed relationships.

So, how well do witnesses in experiments perform? Well, it depends, particularly on the nature and difficulty of the identification circumstances created by the researchers. One might therefore want to closely review studies that bear a strong resemblance to actual crime situations. In the 1995, book Mistaken Identifications: The Eyewitness, Psychology, and Law, Brian Cutler and I examined witness performance in field experiments by Brigham et al. in 1982, Krafka and Penrod in 1985, Platz and Hosch in 1988, Maass and Kohnken in 1989, and Pigott et al. in 1990, all of which approximated conditions that one might observe with actual nonviolent crimes. (See John C. Brigham et al., Accuracy of Eyewitness Identifications in a Field Setting, 42 J. Personality & Soc. Psychol. 673 (1982); Carol Krafka & Steven Penrod, Reinstatement of Context in a Field Experiment on Eyewitness Identification, 49 J. Personality & Soc. Psychol. 58 (1985); Stephanie J. Platz & Harmon M. Hosch, Cross Racial/Ethnic Eyewitness Identification: A Field Study, 18 J. Applied Soc. Psychol. 972 (1988); Anne Maass & Guenther Kohnken, Eyewitness Identification: Simulating the "Weapon Effect," 13 L. & Hum. Behav. 397 (1989); Melissa A. Pigott et al., A Field Study of the Relationship between Quality of Eyewitnesses’ Descriptions and Identification Accuracy, 17 J. Police Sci. & Admin. 84 (1990).) Across those studies, with more than 500 identification attempts, witnesses made correct identifications in slightly less than 42 percent of target-present arrays and made false identifications in nearly 36 percent of the target-absent arrays.

Though that last number looks appallingly high, we must remember that these studies use identification procedures in which most lineup members—what researchers call "foils"—are known to be innocent. So do the police in real life. Thus, if those 36 percent false identifications are distributed equally across the foils and the suspect, the rate of mistaken suspect identifications will be much smaller. Unfortunately, as we shall see, it is very unlikely that the choices are distributed equally, so it is premature to breathe too deep a sigh of relief.

Researchers can manipulate features of their studies in ways that permit them to push witness performance up or down with relative ease, but the identification rates in these field experiments are well within the range of choosing rates reported in archival studies of witnesses. Similar patterns prevail even in experimental studies that lack some of the features of realism characteristic of the studies on which Cutler and I focused. For example, Ebbesen and Flowe found that in 114 target-present arrays (arrays in which the "perpetrator" was included and all members of the array were presented simultaneously), witnesses made a correct identification 49 percent of the time. (Ebbe E. Ebbesen & Heather D. Flowe, Simultaneous v. Sequential Lineups: What Do We Really Know? (unpublished manuscript 2001).) (Unfortunately, Ebbesen and Flowe do not report mistaken identification rates.) In 84 target-absent arrays, witnesses incorrectly identified someone in the array 49 percent of the time.

In 2001, Steblay et al. reported the results of a meta-analysis comparing witness performance on arrays presented simultaneously and those presented sequentially (i.e., arrays in which faces are presented one at a time and the witness is asked whether they recognize a face before moving on to the next face). (Nancy M. Steblay et al., Eyewitness Accuracy Rates in Sequential and Simultaneous Lineup Presentations: A Meta-analytic Comparison, 25 L. & Hum. Behav. 459 (2001).) Overall, the authors note that, across 22 studies, the correct identification rate for traditional simultaneous photo target-present arrays was 50 percent, the mistaken identification rate of foils in target-present simultaneous arrays (reported in 12 studies) was 24 percent, and the no-identification rate (reported in 13 studies) was 26 percent. The reason the number of studies varies across outcomes is that Steblay et al. were drawing on all the data available and studies varied in procedures and types of data reported.

Most of these studies also looked at witness performance in target-absent arrays. Witnesses in these studies are randomly assigned to target-present and target-absent conditions to ensure that the witnesses in the two conditions are not different from one another. It is particularly interesting to examine choosing behaviors of witnesses who might have made a correct identification in a target-present lineup (50 percent of witnesses in Steblay et al.) when the target is replaced with an innocent foil. Ideally, of course, one would like to think that they would all reject the lineup, but that is not what happens. In 24 studies with target-absent arrays, Steblay et al. report that the correct rejection rate was 49 percent (with a 51 percent false identification rate across 25 studies). There are two notable features to these results. First, it is clear that a significant percentage—essentially half—of witnesses who might identify the perpetrator/target if present do not say "not there" when the target is not there, but instead choose one of the foils. This fact should prompt us to think that a significant percentage of perpetrator/target identifications are little better than guesses—otherwise witnesses would reject the lineups when the target is removed.

Second, the 51 percent false identification rate in target-absent arrays works out to an average guessing rate per foil in six-person arrays of nearly 9 percent per foil. But Steblay et al. further note that in 15 studies that examined misidentifications of foils who resembled the perpetrator—what are termed "designated innocent suspects"—the misidentification rate was 27 percent. This "bias multiple" for innocent suspects versus other foils (27 percent versus 9 percent) is on a par with the multiplier observed in studies of actual identifications, a point discussed at more length below.

These numbers are offered to make several basic points: (1) overall, these experimental studies yield correct witness decisions about half the time; (2) a significant amount of guessing goes on in these studies; (3) guessing rates are probably significantly higher in target-absent arrays than in target-present arrays because many witnesses who might recognize the target/perpetrator, if present, nonetheless choose someone else if the perpetrator is absent; and (4) there are strong indications that guessers "identify" suspects at much higher rates than they choose foils.

Archival studies. Several important studies of eyewitness identification behavior in actual cases exist, including Slater in 1994, Tollestrup et al. in 1994, Wright and McDaid in 1996, and Behrman and Davey in 2001. (A. Slater, Identification Parades: A Scientific Evaluation. (Police Research Award Scheme, Police Research Group, Home Office, 1994) (reported in Tim Valentine & Pamela Heaton, An Evaluation of the Fairness of Police Line-Ups and Video Identifications, 13 Applied Cognitive Psychol. S59 (1999)); Patricia A Tollestrup et al., Actual Victims and Witnesses to Robbery and Fraud: An Archival Analysis, in Adult Eyewitness Testimony: Current Trends and Developments 144 (David E. Ross, J. Don Read & Michael P. Toglia eds. 1994); Daniel B. Wright & Anne T. McDaid, Comparing System and Estimator Variables Using Data from Real Line-Ups, 10 Applied Cognitive Psychol. 75 (1996); Bruce W. Behrman & Sherrie L. Davey, Eyewitness Identification in Actual Criminal Cases: An Archival Analysis, 25 L. & Hum. Behav. 475 (2001).)

Slater examined identification attempts by 843 British witnesses who viewed 302 suspects. Slater found that foils were identified by 190 witnesses (22.5 percent). Wright and McDaid examined identification attempts of 1,561 British witnesses who viewed 616 suspects in live lineups and found that 611 witnesses (39.1 percent) picked the suspect, 310 witnesses picked a known-innocent foil (19.9 percent), and 640 (41 percent) made no identification. In short, one in three identifications was of an innocent foil (310/(310 + 616)). It is likely that witnesses in both these studies were confronted with nine-person arrays, the British standard. Tollestrup et al. report positive identifications of suspects in 32 percent of the 167 Vancouver photo arrays (most with eight photographs) they studied. The police records did not permit the researchers to differentiate nonidentifications from foil identifications.

The most extensive American report is the 2001 study by Behrman and Davey, who examined witness behavior in 271 actual police cases in Sacramento, California, involving 284 photographic lineups, 258 field show-ups, 58 live lineups, and 66 lineup identifications preceded by earlier identifications. The 58 live lineups supply the most relevant information because the researchers were able to ascertain the rates at which witnesses selected suspects and foils in the lineups. In those lineups, 50 percent of witnesses selected the suspect, 24 percent selected a foil, and 26 percent made no choice. In short, as in the Wright and McDaid study, about one in three positive identifications (24 percent/(24 percent + 50 percent)) was of an innocent foil. In the 284 photographic arrays examined by Behrman and Davey, 48 percent of witnesses identified the suspect, but the records did not permit calculation of the rate of foil identifications.

I believe the Behrman and Davey study can offer us very useful insights into police and witness performance in lineups. However, there are limits to those insights. We must first note that the errors reported by Behrman and Davey are not errors that give rise to erroneous convictions. Foil identifications do not give rise to criminal prosecutions. Nonetheless, there are likely to be worrisome errors—identifications of innocent suspects—concealed in the Behrman and Davey’s numbers, and in the following paragraphs I make an effort to ferret out those numbers. Second, we must recognize that these are archival data, and it is therefore impossible to establish authoritatively the proportions of suspect identifications that are perpetrator identifications as opposed to misidentifications of innocent parties. This means that we will be forced to make some assumptions about the composition of the lineups presented to the witnesses in the Behrman and Davey study. It will be useful to formulate these assumptions in light of other analyses reported by Behrman and Davey and in light of other research.

Table 1 encompasses the full range of decisions made by the witnesses studied by Behrman and Davey, though those researchers do not fill in the blanks in that table, nor did they attempt to do so. Ultimately, what I want to do is fill in Table 1 with some plausible numbers. The numbers already indicated in the table are those generated by the eyewitnesses in the study. Other numbers—indicated by letters for ease of reference—will have to be generated by inference, assumption, and testing. For purposes of this analysis, I wish to differentiate witnesses who are guessing from witnesses who are making memory-based identifications of a perpetrator (cell f) or memory-based rejections of lineups (cell a). These witnesses know what the perpetrator looks like, and they know whether he or she is in the lineup or not. In this logic, there are no memory-based "identifications" of innocent suspects (cell i). Rather, those mistaken identifications of suspects are mere guesses. These are bad or harmful guesses insofar as they can give rise to erroneous prosecutions and convictions. This cell is the source of the Innocence Project exculpations, and is one of the most critical and interesting cells in Table 1. If the witnesses who identify innocent suspects really remembered what the perpetrator looked like, they would have rejected the lineup (cell a).

In fact, as Table 1 makes clear, the Behrman and Davey data include other types of guesses as well: (1) the 24 percent of witnesses who guessed foils (cells k and l), guesses that are not harmful to the foils but carry other costs, noted below; (2) a portion of the suspect identifications that were "lucky" guesses of perpetrators (cell h), good luck in the sense that the witnesses guessed a guilty person; and (3) witnesses with poor memories who lacked the temerity to make an identification. That is, they end up in cells c and d rather than cells h, i, k, or l.

As noted, filling in these cells will require at least some starting assumptions. Some crude starting assumptions do not work logically. For example, consider cells m and n. A tempting first assumption is that the Sacramento police have been extremely efficient and managed to place 58 actually guilty suspects in arrays. That assumption would lead us to conclude m = 58 and n = 0, and that 50 percent of the Behrman and Davey witnesses are in error: the 24 percent who selected a foil plus the 26 percent who failed to identify the suspect/perpetrator. These witness error rates are disturbingly high insofar as a very large proportion of guilty perpetrators escape detection and prosecution. However, under this assumption of high police effectiveness, no harm to innocent suspects occurs (cell i). The trouble, of course, is that we know from archival research on mistaken convictions and from the Innocence Project results that the assumption that all suspects are guilty is simply not viable. Cell i is likely to have some entries.

Instead, we must move to a potentially more reasonable second assumption that the Sacramento police have assembled a mixture of perpetrators and innocent suspects for witnesses to scrutinize. Let us further assume that the witnesses are performing as well as possible given that they are confronted with a mixture of guilty and innocent suspects—that is, the witness decisions appear only in cell a (correct rejections of suspects when the suspect is innocent), cell f (correct identifications of guilty suspects), and cell k—the foil identifications reported by Behrman and Davey. In this hypothetical, there would be no entries for cell l because we are assuming the best.

Under the best of circumstances, the Behrman and Davey witnesses might thus make 50 percent correct perpetrator identification (cell f) and 26 percent correct rejections of perpetrator-absent array (cell a). However, 24 percent of the witnesses clearly still make an error. These are the witnesses who really did not remember much about the perpetrator but nonetheless took a guess and erroneously selected a foil. Under this second set of assumptions, the eyewitness errors also (and unrealistically) do no harm to innocent suspects because the foils are not erroneously prosecuted. But several other harms are done. First, time and effort are wasted insofar as the police did not need to "rule out" the foils. Second, even if we assume the suspect identifications and lineup rejections are all correct, the police are still confronted with ambiguity about the implications of the witness foil identifications: is the suspect guilty or not? Third, the credibility of the witnesses who chose foils is impugned and will be impugned in court should they later identify a suspect from another identification parade. Finally, given that witnesses are error prone, the police might be upset insofar as their lineups contain innocent suspects and should be more upset as the proportion of innocent suspects increases (which translates to more wasted effort) and witness errors increase (which translates to more missed prosecutions).

But what about empty cell i? Of course, we know there are erroneous prosecutions, and in the Behrman and Davey numbers they would have to arise from the 50 percent suspect identifications. How many of those identifications are identifications of innocent (cell l) as opposed to guilty suspects (cell k)? The answer to that question is a product of two, not well-established, numbers, namely, (1) the rates at which witnesses make "identifications" of suspects based on guesses (cells h and i), as opposed to reliable memory-based identifications (cell f), and (2) the mixture of guilty and innocent suspects who are presented to witnesses (cells m and n). Let us start with the first of those numbers.

Guessing the suspect. How might the 29 positive identifications break down into cells f, h, and i? We can actually make some good estimates of these numbers based on the Behrman and Davey results in combination with other archival research. Consider the fact that the Behrman and Davey witnesses were confronted with six-person lineups and that the 24 percent of witnesses who chose a foil spread their guesses over five foils. This indicates that each foil is, on average, generating guesses from about 5 percent of witnesses. Of course, there is every reason to think that at least an equal percentage of witnesses guessed/chose the suspect. If so, at least 5 percent of the 50 percent of witnesses who identified a suspect also merely guessed and happened by chance to guess the suspect. Thus, it is likely that at least 10 percent of all suspect "identifications" are the product of mere guesses. If all these witnesses guessed wrong and picked an innocent suspect, then 10 percent of the identifications would be identifications of innocent individuals, about three of the 29 identifications made by the witnesses Behrman and Davey studied. However, if the witnesses were merely guessing, they would not always be guessing innocent suspects; some of their guesses would be bad guesses (innocent suspects—cell i) and some lucky guesses (perpetrators—cell h). The relative proportions would depend on the mixture of guilty and innocent suspects these guessers encounter.

Is the 5 percent suspect-guessing rate viable? In fact, we have good reason to believe that the rate at which suspects are identified as a result of guessing is actually higher, and probably much higher, than the 5 percent guesses accorded to the average foil. As noted above, Steblay et al. reported that individuals designated as innocent stand-ins for perpetrators in experimental studies—generally so designated because they most closely resembled the perpetrator—were misidentified approximately three times more often than other foils. Such multipliers—which really reflect the degree of bias against the suspect—are not limited to experimental studies. Researchers have examined these biases in actual arrays presented to eyewitnesses. The basic methodology in these studies—the so-called mock witness method—is to present a photograph of the array (either the photographs used in the case or a photograph of the lineup) to individuals who were not at the scene of the crime. These individuals are given the description of the perpetrator provided to the police by the witness(es) and are then asked to identify the suspect from the arrays. In a perfectly fair array, guesses should be distributed equally across the members of the array. This is not what happens with real lineups.

In 1999, Brigham et al. were able to test the proportion of mock witnesses who chose the suspect in 18 cases involving 26 witness (or composite witness) descriptions. (John C. Brigham et al., Applied Issues in the Construction and Expert Assessment of Photo Lineups, 13 Applied Cognitive Psychol. S73 (1999).) Suspects in the arrays they studied were selected an average of 2.6 times as often as one would expect by chance alone. If the same ratio prevailed in the Behrman and Davey lineups, the percentage of suspects selected as a result of guessing would be 13 percent: that is, 2.6 (the multiplier) x 5 percent (the rate at which the average nonsuspect foil was selected)—over 25 percent of all positive suspect "identifications." Wells and Bradfield, in a similar study of 10 photo arrays and lineups, found suspects selected, on average, more than 2.1 times as often as expected by chance. (Gary L. Wells & Amy L. Bradfield, Measuring the Goodness of Lineups: Parameter Estimation, Question Effects, and Limits to the Mock Witness Paradigm, 13 Applied Cognitive Psychol. S27 (1999).) Of course, the arrays in these studies were not randomly selected—they were arrays from contested identifications—so caution must certainly be exercised with such extrapolations.

In contrast, Valentine and Heaton attempted to study suspect bias using photographs of a representative set of 25 lineups and 16 video parades from records of the Yorkshire police in the United Kingdom. (Tim Valentine & Pamela Heaton, An Evaluation of the Fairness of Police Line-Ups and Video Identifications, 13 Applied Cognitive Psychol. S59 (1999).) The video parades consist of 10-second clips showing an individual’s head and shoulders as they move from full face to left profile to right profile and back to full face. The clips can be selected from a large pool of clips. In contrast to the live lineups in which individuals are shown all at once, videoclips are shown one at a time. As in other studies, the witness description was read to each participant who was instructed to pick the person in the lineup they thought was most likely to be the suspect. Participants were asked to view members of lineups at least twice before they made their decision. Participants who viewed videotapes were instructed to view the entire tape twice before making a choice. All arrays contained nine members, with an expected guessing rate of 11.1 percent per face. Witnesses selected lineup suspects 24.8 percent of the time, a rate 2.2 times chance, and video parade suspects 15.1 percent of the time, 1.4 times chance. The difference between lineups and video parades might be attributable to the methods of foil selection (perhaps more foils resembling witness descriptions are available for video presentation), the method of presentation (simultaneous versus sequential), or other factors.

In short, mock witness research indicates that lineups are generally biased against defendants. These biases are of a magnitude also reported in experimental research by Steblay et al. (Steblay et al., supra.) Assuming that Behrman and Davey’s lineups are similarly biased, we can easily compute a value for cell j of Table 1. Given that 24 percent of their witnesses identify a foil—nearly 5 percent per foil—if we used the Brigham et al. multiplier of 2.6 or the Steblay et al. multiplier of 3.0, we might estimate that 13 to 15 percent of Behrman and Davey’s witnesses are guessing the suspect. Let us say that cell j is 14 percent (8 of the 58 witnesses). This would imply that 38 percent of Behrman and Davey’s witnesses are merely guessing insofar as they are either guessing a foil (24 percent or N = 14) or are guessing the suspect (14 percent or N = 8). Thus, more than one-fourth of the 29 positive suspect identifications made by the Behrman and Davey witnesses may be nothing more than guesses. In contrast, 36 percent of witnesses might be "picking" the suspect in the sense that they are drawing upon a memory for what the perpetrator looked like (29 choices minus 8 suspect guesses equals 21 identifications, in cells f and g).

Of course, the nature of the harm done by the guesses will still depend on the proportion of suspects who are innocent. We still have to figure out how these guesses are distributed into cells h and i. Some witness guesses will produce the "lucky identifications" of perpetrators that we spoke of earlier. These "identifications" are not the product of reliable witness memories or reliable evidence. Some of us may be displeased by the prospect that guilty people are being convicted with unreliable evidence, whereas others will be pleased that perpetrators are apprehended, even if by lucky guesses. However, everyone should be concerned when innocent people are prosecuted on the basis of bad guesses.

How bad might the bad-guess situation be? That depends on police effectiveness. Specifically, it depends on the proportion of lineups that contain perpetrators rather than innocent suspects. If, in the 43 Behrman and Davey lineups from which witnesses made choices, 80 percent of the suspects were guilty, then 34 suspects were guilty and nine were innocent. Witnesses making positive identifications but merely guessing suspects at the rate of 14 percent would thus guess/identify about one innocent suspect (.14 x 9 = 1.26), or 4 percent of the 29 suspect "identifications" observed by Behrman and Davey. However, if the base rate of perpetrators were 50 percent (21.5 guilty and 21.5 innocent), then half of suspect guesses will "identify" perpetrators and half will mistakenly "identify" a truly innocent person—in this instance, about three suspect identifications (.14 x 21.5) will "identify" an innocent person—and 10 percent of all suspect "identifications" will be in error. Of course, these numbers ignore the problem alluded to above—evidence that choosing/guessing rates are much higher in target-absent lineups than in target-present lineups—a problem we take into consideration below.

The base rate of guilty perpetrators in lineups. How effective might the Sacramento police be in assembling their lineups? Do their lineups contain a high proportion of guilty suspects? Is the rate of guilty suspects in Sacramento in the low range of 50 percent or in the higher range of 80 percent—or perhaps even higher? More importantly, can we extract a reasonable estimate of the mix of guilty and nonguilty suspects from the Behrman and Davey data? In the Behrman and Davey cases, it is not entirely clear just how high the proportion of perpetrators could be, at best. The first stumbling block is that 26 percent of witnesses (N = 15) made no identifications. If all 26 percent of nonchoosers were correct, the maximum percentage of Sacramento lineups containing a suspect would be 74 percent. Note that this is one instance in which there is a direct trade-off between police and eyewitness errors: to the extent the rate of correct nonidentifying witnesses increases, the rate of perpetrator-present arrays has to go down.

Of course, it is quite unlikely that all 15 of the no-choice witnesses were correct. As we have already seen, there is good reason to think that more than one-third of the Behrman and Davey witnesses (N = 22, that is, the 14 guessing a foil plus eight guessing the suspect) were simply guessing affirmatively. Those would be the witnesses with poor memories who were nonetheless motivated to take a guess and produced a mixture of foil and suspect identifications. We surely would not be surprised to learn that some other witnesses with poor memories were not sufficiently motivated to make a guess, which accounts for cell e. These witnesses make errors when confronted by a perpetrator they fail to identify—cell c—and make fortuitously "correct" "decisions"—both terms really deserve the quote marks—when they make no choices from lineups containing innocent suspects (cell d). Such poor memory/no choice witnesses are among the 15 Behrman and Davey, witnesses who made no choice. Of course, to the extent these nonchoosers missed the perpetrator, the base rate of perpetrators in lineups could, plausibly, creep higher than 74 percent.

Although we do not know the base rate at which innocent individuals end up as suspects in identification procedures, we have some clues from archival studies such as Behrman and Davey’s, in which researchers classify cases in terms of the levels of extrinsic evidence available in case files in order to evaluate the validity of identifications. The notion is that identifications in cases with significant extrinsic evidence are more likely to be perpetrator identifications than is true in cases with little or no extrinsic evidence, and we may learn something about guilty suspect base rate from the proportion of cases in which some extrinsic supporting evidence is present. In their study, Behrman and Davey formed three categories of extrinsic evidence: none, minimal (e.g., anonymous information, co-felon information, or similar modus operandi), and substantial (e.g., suspect on surveillance tape, stolen property in suspect’s possession, physical evidence, or confessions). (Some, aware of the prosecutorial repudiation of the Central Park jogger confessions in New York City, which are in the news as this article is being written, may wish to quarrel with this latter judgment.)

The lineup cases Behrman and Davey study actually appear to be relatively high in extrinsic evidence. In comparison, two-thirds of the photographic identification cases they examined fell into the no extrinsic evidence category. In their lineup cases, the evidence was thus much stronger. Among the 43 lineup cases in which witnesses made a positive identification (29 suspect choices and 14 foil choices), 27 (63 percent) had substantial extrinsic evidence, 7 had minimal evidence (12 percent), and 9 (16 percent) had none. (Behrman and Davey unfortunately do not report the evidence breakdown for the 15 cases in which witnesses made no identifications.)

At first blush, it might be tempting to conclude that 63 percent of the positive identification cases—the 27 substantial extrinsic evidence cases—are perpetrator-present cases. We might do so with some comfort if witnesses identified the suspect in all of those cases, but the fact is that nine of those 27 identifications were identifications of foils. Though this may initially give us pause regarding the relationship between extrinsic evidence and guilt in these cases, it turns out that the foil identification rate was approximately one in three across all three evidence categories. In some ways this is not surprising. The results merely indicate that witnesses with poor memories of the perpetrator who are inclined to make guesses are equally likely to be associated with cases in which substantial, minimal, and no extrinsic evidence is developed. The fact that witnesses willing to make a guess are distributed evenly across the evidence categories means that the foil choices of such witnesses tell us nothing at all about the actual guilt of the suspects. In short, the police may have had the actual perpetrator in the 27 substantial evidence lineups, but nine of those lineups were presented to witnesses with poor memories for the perpetrator who were nonetheless willing to make a guess of a foil. (Indeed, as we have already seen, additional guessers would have selected the suspect.)

Still, is a 63 percent guilty rate plausible in light of the Behrman and Davey results? It seems a reasonable starting point, and it is possible to test this number in light of other results derived from the Behrman and Davey report and the numbers we derived from studies of suspect bias in lineups.

Putting it all together. We know that 26 percent of Behrman and Davey’s witnesses made no choice (15 witnesses) and 74 percent made a choice from their lineups (43 witnesses). Furthermore, we have estimated that about 50 percent of the witnesses who made "identifications" were guessing—14 witnesses who guessed a foil plus eight witnesses who guessed a suspect (cell j) versus 21 witnesses who identified a suspect from memory (cells f and g).

I began my effort to complete Table 1 by assuming—as is consistent with the empirical research—that both lineup rejection rates and foil guessing rates (and, of course, rates of suspect guessing) would be higher in target-absent than in target-present lineups. In the meta-analyses considered earlier, foil guessing rates and rejection rates both doubled because half of target-absent witnesses who might have identified the target from a target-present array chose a foil (including the target-substitute) and half rejected the lineup. Consistent with the meta-analytic research, I also assume the foil guessing rates and the rate at which witnesses would erroneously reject lineups are of comparable magnitude in target-present arrays.

Once those constraints were in place, I was able to generate the remainder of the table, as reported in Table 2. Surprisingly, a fit to the Behrman and Davey data was quite easy to obtain. The table reflects two minor adjustments to the starting assumptions. First, there is an upward adjustment in the guilty-suspect percentage, from 63 percent to 67 percent. Second, the rates of lineup rejections and identifications roughly double in perpetrator-absent lineups as compared to perpetrator-present lineups. The lineup rejection rate increases from 18 percent (seven of 39 witnesses) to 42 percent (eight of 19 witnesses), the rate of foil identifications increases from 18 percent (seven of 39 witnesses) to 37 percent (seven of 19 witnesses), and the suspect guessing rate increases from 10 percent (four of 39 witnesses) to 21 percent (four of 19 witnesses). The balance in rejections and foil identifications in perpetrator-present lineups is preserved. Note that the rate of guess-rejections is roughly comparable in perpetrator-present and perpetrator-absent lineups. Thus, the table assumes that the increase in the proportion of rejected lineups is driven entirely by truly correct rejections—five witnesses. This result is plausible; these are witnesses who have a good memory for the perpetrator and would have identified the perpetrator had he/she been in the lineup.

If these numbers are a fair characterization of the suspects and witnesses represented in the Behrman and Davey data, and I believe they are, they suggest the following answers to the first five questions posed at the outset of this article:

1. Witness guessing produces a modest (though worrisome) number of mistaken identifications of innocent suspects. In this instance, an estimated four of 29 "identifications" are unlucky guesses, or about 14 percent. The primary reason innocent suspects are not identified more often, despite the suspect-biased lineups incorporated into these analyses, is that two-thirds of positive identification guesses are distributed across foils, where they can do little harm. A secondary protection afforded innocent suspects arises from the fact that police performance in lineups is not abominable: two out of three suspects are guilty. If police performance improved and lineups contained a smaller percentage of innocent suspects, the proportion of guesses that result in innocent suspect identifications would similarly decline, a point explored below. Overall, the ratio of perpetrator identifications to innocent suspect identifications (a convenient way to gauge witness effectiveness) is 6.25:1 (i.e., 25:4).

2. Witnesses are arguably overeager when it comes to making guesses that seem to have no basis in memory. Thirty-two of the 58 witnesses appear to be guessing. Whether this overeagerness arises from witness internal motivations and/or is prompted by subtle (or not so subtle) police encouragement is impossible to determine from these data.

3. The Sacramento police are doing a moderately good job of presenting mostly perpetrators rather than innocent suspects to witnesses. The ratio is 2:1 for guilty versus not guilty suspects.

4. Witness performance is not terrific. Witnesses appear to be making correct judgments 64 percent of the time in perpetrator-present lineups (25 identifications from 39 witnesses), and 42 percent of witnesses in perpetrator-absent lineups (8 of 19) correctly reject the lineup (of which more than 15 percent arise from guesses).

5. A significant portion, about 16 percent (4/25), of perpetrator "identifications" comes from witnesses who appear to be merely guessing. These are the lucky guesses. If these "identifications" are ignored, the ratio of memory-based perpetrator identifications to innocent suspect identifications is 5.25:1 (21:4).

Eyewitness error rates and police, prosecutors, and courts. It should be emphasized that proponents of the courtroom use of eyewitness identifications—police, prosecutors, and even the courts that admit eyewitness identification evidence—currently have no evidence that the numbers submitted above unfairly characterize the products of current identification practices. Indeed, I know of no comparable numbers or analyses. However, even if these numbers are dead-on accurate for Sacramento during the period of the Behrman and Davey study, the numbers in other jurisdictions could be dramatically different, depending on police practices. For example, some jurisdictions may undertake lineups in cases with much less—or much more—extrinsic evidence as compared to Sacramento, or in some jurisdictions police practices may encourage more or less guessing by witnesses. Of course, the situation with respect to identifications of innocent suspects would actually be worse if the proportion of perpetrator-absent identification procedures is higher than 33 percent, and at present no evidence exists that the proportion of perpetrator-absent procedures is not higher—even vastly higher—than 33 percent. All other things being equal, as the proportion of perpetrator-absent identification tests presented to witnesses increases, the proportion of positive identifications of innocent suspects increases.

These analyses do not answer other questions that would help us to understand and estimate the frequency with which erroneous convictions arise from mistaken eyewitness identifications. Even assuming the Table 2 numbers offer an accurate characterization of mistaken suspect identification rates, we still do not know what happens to the four mistakenly identified suspects, as opposed to the 25 "correctly identified" ones. (I use quotes because four of those identifications appear to be lucky guesses.) Are mistakenly identified suspects able to clear themselves with some degree of regularity? (Recall that Behrman and Davey reported that levels of incriminating evidence were not associated with witness choosing rates.) Are charges more likely to be dropped against innocent suspects because witnesses who identify innocent suspects tend to be less confident? (Behrman and Davey do report that highly confident witnesses tend to choose foils at lower rates than witnesses with lower confidence.) And what is the impact of plea bargaining? We know that over 90 percent of defendants enter pleas; but what about innocent defendants? It seems likely that innocent defendants will enter pleas at much lower rates than guilty defendants. Indeed, it is quite plausible that plea bargaining filters out most of the guilty defendants identified by eyewitnesses with the result that judges and juries see a mix of eyewitness cases containing a much larger (perhaps vastly larger) proportion of innocent defendants than the 14 percent figure indicated in Table 2.

Even in the absence of answers to questions such as these, we can still consider whether ways exist to significantly reduce the risks faced by innocent lineup members.

Given our concern about mistaken convictions of innocent individuals, we might first ask if it is possible to get the rate of innocent suspect identifications well under 14 percent. Are there easily employed tools the police might use to reduce these errors? The answer is a resounding yes.

Use larger lineups. One of our objectives in conducting lineups is to maximize the ratio of correct to incorrect suspect identifications. An obvious first step in realizing that objective would be to increase the size of identification arrays from the traditional five or six to a much larger number. Arrays of nine—as used in Britain—would, for example, help to spread "identifications" by witnesses who are merely guessing across a larger number of known-innocent foils. For example, if the witness choosing patterns reflected in Table 2—which are based on suspect-biased six-person arrays in which suspects are 2.9 times as likely as known-innocent foils to be chosen by guessers—were applied to twelve-person arrays, the pattern of results for suspect and foil guesses would change dramatically. In essence, suspect and foil guesses would be redistributed across 12 individuals. Mathematically, we have 11 foils with an average probability of x plus a suspect with a probability of 2.9x, and if we divide the 38 percent guessing rate in Table 2 by 13.9 (the 11 + 2.9 foils across which the guesses are spread), we obtain an average foil guessing rate (x) of 2.73 and a suspect guessing rate of 2.9 x 2.73 = about 8 percent. Not surprisingly, this is almost a 50 percent reduction in the suspect guessing rate associated with six-person lineups and, if applied (with some rounding) to the numbers in Table 2, would yield a pattern of guessing shown in Table 3.

The ratio of perpetrator identifications to innocent suspect identifications would rise from 6.25:1 (25:4) for the six-person lineups to 11.5:1 (21 perpetrator identifications plus two perpetrator guesses versus two innocent suspect guesses) in 12-person lineups. Less than one in 10 identified suspects would be innocent.

Other possibilities for procedural reform that might improve the numbers in our table abound. In 1998 several colleagues and I published a "white paper" on identification procedures in which we suggested several changes in identification procedures that we believed, based on empirical research, would reduce rates of erroneous identifications and thereby improve the diagnosticity of eyewitness identifications. (Gary L. Wells et al., Eyewitness Identification Procedures: Recommendations for Lineups and Photospreads, 22 L. & Hum. Behav. 603 (1998).) Among our suggestions are the items listed below.

Use blind presentation. The person who conducts the lineup or photo spread should not be aware of which member of the lineup or photo spread is the suspect. The rationale for this procedure is that blind presentation will eliminate the possibility that police officers administering identification procedures can wittingly or unwittingly communicate something to a witness about which member of an identification parade is the suspect. This could reduce the rate of guessing by witnesses with poor memories.

Give strong cautionary instructions. Eyewitnesses should be told explicitly that the person in question might not be in the lineup or photo spread and therefore should not feel that they must make an identification. They should also be told that the person administering the lineup does not know which person is the suspect. The cautionary instruction has been shown to reduce guessing by witnesses.

Match foils to the description of the perpetrator. The suspect should not stand out in the lineup or photo spread as being different from the foils, based on the eyewitness’s previous description of the culprit or based on other factors that would draw extra attention to the suspect. The nonobvious point emphasized in this rule is that foils should not necessarily be selected so as to look like the suspect, but instead should be selected so that they fit the description that the eyewitness had given of the culprit. Such selection strategies can reduce biases against suspects. How much of a difference could a substantial reduction in suspect bias produce? What if the suspect bias multiplier of 2.9 used in Table 2 was reduced to 1.3? That is slightly smaller than the multiplier reported by Valentine and Heaton in their assessment of Yorkshire video parades, so there is some reason to think this number is attainable. The effects of such a change would be reflected in the redistribution of the 38 percent suspect and foil guesses in Table 2. Using computations like those for expanded lineups above, the results for a six-person lineup are that the average foil would draw 6 percent of the guesses and the suspect about 8 percent—essentially the same results as those in Table 3. The reduction in erroneous suspect identifications achieved by halving the suspect bias multiplier is, not surprisingly, roughly equivalent to doubling the size of lineups.

Collect confidence judgments. A clear statement should be taken from the eyewitness at the time of the identification and before any feedback as to whether he or she identified the actual culprit. The motivation for this rule is not to change choosing patterns but to ensure that whatever diagnostic value witness confidence has with respect to witness accuracy is preserved at the time the identification is made and confidence levels are not pushed up or down by events subsequent to the identification. (See C.A. Elizabeth Luus & Gary L. Wells, The Malleability of Eyewitness Confidence: Co-Witness and Perseverance Effects, 79 J. Applied Psychol. 714 (1994).)

Sequential presentations. We also noted a fifth recommendation in our white paper that we considered but did not advance because we thought it less easily adopted than the first four. As we stated therein:

Perhaps the most important procedural variation that we have not incorporated into the core rules at this time is the use of the sequential lineup procedure. In a sequential procedure, an eyewitness views only one lineup member at a time and makes a decision (that is the perpetrator or that is not the perpetrator) regarding each person before viewing another lineup member. When compared to the usual simultaneous procedure, it is clear that the sequential procedure produces a lower rate of mistaken identifications (in perpetrator-absent lineups) with little loss in the rate of accurate identifications (in perpetrator-present lineups). Since the time it was first introduced (Lindsay & Wells, 1985), there have been many replications in the United States, Canada, Germany, and the United Kingdom of the superiority of the sequential procedure over the simultaneous procedure.

(Wells et al., supra, at 639.)

The Steblay et al. meta-analysis, cited above, examined 23 papers containing 30 comparisons of sequential and simultaneous procedures, based on 4,145 research participants. These experiments were conducted around the world and employed a wide array of research participants, formats (photo, video, or both), types of crime (robbery, theft, other, or no crime), and event stimuli (video, live, slides, or transparencies). The authors reported that, across 25 studies, witnesses were about twice as likely to make a false identification from a simultaneous array, with 51 percent mistaken identifications, in target-absent arrays, compared to 28 percent mistaken identifications in sequential target-absent arrays. More pointedly, choices of designated innocent suspects were reduced from 27 percent in simultaneous arrays to 9 percent in sequential arrays. The authors also reported a 15 percent loss in correct identifications when moving from simultaneous to sequential procedures, from 50 to 35 percent perpetrator selections, so the gains were not unalloyed. Some readers may recognize that the loss in positive identifications may be largely attributable to "lucky guessers" being converted into people failing to make a choice. If those readers are among the group who find it problematic to secure convictions based on guesses, this may not seem a loss.

Despite possible losses in perpetrator identifications, the change in procedure would clearly and substantially reduce the rates at which innocent suspects were selected. If similar rates of performance occurred in lineups composed like those in Sacramento, 67 percent of which contained the perpetrator, the ratio of correct identifications to erroneous suspect identifications would change substantially as compared to Table 2. Imagine correct identifications drop from 64 percent accurate to 49 percent accurate, that is, from 25 to 19 correct identifications, and suspect identifications drop, let us say, from 14 percent to 7 percent, that is, from four to two erroneous suspect identifications. Whereas the original ratio of correct to incorrect identifications in Table 2 is 25:4 or 6.25:1 for simultaneous lineups, the ratio for sequential procedures would be 19:2 or 9.5:1. The overall increase in odds is somewhat short of the doubling of odds achieved with increasing lineup size from six to 12 or halving the suspect bias multiplier, but it is still substantial.

Make more conservative use of lineups. Another strategy for improvement involves police using lineup identifications more conservatively. Specifically, they might be used only when there is a reasonably strong likelihood the suspect is the perpetrator. (There are reasons to resist this suggestion, discussed below.) If, instead of 67 percent of suspects being guilty, the proportion of suspects in Sacramento had been 90 percent and nothing else had changed, the pattern of results would have been roughly that shown in Table 4 (with rounding to whole numbers). As is obvious from the table, greatly increased police effectiveness substantially reduces the number of mistaken "identifications" of suspects by guessers. Because the number of correct identifications increases—due to the fact more lineups contain perpetrators and not because of any improvement in witness performance—the ratio of perpetrator identifications to innocent suspect identifications would rise dramatically from the 6.25:1 of Table 2 to 33:1. About 3 percent of identified suspects would be innocent.

This change in practice would produce a much more dramatic improvement in the ratio of correct to incorrect identifications than would be achieved by doubling the size of lineups or substantially reducing suspect lineup bias (which roughly halves the number of innocent suspect "identifications" in the examples above) or shifting to sequential lineups (which might halve the number of innocent suspect "identifications" but also produce some loss in correct identifications). Of course, all these elements could be combined and the result could be that erroneous suspect identifications would be reduced from the estimated 14 percent of suspect identifications in the Behrman and Davey study to much less than 1 percent of all suspect identifications. Most of the beneficial effect is, of course, obtained by presenting witnesses with fewer innocent suspects and converting identifications of innocent suspects into identifications of foils.

Of course, it may simply be impossible for the police to attain a 90 percent figure. Lineups are partly used as investigatory tools and partly used to generate forensic evidence confirming other evidence or strong suspicions. Any policy that, for example, limited lineups to situations in which there was already some extrinsic evidence would undercut the value of lineups as investigatory tools. The tension between the investigatory and evidence testing use of lineups is perhaps best illustrated if we imagine that the police are forced to specify, from the outset of their investigation, whether they will use a particular witness or subset of witnesses for investigatory purposes or for the generation of forensic evidence. How might witnesses in the two categories be handled differently?

If police knew that a witness used for purely investigatory purposes could not be used in court as a vehicle for identifying a perpetrator, this would free the police to use that witness in a rather different manner. Such witnesses might be encouraged to examine mugshots for the sole purpose of generating "identifications" of as many individuals who might plausibly be the perpetrator as possible. Police could then use this information as a basis for further investigation and not have to worry that the credibility of the witness might be undermined in court because the witness made multiple identifications. The witness simply would not be permitted to make an in-court identification. On the other hand, witnesses used to generate forensic evidence would be handled differently as well. First, they would be protected from contamination by discouraging them from speaking to other witnesses, not permitting them to undertake investigatory inspections of mugshots, and the like. They would only be administered test procedures designed to maximize the diagnostic or probative value of any identifications that they did make. Procedures would be designed to maximize the ratio of correct identifications of perpetrators to incorrect identifications of innocent suspects.

We can therefore see that if police were prepared to undertake reform of their identification procedures in line with the recommendations above, there clearly are a number of steps they could take. Perhaps they would do so because they recognize the importance of generating reliable evidence rather than merely generating some evidence from eyewitnesses, or because prosecutors refused to use identifications that are not the product of sound procedures, or because courts refused to admit evidence derived from unsound procedures that produce avoidable errors.

Shortly after publication of the 1998 white paper, Janet Reno formed a Technical Working Group for Eyewitness Evidence. The group included researchers (including some authors of the 1998 piece), prosecutors, and police. Eventually, the U.S. Department of Justice (DOJ) issued a document entitled Eyewitness Evidence: A Guide for Law Enforcement. (U.S. Dep’t of Justice, Eyewitness Evidence: A Guide for Law Enforcement (1999) (Report NCJ 178240), available at http://www.ncjrs.org/pdffiles1/nij/178240.pdf.) This document contains the first set of national guidelines in the United States for the collection and preservation of eyewitness evidence.

As summarized in an article by the research participants (see Gary L. Wells et al., From the Lab to the Police Station: A Successful Application of Eyewitness Research, 55 Am. Psychologist 581 (2000)), the guidelines include a number of recommended practices that emanate directly from the 1998 recommendations. These include cautioning witnesses against guessing, using only one suspect per identification procedure, selecting fillers that generally fit the witness’s description of the perpetrator, telling eyewitnesses that the person who committed the crime may or may not be in the lineup, and avoiding reports to the witness of any information regarding selected individuals before obtaining the witness’s statement of certainty.

However, Wells et al. emphasize two main shortcomings to the DOJ guidelines. First, they note the failure to implement double-blind administration of identification procedures in which the police officer administering the lineup does not know the identity of the suspect. Second, they emphasize the failure to name the sequential procedure as the preferred lineup procedure. Despite the fact that the DOJ guidelines did not enthusiastically recommend these two procedures, the New Jersey attorney general adopted them, so some progress has been made.

Given the fact that the diagnosticity of lineups can be improved using practices such as those enumerated above, the DOJ guidelines are a useful start, but they do not encompass even a majority of the recommendations made above and they are essentially advisory. In this light I would ask readers to return to the second set of questions posed at the outset of this article (see page 37), but not really addressed herein. Though space dictates that we cannot fully answer these questions, readers can no doubt begin to formulate their own answers in light of the discussion above.

with N’s and Percentages of Totals Indicated

*Numbers and percentages are rounded.

Return to Table of Contents - Spring 2003

Return to Criminal Justice magazine home page