Agencies should ensure that all forensic records, including conclusions and underlying bench notes, are:
- (a) documented in reports that
- (1) use consistent, accurate, and straightforward terminology,
- (2) include complete and thorough explanations of methods and results,
- (3) distinguish data from interpretations, opinions, and conclusions,
- (4) disclose relevant laboratory-wide data, including protocols and procedures, proficiency results, and quality reports, and
- (5) disclose all known limitations, measures of uncertainty, and error rates; and
- (b) disclosed to the prosecution and defense on equal terms and on an ongoing and timely basis, to the extent permitted by legal rules and court orders.
Comment:
a. Documentation. Agencies should document and retain all steps taken in forensic analysis. That work may involve very different tasks, including making measurements, reaching judgments and conclusions, and searching through databases. Each step taken should be documented, recorded in a routine form, and explained in a manner that would allow another expert to understand how the relevant conclusions were reached. Agencies should make the documentation available to defense lawyers and to prosecutors, as well as to courts.
b. Complete and thorough reports. Reports generated as the result of a scientific analysis should be complete and thorough. In the past, forensic examiners often issued only brief certificates of analysis or other summary lab reports. They tended to report results using technical terms that are not clearly understandable to lawyers or other non-forensic specialists. Forensic laboratory reports too often have contained little to no documentation as to how those results were reached or what they mean. Agencies should ensure that the conclusions expressed in reports, at a minimum, describe the materials used, the methods and procedures followed, and the conclusions reached, together with the limitations of those conclusions.
c. Distinguish data. Forensic examiners should take care to set out which of their forensic findings rely on or present data, and which findings reflect their own evaluation, interpretation, and judgment of data. Examiners have often presented conclusions that have included objective measures, but also interpretative judgments, in a way that makes the reports difficult to evaluate critically. Nor should an examiner’s own personal judgments be sufficient to reach forensic results, unless the examiner can demonstrate that those judgments are repeatable, reproducible, and accurate. Agencies should ensure that the empirical bases for any judgments be disclosed, along with the uncertainty in any such measurements, as discussed in Comment d.
d. Limitations, uncertainty, and error rates. Forensic reports should identify the sources of uncertainty in the procedures used and conclusions drawn, along with estimates of the level of confidence in the results. Thus, any standard deviations, or the amount of variation in values, should be reported. Similarly, with regard to error rates, any standard deviations, or the amount of variation in values, should be reported, as should information not only about false-positive errors, but also about false-negative errors and erroneous, inconclusive determinations.
Agencies also should report information regarding the performance of their forensic professionals. As part of a sound, quality program, agencies should test the accuracy and performance of systems and individual examiners. One method of doing so is to conduct routine proficiency tests, which can measure the accuracy and performance of a particular examiner. Ideally, agencies should conduct rigorous and blind testing, so that people do not know that their performance is being tested at the time they are performing the work. The results of such in-house proficiency testing should be included routinely as a part of standard forensic reports, as it bears upon the accuracy of that particular person’s work and that of the lab.
Agencies should also disclose information regarding the quality of laboratory or system-wide casework, as it bears on the accuracy of the entire laboratory or system. Although it may not be practicable to provide as much detail as might be expected in a research paper, there should be sufficient content to allow a nonscientist reader to understand what has been done and to permit informed, unbiased scrutiny of the conclusion. If error rates are not known, then that is also important information that should be disclosed.
e. Appropriate conclusions. Agencies should ensure that the conclusions expressed in reports and in testimony are not overstated. The conclusions that an examiner formulates should reflect accurately the limitations of the methods used, and any error rates or uncertainty in those methods. Traditionally, many forensic practitioners not only have used terminology that is difficult for a layperson to understand, but have offered definitive and highly conclusory statements in their reports and testimony that were not supported by scientific research or the methods used.
f. Consistent reporting standards. Reporting standards that are adopted as agency policy should set out uniform language for reporting in written reports and in testimony. Ideally, standards for reporting language should follow standards that are commonly used across jurisdictions and are based on national standard-setting.
g. Disclosure. To the extent permitted by law, examiners should provide ongoing disclosure of forensic reports and laboratory case notes to the prosecution and the defense on equal terms. Specific discovery rules, constitutional rules, court orders, or other laws may constrain the ability of forensic professionals to provide such disclosures in particular jurisdictions or cases. Still, the preference for full and evenhanded disclosure ensures that actors in the legal system can understand and make sound use of forensic evidence. Forensic professionals also should make themselves available out of court to explain their analyses to prosecutors and defense counsel, so that those actors can understand the processes followed and conclusions reached. Absent legal barriers, the disclosure obligation is an ethical and scientific obligation of forensic professionals. Such ongoing disclosure should continue after a conviction, to permit the use of forensic evidence that might support appellate or post-conviction relief.
Reporters’ Notes
1. Generally. It is essential that forensic professionals properly document and report their work. Documentation, in the crime-scene-investigation context, is defined as “the recording of information/data at crime scenes. Forms of documentation may include, but are not limited to, notes, photography, video, sketches, measurements, analysis/testing results, etc.” N.Y. State Crime Lab. Advisory Comm., NYCLAC Special Project on Report Standardization 13 (2018), http://www.criminaljustice.ny.gov/forensic/forms/nyclac-report-standardization-project-march2018.pdf. Documentation, however, is equally important after the collection stage, when information and data then is analyzed. Professionals should record and report each step in their analyses, so that lawyers, judges, and jurors can understand the work that was done, and the conclusions reached, so that other experts can reproduce the results.
2. Reporting. The reporting of forensic results traditionally has not conformed to sound scientific methods. Instead, forensic reports often have been conclusory. They have reflected personal opinions and judgments, without any information about the level of uncertainty in such opinions and judgments. They have not always included clear, documented results to help actors in the legal system understand what was done and what the results meant. Some reports contain only identifying and agency information, a brief description of the evidence being submitted, a brief description of the types of analysis requested, and a short statement of the results (e.g., “The green, brown plant material in item #1 was identified as marijuana”). Agencies commonly fail to include a description of the methods or procedures used, much less their limitations. Most agencies do not discuss in their forensic-laboratory reports measurement uncertainties or confidence limits—although that is a particularly urgent need when technical and scientific analysis is performed. Lawyers cannot be expected to understand the methods that a forensic examiner used, their limitations, and what conclusions the examiner reached, based on these cursory lab reports.
Scientific disciplines outside of forensics have standards, templates, and protocols for data reporting. That is because scientists have an ethical responsibility to assist nonscientists in understanding their findings and expert opinions before those findings and expert opinions are used as decision aids. Loene M. Howes et al., Forensic Scientists’ Conclusions: How Readable are They for Non-Scientist Report-Users?, 231 Forensic Sci. Int’l 102 (2013). The same standards, templates, and protocols used in other scientific disciplines can and should also be used in forensics. The American Statistical Association (ASA) has set out guidelines for reporting forensic evidence. The ASA guidelines state that all statements and opinions should “accurately convey the strengths and limitations of forensic findings.” Am. Stat. Ass’n, American Statistical Association Position on Statistical Statements for Forensic Evidence 1 (Jan. 2, 2019), at https://www.amstat.org/asa/files/pdfs/POL-ForensicScience.pdf. That simple principle can guide all forensic reporting. Elaborating on that principle, the ASA emphasizes that in order to make any statements regarding a probability, there “must be data from a relevant population.” Id. at 3. Without such data, statistical statements lack empirical support. Forensic examiners often make probabilistic claims based on their own “subjective sense of how probable the evidence is . . .” Id. In such situations, the ASA strongly counsels against ever suggesting that evidence came from the same source. Even with strong statistical evidence, there is uncertainty in measurement and the possibility that evidence may have come from a different source. Such uncertainty must be measured and disclosed. Without any such statistical evidence and any information about uncertainty, probabilistic claims are particularly problematic. Any claim should instead be set out in “a comprehensive report by the forensic scientist,” which “should report the limitations and uncertainty associated with measurements, and the inferences that could be drawn from them.” Id. For forensic methods that lack statistical support, the ASA recommends acknowledging the lack of a statistical or empirical basis in reports, and then clearly describing what comparisons were made and what steps were followed. Id. at 5.
3. Documentation. Proper reporting begins with documentation at the crime scene. The initial responding officer at a crime scene should produce clear, concise, documented information encompassing his or her observations and actions. That documentation is vital in providing information to substantiate investigative considerations. Tech. Working Grp. on Crime Scene Investigation, Crime Scene Investigation: A Guide for Law Enforcement 17-32 (2000). Documentation should continue at the laboratory. Laboratories should, as some do, make public their procedures for the careful documentation of each method they use. Va. Dep’t of Forensic Sci., Virginia Department of Forensic Science Quality Manual 17 (2018), https://www.dfs.virginia.gov/wp-content/uploads/2018/08/100-D100-DFS-Quality-Manual.pdf. At the very least, sufficient documentation is needed to reconstruct the analysis, if necessary. Another expert should be able to understand what was done.
Fingerprint analysis provides an example. Documentation should make clear what features were relied upon when a decision was made that the prints in question had a common source. By documenting the relevant information gathered during the analysis, evaluation, and comparison of latent prints, and the basis for the conclusion, the examiner creates a transparent record of the method. This provides courts with additional information on which to assess the reliability of the method in a specific case.
Currently, there is no requirement at many agencies for forensic examiners to document which features within a latent print support their reasoning and conclusions. Exec. Off. of the President, President’s Council of Advisors on Sci. & Tech., Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods 143 (2016). Automated systems, such as the Mideo System at some labs, can create a digital record of what features were identified and compared, together with any changes made during the process in what was identified and relied upon to reach conclusions. Still better are systems that record the entire examination and decisionmaking process.
A wide variety of terms have been used by forensic examiners in reports and in court testimony to describe their findings and conclusions, and the degrees of association between evidentiary material (e.g., hairs, fingerprints, fibers) and particular people or objects. Such terms include but are not limited to “match,” “consistent with,” “identical,” “similar in all respects tested,” and “cannot be excluded as the source of.” The use of such terms can affect how the trier of fact perceives and evaluates evidence. Yet, the forensic-science disciplines have not reached agreement or consensus on the precise meaning of any of those terms. Although some disciplines have developed vocabulary and scales to be used in reporting results, they have not become standard practice. This imprecision in vocabulary stems in part from lack of basic research in many fields of forensic science and the corresponding limitations in interpreting the results of forensic analyses. Nat’l Rsch. Council, Comm. on Identifying the Needs of the Forensic Sci. Cmty., Strengthening Forensic Science in the United States: A Path Forward 185 (2009) (2009 NRC Report).
Thus, it is crucial that examiners “prepare reports and testify using clear and straightforward terminology, clearly distinguishing data from interpretations, opinions, and conclusions and disclosing known limitations that are necessary to understand the significance of the findings.” Nat’l Comm’n on Forensic Sci., Recommendation to the Attorney General: National Code of Professional Responsibility for Forensic Science and Forensic Medical Service Providers 3 (2015), https://www.justice.gov/archives/ncfs/file/795311/download. The U.S. Department of Justice has issued its own uniform standards for reporting and for testimony regarding several forensic disciplines. In general, those standards include probabilistic language, they caution against making any claims regarding error rates of zero, and they bar the use of potentially misleading terms like “reasonable scientific certainty.” See, e.g., U.S. Dep’t of Just., Approved ULTR for the Forensic Firearms/Toolmarks Discipline (2019), at https://www.justice.gov/olp/page/file/1083671/download. The ASA, as noted, has made recommendations concerning reporting forensic findings. The OSAC groups also are producing recommendations specific to particular forensic disciplines.
Forensics reports also should disclose the error rates for a given technique. The insistence by some forensic practitioners that their disciplines employ methodologies that have perfect accuracy and produce no errors has made reporting highly misleading. For example, although DNA analysis is considered the most reliable forensic tool available today, laboratories nonetheless can make errors working with DNA—errors such as mislabeling samples, losing samples, or misinterpreting the data. 2009 NRC Report, supra, at 47. Testing outcomes often are less straightforward than the forensic conclusions advanced in the laboratory report. Regardless of the declarant’s intent, a report that provides forensic “outcomes” may well exaggerate forensic reliability or conceal forensic ambiguities. Pamela R. Metzger, Cheating the Constitution, 59 Vand. L. Rev. 475, 498 (2006). All forms of potential error should be included in such documentation. Those will include research regarding known or potential rates of error. They should include errors due not just to possible false identifications (false positives) but also errors due to false nonidentifications (false negatives) and false inconclusive results (in which the evidence was not indeterminate, but should have resulted in a conclusion).
4. Full and equal disclosure. The law has done little to assist with the quality of documentation or adequate disclosure of forensic evidence. The U.S. Supreme Court repeatedly has held that the police and prosecutors, together, have an obligation under Brady v. Maryland, 373 U.S. 83 (1963), to provide the defense with exculpatory and impeachment evidence. But Brady requires production only of reports that are material to the preparation of the defense. Consequently, if the prosecution receives an expert’s report, the report is discoverable only if it is deemed to be “material.” The problem lies not with the materiality standard, but rather with the person who applies that standard. Leaving the decision to the prosecutor to determine “materiality” can lead to nondisclosure and needless litigation. Paul C. Giannelli, Criminal Discovery, Scientific Evidence, and DNA, 44 Vand. L. Rev. 791, 808 (1991). Federal Rule of Criminal Procedure 16 does provide that expert disclosures, in the form of a written summary of testimony, and an inspection of any scientific test of experiment, be provided by prosecutors to the defense, but written summaries of testimony are only required if it is a witness that the government plans to call at trial. Fed. R. Evid. 16(F)-(G).
Systemic deficiencies could be averted by best practices requiring full-file disclosure and discovery of forensic evidence to both sides, the defense and the prosecution, and not just the bare result, but rather comprehensive documentation of the work performed. Unfortunately, that is not currently the law or the practice in most jurisdictions. Brandon L. Garrett, Constitutional Regulation of Forensic Evidence, 73 Wash. & Lee L. Rev. 1147, 1181 (2016). The Confrontation Clause of the U.S. Constitution similarly is an inadequate guard against undisclosed unreliability or error. As noted above, cross-examination of forensic analysts may not provide a meaningful way to challenge forensic analysis. David Alan Sklansky, Hearsay’s Last Hurrah, 2009 Sup. Ct. Rev. 1, 73-74 (2009) (“In the case of forensic science, meaningful confrontation likely requires a good deal more than disclosure of the results reached by the prosecution’s analysts and their methodology. At a minimum, defendants probably need access to independent experts, to the underlying databases on which the state relies, and—where feasible—to samples and materials that will allow them to carry out their own tests.”).
Thus, it is essential that agencies themselves ensure that forensic evidence is carefully documented and then disclosed both to prosecutors and to the defense, in clear and comprehensive reports.