Key ideas
- NIST organizes the quality of the information it disseminates around utility, objectivity, and integrity, and links scientific results with sound research and statistical methods.
- CERN Open Data combines preserved data with software and documentation; the data level and access policies determine which analyses can be performed.
- Documentation makes it easier to examine methods and analyses, but accessibility or repeating calculations does not, by itself, constitute independent confirmation of a result.
Reproducibility depends, among other things, on whether others can examine how results were obtained and analyzed. But sharing data or describing a method is not, by itself, enough to demonstrate that a conclusion is correct. Documentation provides transparency and can make it possible to repeat analyses; assessing information quality also requires considering its basis, its presentation, and the purpose for which it is intended.
NIST offers an institutional framework for considering the quality of the information it disseminates. Its guidelines are not a report on the reproducibility of scientific studies or an independent assessment of specific findings. Read alongside the National Academies’ approach to reproducibility and replicability and the documentary example of CERN Open Data, they help distinguish criteria for quality, the possibility of analysis, and confirmation of results.
Information quality: three related dimensions
NIST defines quality as a term encompassing utility, objectivity, and integrity. Utility asks whether information serves its intended recipients, including the public. Objectivity has two components: that information is presented accurately, clearly, completely, impartially, and in the appropriate context, and that its content is accurate, reliable, and unbiased. In scientific, statistical, or financial contexts, NIST notes that original and supporting data should be generated, and analytical results developed, using sound research and statistical methods. Integrity refers to protecting information from unauthorized access or modification, as well as corruption or falsification. NIST
These dimensions complement one another, but they are not interchangeable. A clear explanation does not, by itself, guarantee that an analysis is well-founded; available data do not ensure they are useful for every question; and protecting a file does not establish the validity of the conclusions drawn from it. NIST’s framework concerns the quality of the information the institution disseminates, according to its guidelines, not a universal test certifying every scientific result.
Transparency matters when relevant to assessing the utility of information from the public’s perspective, NIST states. In research, making supporting data, procedures, and tools accessible can enable others to understand the work and examine its analyses. However, being able to inspect a result is different from having independently confirmed it.
What data and documentation can contribute
The CERN Open Data portal offers a concrete example of how preserved data are linked to the resources needed to analyze them. According to its description, the portal provides access to data produced by research activities at CERN and includes the software and documentation needed to understand and analyze them. The resource focuses on event data from levels 2 and 3; LHC collaborations may also provide small level 4 samples. The portal’s documentation describes four data levels with different degrees of detail and possibilities for analysis. CERN Open Data
| Data level described by CERN | Content and possibility indicated in the source |
|---|---|
| 1 | Additional information about published results, such as figures and tables. |
| 2 | Simplified formats for outreach and analysis training, such as basic event data in four-vector format. |
| 3 | Reconstructed and simulated collision data, together with specific analysis software, for conducting complete scientific analyses using the existing reconstruction. |
| 4 | Basic raw data, when not already covered by level 3, with reconstruction and simulation software; these may make it possible to produce new simulated signals or reconstruct data again. |
CERN’s description cautions that access depends on each experiment’s preservation and access policies, including embargo periods. Therefore, the existence of a portal does not mean that all data are immediately available or available at the same level of detail. The source also indicates that datasets and other materials in the portal receive persistent DOI identifiers and recommends citing them when reusing them. These are mentioned here as a feature described by the portal; no dataset identifier is reproduced.
In practical terms, data accompanied by software and documentation provide elements for reconstructing or examining analyses, within the limits of the data level and access conditions. This is a condition that can support computational reproducibility, not a guarantee that two teams will obtain identical results or that a conclusion is true.
Reproducibility and replicability are not a single test
The National Academies’ page presents, as context, the repetition of research that produced a discovery as one of the ways the scientific community confirms its validity. This introductory framing should not be confused with a conclusion of the committee described on the same page: the project states that the National Academies would explore issues of reproducibility and replicability in scientific and engineering research. The committee’s planned tasks include developing definitions that account for the diversity of fields, as well as examining factors related to methodology, experimental design, incentives, responsibilities, and intentional manipulation. National Academies
This framing explains why documentation matters: it can make visible which data, methodological decisions, and analyses contribute to a result. But the National Academies material cited here describes the scope and questions of a project; it does not provide final definitions or evidence to quantify the frequency of failures to reproduce or replicate. Nor does it allow us to conclude that NIST’s guidelines or CERN’s model have been shown to improve results throughout research.
It is therefore useful to separate three questions. Is the procedure described sufficiently? This concerns transparency and documentation. Can the data be analyzed and the methods run? This depends, among other things, on access, software, and the available data level. Does other research confirm the result? This requires a different assessment; having files available or repeating calculations does not automatically amount to independent corroboration.
A useful framework, with defined limits
NIST’s guidelines are established in relation to the legislation and federal policies cited by the institution itself. NIST states that they may be reviewed periodically in light of experience, changing requirements, and concerns expressed by the public. Their institutional scope should remain clear: this is a framework for the quality of information disseminated by NIST, not an independent review of the scientific literature or a recent finding about the general state of reproducibility.
The value of combining these resources is that they shed light on different parts of the problem. NIST distinguishes criteria for judging disseminated information; CERN documents a system for accessing data, software, and materials with different levels and conditions; and the National Academies situates reproducibility and replicability as methodological questions that span disciplines. Together, they help explain why documentation matters, without turning availability into proof or attributing to one source what another does not establish.
Reproducible practice requires attention to the quality and context of information, as well as preserving the elements needed to understand and examine an analysis. Transparency can make such scrutiny more feasible. Confirming a result, by contrast, depends on evidence and procedures appropriate to the scientific question; it does not follow merely from the fact that data are documented or accessible. This distinction is essential: documentation makes it possible to evaluate work more effectively, but it does not replace the evaluation of its results.
What we know and its limits
NIST’s primary source describes its institutional guidelines for the quality of the information it disseminates; it does not empirically evaluate the reproducibility of studies or certify findings. The National Academies and CERN references provide methodological context and an example of data and documentation, respectively, but the cited material does not allow reproducibility in science to be quantified or demonstrate that these practices universally improve the validity of results. Data availability depends on their level and the experiments’ access policies.
Follow the evidence
The original sources for exploring this explanation and its limits.
- Reproducibility and Replicability in Science, 2019 National Academies of Sciences, Engineering, and Medicine
- CERN Open Data: Data, Software, and Documentation CERN Open Data Portal
- Reproducibility and information quality: data and methods documentation NIST
Image credits
Pesquisa reprodutível - Replicação e reprodução · Pablo J. Madril · Wikimedia Commons · CC BY-SA 4.0 · Original source · CC BY-SA 4.0 · Resized and re-encoded as JPEG; full framing preserved.