Science to understand a changing world.
RSS
Research Article

An AI analysis finds that only one in eight clinical studies share code

A team led by researchers from MIT and Dana-Farber Cancer Institute examined nearly 4,000 clinical prediction studies and found that sharing code does not guarantee it is documented well enough to be reused.

Minimalist text 'Medical Research' in colorful letters on a neutral background.Contextual image

Key ideas

  • The team led by researchers from MIT and Dana-Farber Cancer Institute examined nearly 4,000 clinical prediction model studies using language models, software tools, and human checks.
  • Approximately one in eight studies reported sharing code, and the documentation and conditions for reusing it varied widely.
  • The findings are informing TRIPOD-Code, a guide in development; the evaluation acknowledges possible classification errors and does not present every count as exact.

What the team reviewed

A study of clinical prediction models examined how scientific papers share the code used in their analyses. The international team, led by researchers from MIT and Dana-Farber Cancer Institute, analyzed nearly 4,000 studies that developed, updated, or validated models to estimate patient outcomes, such as the risk of developing a disease. The full papers were available in PubMed Central’s open-access collection.

According to information released by MIT, the researchers used large language models to locate statements about code availability in the papers and extract links to the associated repositories. Software tools then retrieved those repositories for evaluation. The analysis went beyond checking whether a link existed: it also considered aspects such as documentation, software requirements, and reuse conditions.

The methodological interest is that data and code can help examine what was done in a study: for example, how patients were selected, how missing values were handled, and how outcomes were evaluated. Decisions made during data preparation, model development, and evaluation can influence results; a methods section alone may not describe them in all the detail needed.

Sharing does not always enable reuse

The study found two main shortcomings: code sharing remained uncommon, and when code was made public, the material did not always include the documentation and supplementary information needed to reuse it. Overall, approximately one in eight studies reported sharing code. Repository quality varied: some were organized and described in detail, while others offered little guidance on how to understand or run them.

The review also found differences between journals and countries. Among journals, the proportion of studies sharing code ranged from zero to more than 70 percent. Some journals with low rates had written policies on code sharing. The finding suggests that having a policy is not necessarily enough for the practice to become widespread; the available information does not, by itself, establish what causes the observed differences.

Aspect evaluated What the MIT source describes
Availability Approximately one in eight studies reported sharing code.
Repositories Their organization and documentation varied widely.
Evaluation Documentation, software requirements, and reuse conditions were considered.
Variation The proportion of studies with shared code ranged from zero to more than 70 % depending on the journal.
Example of useful practice Some teams provided synthetic data to facilitate reuse when the original patient records could not be made public.

Synthetic data are an example of how to provide supporting material without publishing the original records, according to MIT. The source does not detail here what methods were used to generate them or assess whether all synthetic datasets allow the results of the corresponding studies to be reproduced.

What automation contributes and where its limits lie

Information about code appeared in many different ways in papers and repositories, so it could not simply be searched for in a uniform field. Language models helped process this unstructured material at scale. However, the evaluation included human checks: two researchers independently reviewed 500 papers and a separate sample of repositories, resolved discrepancies, and compared their judgments with the automated results.

Agreement was higher for concrete features—for example, the presence of a license file—than for assessments of whether the documentation was adequate. For this reason, the authors emphasize the general patterns and caution that classification errors may occur; they do not present every count as exact. Automation made it possible to expand the analysis, but did not eliminate the need to define criteria or interpret qualitative aspects.

A guide in development

The results are informing the development of TRIPOD-Code, a guide that aims to clarify what elements should be shared with analytical code. According to MIT, this initiative builds on TRIPOD and TRIPOD+AI, checklists for reporting the development and evaluation of clinical prediction models. The team is developing recommendations through an expert-review process; the source describes the guide as work in progress, not as a finalized standard.

The need for guidance on these practices reflects the fact that clinical prediction research brings together people from medicine, statistics, and computer science, whose expectations about documentation and dissemination may differ. TRIPOD-Code’s stated aim is to offer practical guidance to researchers and clearer expectations for journals. The study provides descriptive evidence for these discussions, but does not, by itself, show that a future guide will increase code availability or reuse.

As general institutional context, the National Science Foundation describes the support it provides for research and education in science and engineering, but this information does not independently confirm the results of this clinical analysis. NSF: research support. Its directory on materials research concerns physical and materials science programs, a field distinct from code analysis in clinical prediction studies, and therefore does not provide specific corroboration of the finding.

The National Science Foundation references provide institutional context, not independent verification of the analysis results.

What we know and its limits

The analysis was limited to clinical prediction studies included in PubMed Central’s open-access collection and to the papers and repositories that could be examined. The source cautions that classification errors are possible, especially in qualitative judgments about the adequacy of documentation, so the counts should be interpreted as general patterns rather than exact figures. The results describe publication practices and do not demonstrate that editorial policies cause those practices or that TRIPOD-Code will change them. The NSF references provided offer institutional context and do not independently corroborate this study.

Original sources3

The original sources for exploring this explanation and its limits.

  1. Studies and Results: How NSF Supports Research NSF
  2. How research into materials is organized and what it produces NSF
    Institutional reference for context. It does not constitute independent confirmation of the main source's findings.
  3. AI helps researchers investigate how thousands of medical studies share code MIT
Image credits

Minimalistic text 'Medical Research' in colorful letters on a neutral background. · Photo by Tara Winstead on Pexels · Original source · Pexels License · Resized and re-encoded as JPEG; full framing preserved.

The SciClue Editorial Team is responsible for creating and publishing this content. AI supports information gathering, source research and drafting within an editorial process that includes source and consistency checks.
Have you found something we should correct?
Explore all concepts

Put it into practice

Energy use calculator Wave laboratory Probability laboratory

Every discovery starts with a question.

Articles, concepts, technologies and indicators. All connected.

Energy Physics Concepts Tools