Four years ago, journalist John Carreyrou questioned the validity of biotech startup Theranos’ revolutionary medical technology, which the company claimed could run hundreds of common blood tests on a tiny microdrop of blood. In 2018, after a string of legal and commercial challenges from investors and medical authorities, Theranos ceased operations completely.

Today, a new generation of innovative biomedical startups continue to win billion-dollar valuations and massive infusions of venture capital as they promise to revolutionize medicine and disrupt massive healthcare markets. According to a new paper authored by three Stanford researchers, many of these companies also publish little to no peer-reviewed research.

Report co-authors Dr. John Ioannidis, Dr. Ioana Cristea and Eli Cahan argue that stealth biomedical companies may offer as much potential for fraud and hype as they do for legitimate innovation. Ultimately, they say, opacity among healthcare startups may pose dangers to human health if products are misrepresented or faulty.

The research involved assembling a list of 18 current and 29 former healthcare ‘unicorns,’ private companies with billion-dollar evaluations developing health products. The analysis revealed that all of these healthcare unicorns combined had collectively published only 425 PubMed papers, most of which are not highly cited and sometimes even are totally tangential to the core projects of the company.

The 2019 paper follows research initially published in 2015, which critiqued Theranos’ lack of peer reviewed science at time when the fraudulent blood testing company had a $9 billion valuation.

“My question was: Was that a solitary case—kind of an exception or bad apple—or is there a fundamental problem that is more pervasive?” said Ioannidis, co-director of the Meta-Research Innovation Center at Stanford (METRICS), in an interview with The Daily. “Is Theranos an exception, or do we see a similar pattern of startups that make spectacular claims, that have spectacular valuations, huge expectations, but little to no evidence?”

In that 2015 paper, he critiqued the divide between health startups such as Theranos and the world of research, writing that “stealth research creates total ambiguity about what evidence can be trusted in a mix of possibly brilliant ideas, aggressive corporate announcements, and mass media hype.” Stealth research is research that is not published in peer-reviewed literature.

At the time, Ioannidis said, Theranos’ lawyers tried to convince him to recant or soften his criticism by explaining that Theranos was successfully obtaining FDA approval. But Ioannidis wouldn’t budge.

Though Theranos’ faulty science was uncovered by Carreyrou in October 2015, Ioannidis and his co-authors maintain that scientific standards for startups lack accountability.

“Given the revolutionary claims made by [today’s] cohort of unicorns — ranging from curative immunotherapies and vaccines for cancer to machine-learning enabled reinvention of drug discovery to nanotechnology-powered genomic sequencing — their impact in the public literature seems empirically unimpressive, close to nil,” the paper states.

These companies included Stemcentrx, which won a $10 billion buyout from pharmaceutical conglomerate AbbVie despite having only 16 published papers and only one highly cited paper. Another former unicorn, Acerta Pharma — which was purchased by pharmaceutical conglomerate AstraZeneca for more than $7 billion despite also having only one highly cited paper — was found to have fabricated preclinical data for its flagship leukemia drug.

Though Ioannidis told The Daily that these warning signs alone are not too worrying, other METRICS researchers at Stanford fear they may represent the tip of a more dangerous iceberg. Patterns of opacity, the co-authors write, extend beyond unicorns to the world of less valuable biomedical startups. One concern is that companies might be cherry-picking data or hiding troubles: the idea that misbehavior is pretty common, but discovered and publicly heralded far less often.

Ioannidis maintains that he does not want dramatic change — only a slight increase in transparency.

“There is no need to disclose things that would otherwise need to be protected. I’m not doing this to disrupt disruptors,” he said.

Nonetheless, some in the biotech investing world are fighting back. In an interview with Forbes, Racquel Bracken, a vice president at Venrock Capital, argued that the companies that Stanford researchers analyzed were generally tech-focused and more digital than biological.

“The headline message of ‘unicorns don’t publish’ is really erroneously concluded based on the methods and companies they picked to include,” said Bracken.

Researcher Eli Cahan responded to this claim on Twitter, arguing that less than one-third of the paper’s companies were digital health companies.

Nina Kjellson, general partner at Canaan Partners, also critiqued the paper in an interview with Forbes, claiming that scientific publications represent “one small piece of what is adding rigor to the process” in private biomedical research. She argues that other checks and balances are enough, and that a call to invest time and money in publishing research reflects an undue and unnecessary burden on biotech startups.

Ioannidis understands this critique but pushes back against the idea that what he is calling for is an oppressive burden.

“I don’t expect startups to become publication factories. It’s not their priority — it should not be their priority,” he said. “But I think that there should be some connection with the scientific literature. It could be just a single paper that is a well-documented paper that defends the case that indeed there is something extraordinary here.”

Sympathizing with the risky environment that startups navigate, Ioannidis believes that the research world can grow to meet the needs of startups just as the startup world grows to meet the needs of society.

“Under the best of circumstances, only a few of these ideas are going to materialize in the way that they are very successful,” he said. “I think that there is a very unique opportunity [at Stanford] to promote some standards of scientific transparency without posing a threat to the innovation and potential for making that a marketable product.”

Contact Cooper Veit at cveit at stanford.edu.