The Well-Made Analysis

Most statistical training covers how to begin a study and says little about how to end one. With Brian Caffo, Stephanie Hicks, and Roger Peng, I developed completion criteria that treat “done” as a claim an analyst can state, check, and defend, and I’ve since recast the question as a stopping problem that weighs further exploration against its cost and the risk of overfitting. That framing matters most once an AI agent runs the analysis, since it can always try one more specification and has no built-in reason to halt.

Public datasets now attract one reanalysis after another, more and more of it automated, and each round draws on the same finite information. With Brian Caffo and Martin Lindquist, I’m developing evidential yield, a measure of how much confirmatory evidence the data can still offer after many earlier studies, along with a rule that flags when one more pass is likelier to manufacture findings than to discover them. Its groundwork is vibration-of-effects and specification analysis, which map how far a result moves across the many defensible ways to run the same investigation, and my open-source software puts those tools in other people’s hands, from forkflow to aws.s3 and the other R packages I’ve authored or maintained, downloaded more than 50 million times in all.

Once a language model can browse, call tools, and read files, anything it takes in may have been written by an adversary, and every step from what it perceives to what it does becomes a point of attack. I study how threats shift as AI systems move from making predictions to taking actions, including prompt injection through retrieved content and poisoned tool outputs. My framework treats every plan step and tool call as an event to monitor, with alert thresholds calibrated so each detector false-alarms in at most 5% of attack-free sessions. In a pre-registered study of 5,580 agent sessions, detectors that checked actions against their sources or against the system being acted on caught attacks early, and enforcement stopped every secret leak. Detectors that compared text with the original task didn’t, and I report that result alongside the ones that worked. I come to this from an earlier career as an information security analyst and data governance specialist, and I’ve trained further through the National Science Foundation’s CARE-AI program. I also advise organizations on AI safety policy and have written three O’Reilly Media chapters on protecting software applications and computer networks.

As a collaborator in the Global Burden of Disease Study, led by the Institute for Health Metrics and Evaluation with more than 16,000 researchers in more than 160 countries, I help produce the figures governments use to compare health progress, set priorities, and allocate resources, estimates that have informed national decisions on alcohol taxation, air quality, and immunization. Studies I’ve co-authored with the network have been cited in more than 200 policy and guideline documents, from the World Health Organization to the Council of the European Union, the World Bank, and the Organisation for Economic Co-operation and Development.

As a member of the National COVID Cohort Collaborative (N3C), I lead a multimodal pipeline that brings together single-cell transcriptomics, drug perturbation data, and electronic health records to rank existing medicines as repurposing candidates for acute kidney injury associated with coronavirus disease 2019 (COVID-19). As a Johns Hopkins Center of Excellence in Regulatory Science and Innovation (CERSI) Scholar, supported by the U.S. Food and Drug Administration, I study how to report analytic variability in ways that serve regulatory decisions. Earlier, as a graduate research assistant on a study funded by the National Aeronautics and Space Administration (NASA), I contributed to analyses of otolith function and sensorimotor performance across spaceflight missions of different durations.

Teaching people to weigh evidence is the classroom side of my methods research. I design instruction and assessment for analytic judgment that hold up when students use AI tools, an approach recognized with an American Statistical Association Best Contributed Paper Award in statistics and data science education.