Data & Research

What Makes Research Empirical

Empirical research is research grounded in observation, measurement, or direct experience rather than in pure argument, tradition, or authority alone. An empirical claim is one that was tested against evidence someone actually collected: an experiment was run, a survey was fielded, records were pulled and counted, behavior was observed and logged. The defining feature is not that the conclusion is correct, plenty of empirical studies turn out to be wrong, but that the claim is in principle checkable against data.

This matters because it draws a useful line between different kinds of claims. "Exercise improves mood" can be investigated empirically: you can measure mood before and after exercise in a sample of people and see what happens. "A balanced life is a good life" is a value judgment that no dataset settles on its own, though data can certainly inform the discussion around it. Confusing the two kinds of claims, treating a value judgment as if a study could prove it, or dismissing an empirical finding as "just an opinion", is one of the more common breakdowns in how people talk about research.

Empirical evidence comes in different strengths depending on how it was gathered. A controlled experiment that randomly assigns people to conditions produces different, generally stronger, evidence about cause and effect than an uncontrolled observation of whoever happened to show up. A large, carefully sampled survey produces different evidence than a handful of anecdotes. None of these are useless, each has a role, but knowing which kind of empirical evidence you are looking at changes how much weight it can bear.

Correlation Versus Causation

Two variables are correlated when they tend to move together: as one rises or falls, the other tends to rise or fall in a somewhat predictable way. Correlation is a purely statistical description of a pattern in data. Causation is a stronger claim: that a change in one variable actually brings about a change in the other.

Correlation on its own cannot establish causation for three classic reasons. First, reverse causation: A and B are correlated, but B might be causing A rather than the other way around. Second, a confounding variable: some third factor C might be driving both A and B, making them move together without either one causing the other. Third, chance: with enough variables and a long enough time series, some will correlate with each other simply by coincidence, a danger that grows the more comparisons are run without correction.

A simple way to interrogate a correlation:

• Direction. If A causes B, does the timing make sense, does A come before B?
• Mechanism. Is there a plausible, specific pathway by which A could cause B?
• Confounders. What third factors could plausibly explain both A and B rising together?
• Dose-response. If A really causes B, does more A tend to produce more B, and not just any A versus no A?

Establishing causation with confidence usually requires a study designed for it. A randomized controlled trial assigns participants to conditions by chance precisely so that, on average, the groups differ only in the variable being tested, which is what lets researchers attribute a difference in outcome to that variable rather than to some pre-existing difference between the groups. When randomization is not possible or ethical, researchers may rely on natural experiments or statistical techniques that attempt to approximate it, but these require much more careful argument and are correspondingly easier to get wrong.

Qualitative Versus Quantitative Methods

Quantitative research collects information that can be counted or measured on some numeric scale: test scores, click-through rates, blood pressure readings, survey ratings on a 1 to 5 scale. It is analyzed with statistics, looking at averages, spreads, proportions, and relationships across a sample, and it tends to be strong at answering questions about how much, how many, or how often, and at generalizing a finding from a sample to a wider population.

Qualitative research collects information that is not reduced to a number: interview transcripts, open-ended survey answers, field notes from observing a setting, the content of documents. It is analyzed by identifying themes, categories, and patterns of meaning in the material. Qualitative work tends to be strong at answering why or how something happens, at surfacing categories and distinctions researchers did not think to ask about in advance, and at capturing context and nuance a fixed-response survey item cannot hold.

Neither approach is inherently more rigorous than the other, they answer different kinds of questions and have different failure modes. Quantitative work can be rigorous in its sampling and statistics while asking a question that is too narrow to be meaningful, or it can reduce something genuinely complex to a number that strips out what mattered. Qualitative work can surface rich, accurate insight into a small number of cases while offering little basis for generalizing to people or situations outside the study. A great deal of applied research mixes both: a quantitative survey to find a pattern, followed by qualitative interviews to understand why the pattern exists, or vice versa.

Primary Versus Secondary Sources

A primary source is firsthand evidence: the original dataset, the original experiment report, a direct transcript, an original document from the time and place being studied. A secondary source discusses, summarizes, or interprets primary sources: a news article covering a study, a textbook chapter synthesizing a body of research, a blog post explaining someone else's findings.

Secondary sources are enormously useful, nobody has time to read every primary study on every topic, and a good secondary source can accurately compress a large and technical literature into something accessible. The risk is that each step of summarizing introduces a chance for nuance to be lost, caveats to be dropped, or a finding to be overstated for the sake of a cleaner headline. A study that found a modest, conditional effect in a narrow population can turn into a sweeping claim by the time it has passed through two or three layers of secondary reporting.

Practical habits for working with sources:

• Trace claims back. When a striking statistic is being cited, look for where it traces back to: is there a named original study or dataset, or does the trail just lead to another secondary source citing the first one?
• Check what the primary source actually says. Headlines and summaries sometimes state a conclusion more strongly, or more generally, than the underlying study supports.
• Note the population and conditions. A finding from one population, time period, or context does not automatically transfer to a different one.
• Treat secondary sources as a map, not the territory. They are useful for finding primary sources and for a first pass at understanding, but a load-bearing claim deserves a look at the original.

How Peer Review Works

Peer review is the process academic journals commonly use before publishing a study: the manuscript is sent to independent experts in the same field, who assess whether the methods are sound, whether the conclusions are actually supported by the data presented, whether the statistics were used appropriately, and whether limitations are honestly disclosed. Reviewers then recommend acceptance, rejection, or revision, often requiring authors to address specific concerns before the work can be published.

It is important to understand what peer review does and does not check. Reviewers typically evaluate the manuscript as written, they do not re-run the experiment, re-collect the data, or independently verify that the reported numbers are accurate. Peer review catches many methodological problems, logical gaps, and overreaching claims, but it is not a guarantee that a published finding will replicate, nor is it proof the study is free of error or, in rare cases, fraud. It is a meaningful quality filter, not a final verdict.

Being published in a peer-reviewed journal is a reasonable, if imperfect, signal that a study met a baseline standard of methodological scrutiny from people qualified to judge it. The quality and rigor of peer review itself also varies across journals, which is part of why field reputation, replication, and independent scrutiny after publication continue to matter even for peer-reviewed work.

This connects to broader habits in avoiding common reasoning errors and intellectual traps, treating "peer-reviewed" as a binary stamp of truth rather than one useful but limited signal among several is itself a common mistake.

Sample Size and Statistical Significance

A sample is the subset of people, items, or observations that a study actually examines, used as a stand-in for a larger population the researchers want to understand. Small samples are more vulnerable to random noise: a handful of unusual cases can swing an average, a percentage, or a correlation a long way, producing a result that looks dramatic in that particular sample but does not reflect a stable, general pattern. Larger and more carefully selected samples let genuine patterns separate from random noise more reliably, and allow smaller true effects to be detected with reasonable confidence.

Statistical significance is a specific, technical concept that is widely misunderstood. A result is typically called statistically significant when the pattern observed would be unlikely to occur by chance alone if there were actually no real effect, usually judged against a threshold researchers set in advance. Statistical significance says something about how likely a pattern is to be noise, it does not by itself say the effect is large, important, or practically meaningful. A tiny, practically trivial effect can be statistically significant if the sample is large enough, and a meaningful effect can fail to reach statistical significance if the sample is too small.

Questions worth asking about any reported statistic:

• How big was the sample? A finding from a dozen participants deserves more caution than the same finding from several thousand.
• How was the sample selected? A large sample that is not representative of the population a claim is made about can still mislead, regardless of its size.
• How large is the effect, not just whether it is significant? Look for the actual size of the difference or relationship, not only whether a threshold was crossed.
• Was this one result among many comparisons? Testing many relationships increases the odds that some will look significant purely by chance.

Survivorship Bias

Survivorship bias happens when a dataset or an analysis only includes the cases that "survived" some selection process, while the cases that did not survive are missing from the data entirely, which can make the surviving cases look more representative, more successful, or more typical than they really are.

A frequently cited illustration involves aircraft damage analysis during wartime: engineers examining returning planes to decide where to add armor initially focused on the areas most riddled with bullet holes, since that was the visible pattern in the data they had. The insight credited to statistician Abraham Wald was that the planes in front of them were precisely the ones that had survived being hit in those locations. The planes that took damage in other areas, and were shot down as a result, were missing from the sample entirely, which meant the armor needed to go where the surviving planes showed the least damage, not the most.

Survivorship bias shows up constantly outside of that example. Studying only companies that are still in business to learn "what successful companies do" ignores every company that tried the same things and failed. Looking only at long-running social media accounts to learn "what works" ignores every account that tried similar tactics and was abandoned. Asking only current customers what they like about a product ignores the customers who disliked it enough to leave. In every case, the missing data, the cases that did not survive the filter, often contains the more informative signal.

Confirmation Bias in Data Interpretation

Confirmation bias is the tendency to notice, seek out, and remember information that fits what someone already believes, while discounting, under-scrutinizing, or simply not looking for information that does not fit. It affects how data is interpreted at every stage: which studies get searched for, which results get read carefully versus skimmed, and which findings get accepted at face value versus picked apart for methodological flaws.

In practice, confirmation bias tends to show up through a few recognizable patterns. One is asymmetric scrutiny: a study that confirms a prior belief gets accepted with little scrutiny of its sample size or methodology, while a study that contradicts the same belief gets examined closely for every possible flaw. Another is selective search: stopping the search for evidence as soon as a confirming result is found, rather than continuing to look for disconfirming evidence as well. A third is motivated interpretation of ambiguous results: reading a mixed or inconclusive finding as supporting a favored conclusion when it could just as easily be read as neutral or unsupportive.

Practical habits that push back against confirmation bias:

• Apply the same scrutiny regardless of outcome. Ask the same questions about sample size, methodology, and confounders whether a study confirms or contradicts a prior expectation.
• Actively look for disconfirming evidence. Deliberately search for studies or data that would challenge a conclusion, not only evidence that supports it.
• Separate what the data shows from what you want it to show. State in advance what pattern would count as support and what would count as disconfirmation, before looking at the result.
• Be suspicious of a result that confirms everything you already believed. It happens, but it is also exactly the situation where scrutiny is most likely to be skipped.

Reading a Methodology Section Critically

The methodology section of a study describes how the research was actually conducted: who or what was studied, how participants or items were selected, what was measured and how, and what analysis was performed. It is often the least-read part of a study, and also the part most likely to reveal whether a reported finding deserves the weight being placed on it.

A short checklist for reading a methodology section:

• Who or what was studied? Note the size and composition of the sample, and whether it resembles the population the conclusion is being applied to.
• How were participants or items selected? Random selection from a defined population supports broader generalization; a convenience sample of whoever was available does not.
• What was actually measured? Check whether the measurement genuinely captures the concept being discussed, or only a proxy for it that may not behave the same way.
• Was there a comparison or control condition? A measurement with nothing to compare it against makes it hard to know what the number means.
• What did the authors themselves list as limitations? Honest studies generally disclose weaknesses in their own design; it is worth reading that section rather than skipping to the conclusion.

None of this requires technical statistical training to do at a basic level. Most of it is close reading: does the method described actually support the claim being made in the abstract or the headline, or is there a gap between what was tested and what is being claimed?

Common Data Visualization Pitfalls

Charts and graphs can clarify a pattern instantly, or they can distort it, sometimes unintentionally and sometimes not. A few recurring design choices are worth watching for whenever a chart is used to support a claim.

Truncated or Manipulated Axes

A bar chart with a y-axis that does not start at zero can make a small difference between bars look enormous, because the visual height of the bars no longer corresponds proportionally to the underlying values. This is one of the most common ways a chart exaggerates a real but modest difference.

Mismatched or Dual Axes

Charts that overlay two different variables on two different y-axis scales can be arranged, deliberately or not, so that the lines appear to move together dramatically, when the actual relationship between the two variables might be much weaker or entirely absent at matched scales.

Cherry-Picked Time Windows

A trend line can look dramatically different depending on the start and end dates chosen for the chart. A short window selected around a spike or a dip can make a case for a trend that disappears once a longer, more complete time period is shown.

Area and Volume Encoding

Representing a single number as the area or volume of a shape, such as a circle or 3D object, is easy to misjudge visually, because people tend to compare the shapes by an obvious dimension like diameter or height rather than true area or volume, which exaggerates the apparent difference between values.

A quick check for any chart: does the axis start at zero where that matters, is the time window representative rather than cherry-picked, and would the same data look dramatically different drawn a different way? If the answer to that last question is yes, the chart is doing some of the persuading, not just the data.

Reproducibility in Research

Reproducibility refers to whether a study's findings can be obtained again, either by re-analyzing the same data with the same methods, or, more demandingly, by running a new study designed to test the same question and getting a similar result. A finding that reliably reproduces across independent attempts is on much firmer ground than a single study that has never been repeated.

Several fields, including parts of psychology and some areas of biomedical research, have gone through widely discussed periods of reflection after systematic efforts to repeat published findings succeeded less often than expected. The broader lesson drawn from this is not that published research is generally untrustworthy, but that a single study, even a well-conducted and peer-reviewed one, is best treated as one data point rather than a settled fact, particularly when the finding is surprising, the sample was small, or no one has yet attempted to replicate it.

Signs that increase confidence a finding will hold up:

• Independent replication. Other researchers, with their own samples and sometimes their own methods, found a similar result.
• Convergent evidence. Different types of studies, such as both observational data and controlled experiments, point toward the same conclusion.
• A plausible and specific mechanism. There is a clear, testable explanation for why the effect would occur, not just a pattern with no proposed cause.
• Effect size consistency. The size of the effect is similar across studies, rather than shrinking or vanishing as sample sizes grow or methods improve.

This is also why a single news article describing "a new study" warrants a more measured reaction than a body of research that has been examined and reproduced by multiple independent groups over time.

Putting It Together: A Reading Checklist

None of the concepts above require a statistics degree to apply. Together they form a practical checklist for reading any claim that cites data or research.

1. Identify the Type of Claim

Is this a claim about correlation, or about causation? Is it empirical at all, or is it a value judgment being dressed up as a factual finding?

2. Check the Source

Is this a primary source or a secondary summary? If secondary, can the claim be traced back to an original study or dataset?

3. Look at the Sample

How large was it, and how was it selected? Does it resemble the population the conclusion is being applied to?

4. Separate Significance From Size

If statistical significance is mentioned, look for the actual size of the effect as well, not just whether a threshold was crossed.

5. Ask What's Missing

Could survivorship bias explain part of the pattern? Is there a plausible confound that could explain the result instead of the stated cause?

6. Watch Your Own Reaction

Is this conclusion being accepted or rejected more quickly because it confirms or contradicts something already believed, rather than on the strength of the evidence itself?

7. Check Whether It Has Been Replicated

Is this a single study, or a pattern that has held up across multiple independent studies over time?

Applied consistently, this checklist does not require distrusting data, it requires reading it the way careful researchers read each other's work: with interest, and with questions.