Your ANOVA returns p = 0.002. You reject the null hypothesis. Now what? ANOVA only tells you that at least one group mean is different — it does not tell you which groups differ. That is exactly the problem a post hoc test solves.
What Is a Post Hoc Test in Statistics?
A post hoc test in statistics is a follow-up comparison performed after a significant ANOVA result. It identifies which specific pairs of group means are significantly different from each other while controlling the family-wise error rate (the chance of false positives accumulating across multiple comparisons).
ANOVA only gives you a global answer: yes or no, are any means different? Post hoc tests give you the pairwise answer: Group A vs. Group B — significant or not? Group A vs. Group C — significant or not? And so on for every combination.
Table of Contents
Why You Cannot Just Run Multiple t-Tests
The obvious question is: why not just run pairwise t-tests between every group? The answer is the family-wise error rate. Each t-test carries a 5% false positive risk (at α = 0.05). When you run multiple tests simultaneously, those individual risks compound:
| Number of groups | Pairwise comparisons | Family-wise error rate |
|---|---|---|
| 3 | 3 | 14.3% |
| 4 | 6 | 26.5% |
| 5 | 10 | 40.1% |
| 6 | 15 | 53.7% |
With five groups you would have a 40% chance of at least one false positive — even if none of the groups actually differ. Post hoc tests are specifically designed to keep the family-wise error rate at your chosen α level across all comparisons simultaneously.
Step 1: Run a One-Way ANOVA in R First
Post hoc tests are only valid after a statistically significant ANOVA. Here is a complete example using R's built-in PlantGrowth dataset, which records the dried weight of plants under three conditions: a control group, treatment 1, and treatment 2.
# Load data data(PlantGrowth) # Run one-way ANOVA model <- aov(weight ~ group, data = PlantGrowth) summary(model)
The p-value is 0.016 (below 0.05), confirming that the group means are not all equal. ANOVA stops here — it does not tell you whether ctrl differs from trt1, or trt1 from trt2. For that, you need a post hoc test.
When Should You Use a Post Hoc Test?
Use a post hoc test when all three conditions are true:
- Your ANOVA (or Kruskal-Wallis for non-parametric) returns a significant p-value
- You have three or more groups
- You need to know which specific group pairs drive the difference
If ANOVA is not significant, post hoc tests are not appropriate — running them anyway inflates Type I error and invalidates your findings.
Types of Post Hoc Tests in R (with Code)
1. Tukey's HSD (Honestly Significant Difference) — Most Common
Tukey's HSD is the default choice for most researchers. It compares every possible pair of group means simultaneously, controls the family-wise error rate, and assumes equal variances and normally distributed residuals. In R, the base function TukeyHSD() takes the fitted aov object directly.
# Tukey HSD post hoc test — base R TukeyHSD(model, conf.level = 0.95)
How to interpret Tukey HSD output:
diff— the difference between the two group meanslwr/upr— the 95% confidence interval for that differencep adj— the adjusted p-value (Tukey-corrected). If below 0.05, the pair is significantly different
In this example, only trt2 vs. trt1 is significant (p adj = 0.012). The control group does not differ significantly from either treatment individually.
# Visualize Tukey HSD confidence intervals plot(TukeyHSD(model, conf.level = 0.95), las = 2)
Intervals that do not cross zero indicate significant differences. The plot makes this immediately visible across all pairs.
2. Bonferroni Correction — Most Conservative
The Bonferroni correction is the simplest multiple comparison adjustment. It divides your alpha level (usually 0.05) by the number of comparisons being made, making each individual test harder to pass. The result is very strong control of Type I error, but at the cost of lower statistical power — it is more likely to miss real differences than Tukey HSD.
Use Bonferroni when you have a small number of planned comparisons and need conservative control over false positives. In R, the pairwise.t.test() function applies the correction directly.
# Bonferroni post hoc test in R
pairwise.t.test(PlantGrowth$weight,
PlantGrowth$group,
p.adjust.method = "bonferroni")
Again, only trt2 vs. trt1 is significant (p = 0.013). The Bonferroni p-values are slightly larger than Tukey's because Bonferroni is more conservative. If Tukey and Bonferroni disagree, Tukey is typically more appropriate for balanced ANOVA designs.
3. Scheffé's Method — Most Flexible for Complex Contrasts
Scheffé's method is the most conservative of the common post hoc tests, but it is the only one that validly tests all possible contrasts — not just pairwise comparisons. Use it when you want to compare combinations of groups (e.g., the average of groups A and B against group C) or when your group sample sizes are unequal. In R, use ScheffeTest() from the DescTools package.
# Install if needed: install.packages("DescTools")
library(DescTools)
ScheffeTest(model)
Scheffé produces wider confidence intervals and larger p-values than Tukey. Use it specifically when you have unequal group sizes or need to test non-pairwise contrasts; otherwise, Tukey HSD is preferable.
4. Games-Howell Test — When Variances Are Unequal
Tukey HSD, Bonferroni, and Scheffé all assume homogeneity of variance (equal variances across groups). If Levene's test or Bartlett's test indicates unequal variances, switch to the Games-Howell test. It is a robust alternative that does not require equal variances or equal sample sizes. The rstatix package makes this straightforward.
# Install if needed: install.packages("rstatix")
library(rstatix)
# Check if variances are unequal first
levene_test(PlantGrowth, weight ~ group)
# Run Games-Howell post hoc test
games_howell_test(PlantGrowth, weight ~ group)
5. Dunnett's Test — Comparing All Groups to a Control
When your study has a control group and multiple treatment groups, and you only need to compare each treatment against the control (not treatment vs. treatment), Dunnett's test is the right choice. It is more powerful than Tukey or Bonferroni for this specific scenario because it makes fewer comparisons. Use the multcomp package in R.
# Install if needed: install.packages("multcomp")
library(multcomp)
# Dunnett's test: trt1 and trt2 compared to ctrl (reference)
summary(glht(model, linfct = mcp(group = "Dunnett")))
With the Dunnett test, neither treatment differs significantly from the control at α = 0.05. This is consistent with the Tukey result — only trt2 vs. trt1 is the significant comparison in this dataset.
6. Fisher's LSD — Least Conservative (Use with Caution)
Fisher's LSD (Least Significant Difference) is the least conservative post hoc test — it has the most statistical power but the weakest control of the family-wise error rate. It is only recommended when you have exactly three groups and the omnibus ANOVA is significant. With more than three groups, the false positive rate becomes unacceptably high.
library(agricolae) LSD.test(model, "group", console = TRUE)
7. Dunn's Test — Non-parametric Post Hoc (After Kruskal-Wallis)
When your data violate ANOVA's normality assumption, the Kruskal-Wallis test is the non-parametric alternative. After a significant Kruskal-Wallis result, use Dunn's test as the post hoc procedure — not Tukey or Bonferroni, which assume normally distributed data.
library(rstatix) # Step 1: Kruskal-Wallis test kruskal_test(PlantGrowth, weight ~ group) # Step 2: Dunn's test as post hoc dunn_test(PlantGrowth, weight ~ group, p.adjust.method = "bonferroni")
Which Post Hoc Test Should I Use? (Decision Table)
The choice of test depends on three factors: whether variances are equal, whether you have a control group, and whether data are normally distributed.
| Situation | Recommended test | R function | Package |
|---|---|---|---|
| Equal variances, balanced design — most common scenario | Tukey's HSD | TukeyHSD() |
Base R |
| Unequal variances between groups | Games-Howell | games_howell_test() |
rstatix |
| Comparing treatment groups to a control only | Dunnett's test | glht(… mcp(group="Dunnett")) |
multcomp |
| Very conservative control, or planned few comparisons | Bonferroni | pairwise.t.test(… "bonferroni") |
Base R |
| Unequal group sizes, or testing non-pairwise contrasts | Scheffé's method | ScheffeTest() |
DescTools |
| Exactly 3 groups, full power needed | Fisher's LSD | LSD.test() |
agricolae |
| Non-parametric data (after Kruskal-Wallis) | Dunn's test | dunn_test() |
rstatix |
| Non-parametric, ranked data | Nemenyi test | kwAllPairsNemenyiTest() |
NSM3 / PMCMRplus |
Post Hoc Comparison Table: All Test Types at a Glance
The table below summarizes the main properties of each post hoc test to assist selection.
| Post Hoc Test | Main Use | Assumptions / Conditions | Key Features |
| Tukey's HSD | All pairwise comparisons — default choice | Equal variances, normally distributed residuals | Controls family-wise error rate (FWER); balanced power and protection |
| Bonferroni Correction | Few planned comparisons, stringent error control | None specific, but assumes independent tests | Most conservative; divides α by number of comparisons; lower power |
| Scheffé's Method | Any contrast, including non-pairwise; unequal n | Equal variances preferred; tolerates unequal sizes | Most flexible contrast definition; most conservative pairwise |
| Games-Howell | All pairwise comparisons when variances are unequal | Does not assume equal variances | Robust to variance violations; good choice when Levene's test is significant |
| Dunnett's Test | Each treatment vs. a single control group | Equal variances, normally distributed data | More powerful than Tukey when only control comparisons are needed |
| Fisher's LSD | Pairwise comparisons — 3 groups only | Equal variances, normality | Highest power but weakest FWER control; only safe with 3 groups |
| Dunn's Test | Non-parametric pairwise comparison (post Kruskal-Wallis) | No normality or equal variance assumption | Uses rank sums; best paired with Bonferroni or Holm adjustment |
| Holm-Bonferroni | Stepwise correction — less conservative than Bonferroni | None specific | Sequential rejection; more power than plain Bonferroni while controlling FWER |
| Nemenyi Test | Non-parametric pairwise comparison (post Kruskal-Wallis) | No normality or equal variance assumption | Suitable for ranked data; controls Type I error across comparisons |
How to Interpret Post Hoc Test Results
Regardless of which test you use, the interpretation follows the same logic:
- Adjusted p-value < α (usually 0.05) → the two groups are significantly different. Reject the null hypothesis for that specific pair.
- Confidence interval does not include 0 → significant difference. The interval quantifies the size of the difference.
- p adj > 0.05 → you cannot conclude the two groups differ. This is not the same as proving they are equal.
# Example: extracting significant pairs from TukeyHSD output tukey_out <- TukeyHSD(model) as.data.frame(tukey_out$group)[as.data.frame(tukey_out$group)$`p adj` < 0.05, ]
Visualizing Post Hoc Test Results in R
The ggpubr and rstatix packages allow you to add significance brackets directly to boxplots — one of the clearest ways to report post hoc results.
# Install if needed: install.packages(c("ggpubr","rstatix"))
library(ggpubr)
library(rstatix)
# Run Tukey and extract pairs
stat.test <- PlantGrowth |>
tukey_hsd(weight ~ group)
# Boxplot with significance brackets
ggboxplot(PlantGrowth, x = "group", y = "weight",
color = "group", palette = "jco",
add = "jitter") +
stat_pvalue_manual(stat.test, label = "p.adj.signif",
y.position = c(6.5, 6.9, 7.3))
Significance codes on the plot: ns = not significant, * = p < 0.05, ** = p < 0.01, *** = p < 0.001.
Data Analysis Considerations for Post Hoc Tests
Effect Size
A significant post hoc test tells you a difference exists; it does not tell you whether that difference is practically meaningful. Always report an effect size alongside p-values:
- Cohen's d — standardized mean difference between two groups (small: 0.2, medium: 0.5, large: 0.8)
- Eta-squared (η²) — proportion of total variance explained by group membership
- Omega-squared (ω²) — less biased alternative to eta-squared for smaller samples
library(rstatix) cohens_d(PlantGrowth, weight ~ group, paired = FALSE)
Confidence Intervals
The confidence intervals produced by Tukey HSD are often more informative than p-values alone. A narrow interval that excludes zero indicates both statistical and practical significance. A wide interval that barely excludes zero warrants caution even if p < 0.05.
Reporting Post Hoc Tests in APA Style
APA 7th edition guidance: report the omnibus ANOVA first (F, df, p, η²), then state which post hoc test was used and at what α level, then report each significant pairwise comparison with mean difference, confidence interval, and adjusted p-value. Use the term "post hoc test" (no hyphen) throughout.
Latest R Packages for Post Hoc Analysis
emmeans — The Most Flexible Option
The emmeans package (estimated marginal means) has become the most powerful tool for post hoc analysis in R. It works with any model type — linear models, mixed models, GLMs — not just standard ANOVA. It also supports False Discovery Rate (FDR) control as an alternative to FWER control.
# install.packages("emmeans")
library(emmeans)
em <- emmeans(model, ~ group)
pairs(em, adjust = "tukey") # Tukey adjustment
pairs(em, adjust = "bonferroni") # Bonferroni adjustment
pairs(em, adjust = "fdr") # False discovery rate control
rstatix — Tidy Post Hoc Tests
The rstatix package provides pipe-friendly wrappers for all common post hoc tests, returning tidy data frames that integrate directly with ggplot2 and ggpubr for plotting. It supports Tukey, Games-Howell, Dunn, Wilcoxon pairwise comparisons, and more.
Bayesian Post Hoc Approaches
For researchers working within a Bayesian framework, the BayesFactor package provides Bayesian ANOVA followed by Bayesian pairwise comparisons. Rather than p-values, these return Bayes Factors — evidence ratios quantifying how much more likely the data are under one hypothesis vs. another.
# install.packages("BayesFactor")
library(BayesFactor)
bf <- anovaBF(weight ~ group, data = PlantGrowth)
print(bf)
Real-World Post Hoc Test Examples
Medical Research
In clinical studies comparing multiple treatment arms, ANOVA identifies whether any treatment outperforms the others. Post hoc tests then determine which specific treatment pairs drive the effect. For example, a study of musculoskeletal dysfunction in migraine patients used Bonferroni-corrected post hoc t-tests and Dunn's test to compare headache-free participants against episodic and chronic migraine groups (Luedtke et al., 2017). Similarly, a root canal disinfection study used Scheffé post hoc comparisons after one-way ANOVA to assess antibacterial efficacy across treatment groups (Mathew et al., 2022).
Psychology and Education
Educational studies frequently use Tukey post hoc tests to compare performance across instructional conditions or demographic groups. A study of referred sensations in the orofacial region applied Tukey post hoc tests to compare mechanical sensitivity between participants who experienced referred pain and those who did not (Sago et al., 2023).
Market Research
In consumer research comparing multiple brands or product variants, ANOVA reveals whether preferences differ overall. Fisher's LSD (for three-group comparisons) or Tukey HSD (for four or more brands) then identifies which specific brand pairs show the significant gap in consumer preference scores.
Conclusion
Post hoc tests are the essential second step after a significant ANOVA result. ANOVA asks: "do any means differ?" Post hoc tests answer: "exactly which pairs differ, and how much?" The default choice for most researchers is Tukey's HSD (TukeyHSD() in base R) — it balances power and family-wise error control for the most common case of equal variances and balanced group sizes. Switch to Games-Howell when Levene's test is significant, Dunnett's when you only need treatment-vs-control comparisons, and Dunn's test when your data are non-parametric. Always report effect sizes alongside p-values, and never run post hoc tests when the omnibus ANOVA is not significant.
Frequently Asked Questions (FAQs)
What is a post hoc test in statistics?
A post hoc test in statistics is a multiple comparison procedure performed after a significant ANOVA result. It determines which specific pairs of group means are significantly different from each other while controlling the family-wise error rate. The term "post hoc" (Latin: "after this") indicates the test is performed after the omnibus test confirms that at least some means differ.
When should I run a post hoc test after ANOVA?
Run a post hoc test only when (1) your ANOVA p-value is below your significance threshold, (2) you have three or more groups, and (3) you need to identify which group pairs drive the significant result. If ANOVA is not significant, post hoc tests are not appropriate and should not be run.
What is the difference between Tukey HSD and Bonferroni?
Both tests control the family-wise error rate, but differently. Tukey HSD is specifically designed for all pairwise comparisons in ANOVA and provides better statistical power than Bonferroni for that use case. Bonferroni is more conservative — it divides α by the number of comparisons, making each test harder to pass — and is better suited for a small number of planned comparisons. For standard ANOVA with equal group sizes, Tukey HSD is almost always preferred over Bonferroni.
Which post hoc test should I use in R?
For most ANOVA designs: TukeyHSD(model) in base R. If Levene's test shows unequal variances, use games_howell_test() from the rstatix package. If you only need treatment-vs-control comparisons, use Dunnett's test via the multcomp package. For non-parametric data (after Kruskal-Wallis), use dunn_test() from rstatix.
What is a post hoc analysis vs. a planned comparison?
A planned comparison (a priori) is a specific hypothesis tested before data collection — for example, comparing only Group A vs. Group B because theory predicts they should differ. Post hoc analysis is exploratory, performed after seeing a significant ANOVA result, comparing all pairs to find where the differences lie. Planned comparisons have higher statistical power because fewer comparisons are made, but they require preregistration or a clear theoretical basis.
How does the Bonferroni test control the error rate in ANOVA?
The Bonferroni correction divides the desired α level (typically 0.05) by the number of pairwise comparisons being made. For example, with four groups (six pairwise comparisons), each individual test is evaluated at α = 0.05 ÷ 6 = 0.0083. This ensures that across all six tests combined, the probability of at least one false positive remains at or below 0.05.
What is the difference between ANOVA and a post hoc test?
ANOVA (Analysis of Variance) is the omnibus test — it tests whether any differences exist among three or more group means. The result is binary: yes or no. A post hoc test is the follow-up step that pinpoints which specific group pairs are responsible for the significant ANOVA result. ANOVA must be run and must be significant before post hoc tests are valid.
What is the Scheffé test, and when should I use it?
The Scheffé test is the most conservative pairwise post hoc test, but it is also the only one that validly tests all possible linear contrasts — not just pairwise means comparisons. Use Scheffé when your group sizes are unequal, when you want to compare combinations of groups (e.g., the mean of groups A and B vs. group C), or when you need the highest assurance against false positives. Its conservatism makes it less powerful than Tukey HSD for simple pairwise comparisons.
What are the common post hoc tests used in statistical analysis?
The most widely used post hoc tests are: Tukey's HSD (default for equal variances), Bonferroni correction (most conservative, planned comparisons), Scheffé's method (most flexible contrasts), Games-Howell (unequal variances), Dunnett's test (treatment vs. control), Fisher's LSD (three groups only), and Dunn's test (non-parametric, after Kruskal-Wallis). The emmeans package in R supports all of these plus FDR-based alternatives.
Do you need help with a data analysis project? Let me assist you! With a PhD and ten years of experience, I specialize in solving data analysis challenges using R and other advanced tools. Reach out to me for personalized solutions tailored to your needs.
References
- Ewald, B., Ewald, D., Thakkinstian, A., & Attia, J. (2007). Meta‐analysis of B type natriuretic peptide and N‐terminal pro B natriuretic peptide in the diagnosis of clinical heart failure. Internal Medicine Journal, 38(2), 101–113. https://doi.org/10.1111/j.1445-5994.2007.01454.x
- Kalender, İ. (2015). Simulate_cat: a computer program for post hoc simulation for computerized adaptive testing. Eğitimde Ve Psikolojide Ölçme Ve Değerlendirme Dergisi, 6(1). https://doi.org/10.21031/epod.15905
- Luedtke, K., Starke, W., & May, A. (2017). Musculoskeletal dysfunction in migraine patients. Cephalalgia, 38(5), 865–875. https://doi.org/10.1177/0333102417716934
- Mathew, T., BM, S., GV, P., & Jose, J. (2022). Comparative evaluation of the antibacterial efficacy of chlorhexidine and 810 nm diode laser in root canal disinfection. Cureus. https://doi.org/10.7759/cureus.28596
- Sago, T., Costa, Y., Ferreira, D., Svensson, P., & Exposto, F. (2023). Referred sensations in the orofacial region are associated with decreased descending pain inhibition. Pain, 164(10), 2228–2238. https://doi.org/10.1097/j.pain.0000000000002921
- Shahabadi, M. and Uplane, M. (2015). Synchronous and asynchronous e-learning styles and academic performance of e-learners. Procedia - Social and Behavioral Sciences, 176, 129–138. https://doi.org/10.1016/j.sbspro.2015.01.453
- Tukey, J. W. (1953). The Problem of Multiple Comparisons. Mimeographed monograph, Princeton University.
- Scheffé, H. (1953). A method for judging all contrasts in the analysis of variance. Biometrika, 40(1–2), 87–110.