Pragmatic Bayes

Martin Modrák

2025-05-13

What is this about?

Modified from http://www.ForestWander.com, CC BY-SA 3.0 US, via Wikimedia Commons, https://mc-stan.org/bayesplot/

I don’t want to convert you

Philafrenzy, CC BY-SA 4.0, via Wikimedia Commons, [2][3], Public domain, via Wikimedia Commons, Gnathan87, CC0, via Wikimedia Commons

Pragmatic Bayes: a collection of ideas

Contrast with full Bayesian epistemology

No shaming our subjective/objective Bayesian friends!

Epistemology / philsci is not solved

Can we do Bayesian statistics without committing to Bayesian philosophy?

You can like some and reject other ideas

Main Influences

Deborah Mayo

Andrew Gelman

Stan community more broadly

Berna Devezer

Danielle J. Navarro

Mayo : severe testing

Gelman : Theoretical stastic is the theory of applied statistics

Stan : solve problems

Devezer : pluralism, rigour

Navarro : hypothesis testing

And many others

Theoretical background

Ag2gaeh, CC BY-SA 4.0, via Wikimedia Commons

Frequentist calibration

(continuous parameter)

• Confidence interval:
  • For any fixed parameter value, \(x\%\) CI contains the true value at least \(x\%\) of the time.
  • Worst case
  • Usually conservative approximation

Replacing the inequality with strict equality would make the interval not exist in many cases.

Generalizes to confidence sets.

Bayesian calibration

(continuous parameter)

• Credible interval
  • Averaged over the prior, \(x\%\) CrI contains the true value exactly \(x\%\) of the time.
  • Specific values may lead to low coverage
  • Usually exact*

Typically MCMC, so not “exact computation”, but we get an imprecise value for the “exact bound”

For discrete parameters need to convert to probability statements

Frequentist calibration example

\[ y \sim \text{Binomial}(20, \theta) \]

Other inference goals

Frequentist Test = inverted confidence interval

Bayes factor = calibrated posterior model probability

We can more or less convert every uncertainty statement into confidence/credible interval

squared error, bias not here

Pragmatic Bayesian can outfreq the freq

Modified from Smahdavi4, CC BY-SA 4.0, via Wikimedia Commons

Bayes approximates freq

Bernstein-von Mises

We do not live in asymptotic utopia

Most freq methods are approximations!

• ML + Normal approximation
• ML + profile likelihood
  • \(\chi^2\) asymptotics of the likelihood-ratio test
  • Computationally expensive!

Prove that CLT holds

They are valid only in the asymptotic utopia

For profile likelihood, finding a single confidence bound is slightly more costly than fitting the model

Bonus for penalized: problems at boundary values

Bayes as a freq tool - example

Fitting a negative binomial model, with 4 observations per group:

Frequentist via MASS package:

MASS::glm.nb(y ~ group, data = data)

Frequentist via gamlss package:

gamlss::gamlss(y ~ group, family = "NBI")

Bayesian with flat prior via brms package:

brms::brm(y ~ group, data = data, family = "negbinomial", 
  prior = brms::prior("", class = "Intercept"))

Bayes as a freq tool - example II

“The frequentist theory of Bayesian statistics” https://staff.science.uva.nl/b.j.k.kleijn/bkleijn-book-work-in-progress-Sep2022.pdf https://www.jstor.org/stable/pdf/27028770.pdf (dense paper version)

It is hard to be a frequentist!

(with exact finite-sample guarantees)

• Exact freq computation is very hard
• Freq properties hard to empirically check
• Bayesian computation can be verified empirically (SBC, Yao et al.)

Freq. properties cannot be exhaustively checked empirically Bayesian computation can be (SBC + Yao approach)

If we both do Laplace approximations, is there any difference?

checking lme4 results with rstanarm

Pragmatic Bayesian tests their assumptions

KF, Public domain, via Wikimedia Commons

Which assumptions?

All of them

• Prior
• Likelihood
• Computation
• Bayesian workflow

Defensible prior often easy to find

One coherent approach!

Assumptions can be tested also in freq, but less obviously.

Modern Bayesian computation succeeds or fails loudly

Note how the freq NB GLMs fail, but give no warning

Pragmatic Bayesian understands Bayesian limits

TeWeBs, CC BY-SA 4.0, via Wikimedia Commons

Selection effects

10 patients get sick, 10 recover. What is the likely recovery probability?

Should your inference change if you learn that the experimenter would only ever report the results to you if everybody recovered?

In Bayesian statistics it should not! 🤯

\[ \pi_\text{post}(\theta \mid y, \text{ accept(y)}) = \pi_\text{post}(\theta \mid y) \]

The likelihood principle

Selection effects - intuition

Frequentist simulation:

prob_recovery = some_number
repeat {
  y = binomial_rng(N = 10, prob = prob_recovery)
  if(y == 10) then break
}

Bayesian simulation:

repeat {
  prob_recovery = prior_rng()
  y = binomial_rng(N = 10, prob = prob_recovery)
  if(y == 10) then break
}

The Bayesian answer makes sense under repeated draws from the prior!

In practice there is probably a mix of both types of selection (repeating with fixed value, getting a “new draw from the prior”).

Modelling selection

Bayesian can model the frequentist process as a truncated distribution.

In the example, this leads to \(\pi_\text{post}(\theta \mid y) = \pi_\text{prior}(\theta)\).

More generally to an interesting class of models

Pragmatically: we rarely can build a good model of the selection

Example - we want to infer timing of bloom from historical specimens with known date of collection, but only flowering plants are ever collected.

Medical: modelling people who show up at the doctors office

Early stopping

Should your inferences change when you learn that data collection stopped once \(p < 0.05\)?

• The likelihood principle says NO
• No easy cop-out this time

Early stopping - example

Binomial model, max steps = 10000, optional stopping after each step

Stopping when 95% CrI excludes 0.5

Stopping when 95% CrI excludes 0.4 - 0.6

Stopping when the width of 95% CrI is < 0.1

Possible solutions to early stopping

• Smarter stopping rules
• Include time in your model
• Simulate to get freq properties
  • Shoutout to bayesflow

Frequentist don’t have it all

• Freq approaches to early stopping approximate
  • And bespoke/limited
• You need simulations anyway

Limited: no 3 phase group seq + two recalculations

Regulators explicitly require a ton of simulations if you are proposing a non-standard method

Pragmatic Bayesian is not afraid of freq

Lukáš Beneda, CC BY-SA 4.0, via Wikimedia Commons

Some use cases of freq

• Sequential designs
• Freq as approximate Bayes
  • Flat prior + normal approximation
  • Can check with SBC

Pragmatic Bayesian thinks about causality

Authors of the study: Stefan Gustafsson, Erik Lampa, Karin Jensevik Eriksson, Adam S. Butterworth, Sölve Elmståhl, Gunnar Engström, Kristian Hveem, Mattias Johansson, Arnulf Langhammer, Lars Lind, Kristi Läll, Giovanna Masala, Andres Metspalu, Conchi Moreno-Iribas, Peter M. Nilsson, Markus Perola, Birgit Simell, Hemmo Sipsma, Bjørn Olav Åsvold, Erik Ingelsson, Ulf Hammar, Andrea Ganna, Bodil Svennblad, Tove Fall & Johan Sundström, CC BY 4.0, via Wikimedia Commons

Bayesian causality

• McElreath: Full luxury Bayesian inference

• Dawid: Causal inference as decision theory

• Inverse probability of treatment weighting is the selection framework we discussed

We can jointly estimate IPTW and the main model

Pragmatic Bayesian aligns inferences with scientific questions

Bergin, Jerry, Public domain, via Wikimedia Commons

Rich models

• Varying scale between groups already difficult in freq

• Differential equations

• hidden Markov models

• Simulation-based inference (Bayesflow workshop yesterday)

HMMs: Hype Vianey

Rich inferences

Inferences on derived quantities, prediction intervals?

• Just compute per sample!

• Some questions don’t have freq answers without regularization (e.g. mean of general distribution).

  • Once you regularize, you may as well do Bayes.

Inference on linear combinations of parameters is already annoying in freq.

Building freq prediction intervals with finite-sample guarantees is hard!

My personal non-reasons to use Bayes

Steve Evans from Citizen of the World, CC BY 2.0, via Wikimedia Commons

Philosophical arguments that I find not very practically useful.

Prior information

No!

We can rarely elicit and express precise enough priors

Sequential updating?

No!

In practice just fit a big model to all data.

Epistemology?

No!

Severity of tests matters.

Especially no to Bayes factors.

Why I use Bayesian tools

And maybe you should too?

NASA's Scientific Visualization Studio - Advocates in Manpower Management, Inc./Sophia Roberts, University of Maryland College Park/Jeanette Kazmierczak, Public domain, via Wikimedia Commons

Pragmatic reasons

• Useful!
• One coherent approach!
• Finite sample guarantees!
• Rich models!

Thanks for your attention!

Slide & simulation sources:
https://github.com/martinmodrak/pragmatic-bayes