This is the extended website of the Workshop: Hands-On Tutorial on e-Values, Safe Tests and Anytime-Valid Confidence Intervals for Efficient Statistical Inference at the 56th Annual Meeting of the Society for Mathematical Psychology held in Amsterdam, 2023.

# Abstract

Recently developed safe tests based on e-values, and anytime-valid confidence intervals form a suite of statistical methods that simplify and optimise the design, conduct, and inferential process for both single-lab experiments and large-scale multi-lab (replication) studies. Safe tests combine the interpretability of Bayes factors (i.e. measuring evidence for and against a null hypothesis) with frequentist power and type I error guarantees. These guarantees are maintained even if the safe test is conducted after each observation and used to determine whether the experiment should be (prematurely) stopped or continued. Similarly, unlike 95% (Bayesian) credible intervals and 95% (frequentist) confidence intervals, a 95% anytime-valid confidence interval will, with at least 95% chance, cover the true effect size regardless of whether or how data collection is stopped. In this workshop we will provide an introduction to this novel framework of statistical inference, and show how it can be exploited to yield more generalisable conclusions with less data. We will alternate between short theoretical lectures and hands-on practical sessions that focus on designing and making inference for practical problems with R/RStudio.

Grünwald, P. D., de Heide, R., Boehm, U., Turner, R. J., & Ly, A.

# Schedule

We alternate between theory and practical sessions using R. For the practical sessions R Markdown documents are made, and the reader is invited to adapt the code.

Time | Title | Presenter |
---|---|---|

09.00 – 09.30 | Introduction: p-values, Bayes factors and e-values, Rianne’s slides |
Rianne de Heide |

09.30 – 10.30 | Practical 1: Testing the equality of two normal means. R Markdown: 0. Installation guide, I. Testing, II. Design |
Udo Boehm |

10.30 – 11.00 | Confidence/credible intervals, and anytime-valid confidence sequences, Alexander’s slides. |
Alexander Ly |

11.00 – 11.30 | Practical 2: Interval estimation of a normal mean. R Markdown: III. Interval estimation |
Rosanne J. Turner |

11.30 – 12.00 | Extensions to other inference problems | Peter Grünwald |

# Programme

#### Introduction: p-values, Bayes factors and e-values – Rianne de Heide (Vrije Universiteit Amsterdam) [Slides.]

A general overview is provided on p-values, Bayes factors and e-values in the context of sequential testing, and optional stopping in particular.

#### Practical 1: Testing the equality of two normal means – Udo Boehm (CWI Amsterdam)

In this R tutorial we work through the following R Markdown documents:

0. Installing the safestats package

I. Testing: P-values, subjective Bayes factor, and e-variables

II. Design of experiments: Behaviour of e-variables under alternatives

#### Confidence/credible intervals, and anytime-valid confidence sequences – Alexander Ly (CWI Amsterdam) [Slides.]

A general overview is given on the conversion between safe tests/e-variables into anytime-valid confidence sequences.

#### Practical 2: Interval estimation of a normal mean – Rosanne J Turner (CWI Amsterdam/University Medical Center Utrecht)

In this R tutorial we work through the following R Markdown documents:

III. Interval estimation: Classical confidence intervals, (Bayesian) credible interval, and anytime-valid confidence sequences

#### Extensions to other inference problems – Peter Grünwald (CWI Amsterdam and Leiden University)

We brush on general purpose algorithms that allow for the development of e-variables yielding e-values for general testing problems.

# Additional notes and software

– safestats github repository

– In case remote installation doesn’t work: Package tar.gz The download button can be found on top left.