Lessons on Pre-Registrations and Pre-analysis Plans

# Lessons on Pre-Registrations and Pre-analysis Plans
## After Encoding 300 Studies from the AEA RCT Registry

<br/>

<br/>

.small[*Research Transparency and Reproducibility Training (RT2)*<br/>
*UC Berkeley*<br/>
*May 21. 2026*]

---

## About Me / Motivation for Evidence Aggregation

- Today's presentation:
  1. Summary of our work on Reporting Guidelines for Publication Bias (RGPB) in Economics (with Miguel, Pathan, Rocha, Shaji, Sørensen, Tungodden)
  2. Discuss lessons for the pre-registration of your future or ongoing studies.

---

# Summary of Our Study on Pre-Registrations and Reporting of Results in Economics

---
class: contrib

## Contributions

We develop an approach to **standardize and record hypotheses** from the AEA Registry, and run an RCT on study authors to answer:

**Q1. What fraction of pre-registered hypotheses have available results after 8–9 years per study?**
  - Conditional on available, what fraction are null?

**Q2. What prevents researchers from reporting missing pre-registered results?**
  - Lack of *awareness*?
  - Lack of *engagement*?
  - Lack of *resources*?

<div style="margin-top: -15px; visibility: hidden; font-size: 24px;">.hi-num[Our stronger intervention helps explain between a quarter and a third of the File Drawer (28%).]</div>

---

class: center, middle, samp-slide
background-image: url("figs/consort_all.png")
background-size: contain
background-position: center
background-repeat: no-repeat

---
count: false
class: center, middle, samp-slide
background-image: url("figs/consort_all.png")
background-size: contain
background-position: center
background-repeat: no-repeat

---
count: false
class: center, middle, samp-slide
background-image: url("figs/consort_all.png")
background-size: contain
background-position: center
background-repeat: no-repeat

<div style="position: absolute; bottom: 60px; right: 40px; text-align: right; font-size: 29px; color: #c0392b; font-weight: bold;">
Stage 1: Humans only (intentionally no LLM)<br/>
Stage 2: Explore use of LLM (ongoing, not in this paper)
</div>

---

## Main Outcomes

.three-col[
.col[
.col-title[Fraction *Available*]
<table class="otbl"><tr><th></th><th></th><th>Available</th></tr><tr><td class="hyp">H1</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td></tr><tr><td class="hyp">H2</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td></tr><tr><td class="hyp">H3</td><td class="zig">/\/\/\/</td><td class="no-w">No</td></tr><tr><td class="hyp">H4</td><td class="zig">/\/\/\/</td><td class="no-w">No</td></tr></table>
]
.col[
.phantom[.col-title[Fraction *Null*]
<table class="otbl"><tr><th></th><th></th><th>Available</th><th>Null</th></tr><tr><td class="hyp">H1</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td><td class="no-w">No</td></tr><tr><td class="hyp">H2</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td><td class="yes-dg">Yes</td></tr><tr><td class="hyp">H3</td><td class="zig">/\/\/\/</td><td class="no-w">No</td><td class="empty"></td></tr><tr><td class="hyp">H4</td><td class="zig">/\/\/\/</td><td class="no-w">No</td><td class="empty"></td></tr></table>]
]
.col[
.phantom[.col-title[Fraction *Explained*]
<table class="otbl"><tr><th></th><th></th><th>Avail.</th><th>Null</th><th>Explained</th></tr><tr><td class="hyp">H1</td><td class="zig">/\/\/\/</td><td class="no-w">Yes</td><td class="no-w">No</td><td class="empty"></td></tr><tr><td class="hyp">H2</td><td class="zig">/\/\/\/</td><td class="no-w">Yes</td><td class="no-w">Yes</td><td class="empty"></td></tr><tr><td class="hyp">H3</td><td class="zig">/\/\/\/</td><td class="no-w">No</td><td class="empty"></td><td class="no-w">No</td></tr><tr><td class="hyp">H4</td><td class="zig">/\/\/\/</td><td class="yes-b">No</td><td class="yes-b"></td><td class="yes-b">Yes</td></tr></table>]
]
]

---
count: false

## Main Outcomes

.three-col[
.col[
.col-title[Fraction *Available*]
<table class="otbl"><tr><th></th><th></th><th>Available</th></tr><tr><td class="hyp">H1</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td></tr><tr><td class="hyp">H2</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td></tr><tr><td class="hyp">H3</td><td class="zig">/\/\/\/</td><td class="no-w">No</td></tr><tr><td class="hyp">H4</td><td class="zig">/\/\/\/</td><td class="no-w">No</td></tr></table>
]
.col[
.col-title[Fraction *Null*]
<table class="otbl"><tr><th></th><th></th><th>Available</th><th>Null</th></tr><tr><td class="hyp">H1</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td><td class="no-w">No</td></tr><tr><td class="hyp">H2</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td><td class="yes-dg">Yes</td></tr><tr><td class="hyp">H3</td><td class="zig">/\/\/\/</td><td class="no-w">No</td><td class="empty"></td></tr><tr><td class="hyp">H4</td><td class="zig">/\/\/\/</td><td class="no-w">No</td><td class="empty"></td></tr></table>
.small[Denominator varies between descriptive (conditional) and causal analysis (unconditional).]
]
.col[
.phantom[.col-title[Fraction *Explained*]
<table class="otbl"><tr><th></th><th></th><th>Avail.</th><th>Null</th><th>Explained</th></tr><tr><td class="hyp">H1</td><td class="zig">/\/\/\/</td><td class="no-w">Yes</td><td class="no-w">No</td><td class="empty"></td></tr><tr><td class="hyp">H2</td><td class="zig">/\/\/\/</td><td class="no-w">Yes</td><td class="no-w">Yes</td><td class="empty"></td></tr><tr><td class="hyp">H3</td><td class="zig">/\/\/\/</td><td class="no-w">No</td><td class="empty"></td><td class="no-w">No</td></tr><tr><td class="hyp">H4</td><td class="zig">/\/\/\/</td><td class="yes-b">No</td><td class="yes-b"></td><td class="yes-b">Yes</td></tr></table>]
]
]

---
count: false
name: outcomes

## Main Outcomes

.three-col[
.col[
.col-title[Fraction *Available*]
<table class="otbl"><tr><th></th><th></th><th>Available</th></tr><tr><td class="hyp">H1</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td></tr><tr><td class="hyp">H2</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td></tr><tr><td class="hyp">H3</td><td class="zig">/\/\/\/</td><td class="no-w">No</td></tr><tr><td class="hyp">H4</td><td class="zig">/\/\/\/</td><td class="no-w">No</td></tr></table>
]
.col[
.col-title[Fraction *Null*]
<table class="otbl"><tr><th></th><th></th><th>Available</th><th>Null</th></tr><tr><td class="hyp">H1</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td><td class="no-w">No</td></tr><tr><td class="hyp">H2</td><td class="zig">/\/\/\/</td><td class="yes-g">Yes</td><td class="yes-dg">Yes</td></tr><tr><td class="hyp">H3</td><td class="zig">/\/\/\/</td><td class="no-w">No</td><td class="empty"></td></tr><tr><td class="hyp">H4</td><td class="zig">/\/\/\/</td><td class="no-w">No</td><td class="empty"></td></tr></table>
.small[Denominator varies between descriptive (conditional) and causal analysis (unconditional).]
]
.col[
.col-title[Fraction *Explained*]
<table class="otbl"><tr><th></th><th></th><th>Avail.</th><th>Null</th><th>Explained</th></tr><tr><td class="hyp">H1</td><td class="zig">/\/\/\/</td><td class="no-w">Yes</td><td class="no-w">No</td><td class="empty"></td></tr><tr><td class="hyp">H2</td><td class="zig">/\/\/\/</td><td class="no-w">Yes</td><td class="no-w">Yes</td><td class="empty"></td></tr><tr><td class="hyp">H3</td><td class="zig">/\/\/\/</td><td class="no-w">No</td><td class="empty"></td><td class="no-w">No</td></tr><tr><td class="hyp">H4</td><td class="zig">/\/\/\/</td><td class="yes-b">No</td><td class="yes-b"></td><td class="yes-b">Yes</td></tr></table>
.small[Types of explanation range from logistical to analytical (more later).]
]
]

<a href="#reg-descriptives" class="badge-link">Registration descriptives</a>
<a href="#reg-descriptives-encoders" class="badge-link" style="bottom: 100px;">Registration Descriptives Across Encoders</a>

---
class: center, middle
count: false

# Main Finding #1: <br/> Characterizing The File Drawer <br/> for RCTs in Economics

---
name: main-waterfall
class: center, middle, wf-slide
background-image: url("figs/waterfall_all_hyp.png")
background-size: contain
background-position: center
background-repeat: no-repeat

---
count: false
class: center, middle, wf-slide
background-image: url("figs/waterfall_all_hyp.png")
background-size: contain
background-position: center
background-repeat: no-repeat

---
count: false
class: center, middle, wf-slide
background-image: url("figs/waterfall_all_hyp.png")
background-size: contain
background-position: center
background-repeat: no-repeat

---
count: false
class: center, middle, wf-slide
background-image: url("figs/waterfall_all_hyp.png")
background-size: contain
background-position: center
background-repeat: no-repeat

---
count: false

## File Drawer Size: 58%

### Most Hypotheses Are Not Reported 8–9 Years After Registration

---

## Among Available Hypotheses: What Fraction of Results Are Null?

---
count: false
name: null-dist

## Among Available Hypotheses: What Fraction of Results Are Null?

.small[Lots of null results reported: **64.3%**. Goes against priors that support the idea "Most Published Research is False" (Ioannidis, 2005).]

---

## Surprising? Comparing Estimates to Expert Priors

---
count: false
class: priors-slide
background-image: url("figs/combined_hist_priors.png")
background-size: 90%
background-position: center 65%
background-repeat: no-repeat

## Surprising? Comparing Estimates to Expert Priors

---
count: false
name: priors-main
class: priors-slide
background-image: url("figs/combined_hist_priors.png")
background-size: 90%
background-position: center 65%
background-repeat: no-repeat

## Surprising? Comparing Estimates to Expert Priors

<a href="#priors-groups" class="badge-link" style="bottom: auto; top: 15px; right: 20px;">Additional Prior-Data<br/>Comparisons</a>
<a href="#avail-null-chars" class="badge-link" style="bottom: auto; top: 55px; right: 20px;">Availability and Null<br/>Across Characteristics</a>

<div style="position: absolute; bottom: 25px; left: 20px; font-size: 16px; color: #555;">Bars = distribution of expert priors.</div>

---
count: false
class: contrib

## Contributions

We develop an approach to **standardize and record hypotheses** from the AEA Registry, and run an RCT on study authors to answer:

**Q1. What fraction of pre-registered hypotheses have available results after 8–9 years per study?**
  - Conditional on available, what fraction are null?

**Q2. What prevents researchers from reporting missing pre-registered results?**
  - Lack of *awareness*?
  - Lack of *engagement*?
  - Lack of *resources*?

<div style="margin-top: -15px; visibility: hidden; font-size: 24px;">.hi-num[Our stronger intervention helps explain between a quarter and a third of the File Drawer (28%).]</div>

---
count: false
class: contrib

## Contributions

We develop an approach to **standardize and record hypotheses** from the AEA Registry, and run an RCT on study authors to answer:

**Q1. What fraction of pre-registered hypotheses have available results after 8–9 years per study?** .hi-num[42% → File Drawer of 58%]
  - Conditional on available, what fraction are null? .hi-num[64%]

**Q2. What prevents researchers from reporting missing pre-registered results?**
  - Lack of *awareness*?
  - Lack of *engagement*?
  - Lack of *resources*?

<div style="margin-top: -15px; visibility: hidden; font-size: 24px;">.hi-num[Our stronger intervention helps explain between a quarter and a third of the File Drawer (28%).]</div>

---
count: false
class: contrib

## Contributions

We develop an approach to **standardize and record hypotheses** from the AEA Registry, and run an RCT on study authors to answer:

.grey[**Q1. What fraction of pre-registered hypotheses have available results after 8–9 years per study?** .hi-num[42% → File Drawer of 58%]
  - Conditional on available, what fraction are null? .hi-num[64%]]

**Q2. What prevents researchers from reporting missing pre-registered results?**
  - Lack of *awareness*?
  - Lack of *engagement*?
  - Lack of *resources*?

<div style="margin-top: -15px; visibility: hidden; font-size: 24px;">.hi-num[Our stronger intervention helps explain between a quarter and a third of the File Drawer (28%).]</div>

---
count: false
class: center, middle
background-image: url("figs/consort_with_rct.png")
background-size: contain
background-position: center
background-repeat: no-repeat

---

## What Can Be Done to Encourage More Reporting

What are the main constraints preventing researchers from reporting?

- Lack of information about expectation from the scientific community in general? → ***"Awareness" constraint.***

- Lack of time or perceived low interest in one's own research, combined with limited direct engagement from the scientific community → ***"Engagement" constraint.***

- Lack of resources when asked to report on pre-registered hypotheses in a standardized fashion → ***"Resource" constraint.***

### Interventions and arms

| Interventions / Arms | Control | Awareness | Engagement | Resources |
|---|:-:|:-:|:-:|:-:|
| Email + encouragement |   | ✓ | ✓ | ✓ |
| Empty **Results Report** |   | ✓ | ✓ | ✓ |
| Pre-filled **RR** |   |   | ✓ | ✓ |
| RA support (30h / $1,500) |   |   |   | ✓ |

---
name: rr-found-main
class: rr-slide
background-image: url("figs/rr_found.png")
background-size: 43%
background-position: right 3% center
background-repeat: no-repeat

## Sample Pre-Filled Results Report (RR)

Each pre-registered hypothesis gets a one-row block with:

- statement *as we interpreted it* (authors confirm or correct),
- closest matching result we found (β, SE, location),
- pre-registered heterogeneity rows to verify.

If a result could not be found, the same form asks the authors to fill it in or tell us why.
]

<a href="#rr-notfound" class="badge-link" style="right: auto; left: 20px;">Results Not Found</a>

---
count: false
class: rr-slide
background-image: url("figs/rr_found.png")
background-size: 43%
background-position: right 3% center
background-repeat: no-repeat

## Sample Pre-Filled Results Report (RR)

---
count: false
name: rr-details-main
class: rr-slide
background-image: url("figs/rr_found.png")
background-size: 43%
background-position: right 3% center
background-repeat: no-repeat

## Sample Pre-Filled Results Report (RR)

<a href="#rr-details-backup" class="badge-link" style="right: auto; left: 20px;">Details</a>

---
count: false
class: rr-slide
background-image: url("figs/rr_found.png")
background-size: 43%
background-position: right 3% center
background-repeat: no-repeat

## Sample Pre-Filled Results Report (RR)

---
count: false

## RR Zoom — Hypothesis

---
count: false
class: rr-slide
background-image: url("figs/rr_found.png")
background-size: 43%
background-position: right 3% center
background-repeat: no-repeat

## Sample Pre-Filled Results Report (RR)

---
count: false

## RR Zoom — Closest Result

---
count: false
class: rr-slide
background-image: url("figs/rr_found.png")
background-size: 43%
background-position: right 3% center
background-repeat: no-repeat

## Sample Pre-Filled Results Report (RR)

---
count: false

## RR Zoom — Heterogeneity

---

## Did Authors Engage with the Treatment?

---
count: false

## Did Authors Engage with the Treatment?

<table class="eng-tbl">
<thead>
<tr><th></th><th></th><th colspan="3" style="border-bottom: 1px solid #333;">Clicked on:</th><th colspan="3" style="border-bottom: 1px solid #333;">Responded with email:</th></tr>
<tr><th></th><th>(1)</th><th>(2)</th><th>(3)</th><th>(4)</th><th>(5)</th><th>(6)</th><th>(7)</th></tr>
<tr><th>Arm</th><th>N</th><th>Any<br/>(%)</th><th>Results<br/>(%)</th><th>Details.pdf<br/>(%)</th><th>Yes<br/>(%)</th><th>Informative<br/>(%)</th><th>No<br/>(%)</th></tr>
</thead>
<tbody>
<tr><td><em>Awareness</em></td><td>90</td><td>84.4</td><td>82.2</td><td>0.0</td><td>51.1</td><td>28.9</td><td>22.2</td></tr>
<tr><td><em>Engagement</em></td><td>88</td><td>92.0</td><td>92.0</td><td>52.3</td><td>76.1</td><td>60.2</td><td>15.9</td></tr>
<tr><td><em>Resources</em></td><td>88</td><td>90.9</td><td>90.9</td><td>47.7</td><td>69.3</td><td>50.0</td><td>19.3</td></tr>
<tr style="border-top: 1.5px solid #333;"><td>Total</td><td>266</td><td>89.1</td><td>88.3</td><td>33.1</td><td>65.4</td><td>46.2</td><td>19.2</td></tr>
</tbody>
</table>

.small[*Note:* 14 studies expressed interest in RA Support through the Resources arm (2 via funds and 12 in time of Berkeley RAs). In practice, six took up the RA support from Berkeley but none followed up for the funds.]

---
name: te-main

## Main Treatment Effects on Fraction Available

.small[Awareness does not affect Availability. Engagement and Resources increase Availability by ~6–7pp. Additional resources on top of Engagement have no effect → pooling Engagement and Resources arms.]

<a href="#estimation" class="badge-link" style="bottom: auto; top: 15px; right: 20px;">Estimation &<br/>Hypotheses</a>

---
name: te-summary
class: contrib

## Main Treatment Effects

- Awareness has no effect on Available or Null, but a 4pp effect on Explained (SE = 0.021).

- Across all three outcomes, we cannot reject the hypothesis of no incremental effect of Resources → the specification that pools Engagement and Resources is preferred.

- Effect of Pooled (Eng. or Res.):
  - **Available:** 0.062 (SE = 0.015)
  - **Null:** 0.042 (SE = 0.012)
  - **Explained:** 0.094 (SE = 0.022)

- The effect on Null (unconditional) is driven by more hypotheses being reported. The composition of newly reported results is identical to the pre-treatment distribution.

<a href="#reg-table-backup" class="badge-link" style="right: auto; left: 20px; bottom: 15px;">Full Regression Table</a>
<a href="#null-by-treatment" class="badge-link" style="right: auto; left: 180px; bottom: 15px;">Null by Arms</a>

---

<table class="reg-tbl">
<thead>
<tr><th style="text-align:left">Type of Explanation</th><th>Hypotheses</th><th>Studies</th><th>Estimate</th><th>SE</th></tr>
</thead>
<tbody>
<tr style="border-top: 1.5px solid #333;"><td style="text-align:left">Failed or no intervention</td><td>18.2%</td><td>31.2%</td><td>0.050</td><td>(0.016)</td></tr>
<tr><td style="text-align:left"><b>+ Intervention, no data</b></td><td><b>31.2%</b></td><td><b>33.3%</b></td><td><b>0.094</b></td><td><b>(0.022)</b></td></tr>
<tr style="border-top: 1px solid #999;"><td style="text-align:left">+ Intervention, bad data</td><td>6.0%</td><td>6.2%</td><td>0.108</td><td>(0.023)</td></tr>
<tr><td style="text-align:left">+ Yes data, to-do analysis</td><td>20.4%</td><td>29.2%</td><td>0.120</td><td>(0.024)</td></tr>
<tr><td style="text-align:left">+ Yes data, but low power</td><td>19.4%</td><td>10.4%</td><td>0.132</td><td>(0.025)</td></tr>
<tr><td style="text-align:left">+ Disagree with authors</td><td>4.7%</td><td>6.2%</td><td>0.138</td><td>(0.025)</td></tr>
<tr style="border-top: 1.5px solid #333;"><td style="text-align:left">Total <em>N</em></td><td>653</td><td>48</td><td></td><td></td></tr>
</tbody>
</table>

.small[Rows are cumulative. Estimates show the pooled (Engagement or Resources) treatment effect as each explanation category is added.]

<a href="#newly-available" class="badge-link" style="right: auto; left: 20px; bottom: 15px;">Newly Available</a>

---
class: center, middle
count: false

# Main Finding #2: <br/> Understanding What is Behind <br/> Missing Results in Pre-Registered RCTs in Economics

---

background-image: url("figs/filedrawer_pre_all.png")
background-size: 75%
background-position: center 45%
background-repeat: no-repeat

## How Much of the File Drawer Can We Recover 8–9 Years After by Relaxing Constraints?

---
count: false
background-image: url("figs/filedrawer_pre_t1.png")
background-size: 75%
background-position: center 45%
background-repeat: no-repeat

## How Much of the File Drawer Can We Recover 8–9 Years After by Relaxing Constraints?

---
count: false
background-image: url("figs/filedrawer_post_t1.png")
background-size: 75%
background-position: center 45%
background-repeat: no-repeat

## How Much of the File Drawer Can We Recover 8–9 Years After by Relaxing Constraints?

.fd-bullets[
- *Awareness* only recovers a few explanations — 7% of the File Drawer (FD) (4pp/57pp)
]

---
count: false
background-image: url("figs/filedrawer_pre_t2t3.png")
background-size: 75%
background-position: center 45%
background-repeat: no-repeat

## How Much of the File Drawer Can We Recover 8–9 Years After by Relaxing Constraints?

.fd-bullets[
- *Awareness* only recovers a few explanations — 7% of the File Drawer (FD) (4pp/57pp)
]

---
count: false
background-image: url("figs/filedrawer_post_base.png")
background-size: 75%
background-position: center 45%
background-repeat: no-repeat

## How Much of the File Drawer Can We Recover 8–9 Years After by Relaxing Constraints?

.fd-bullets[
- *Awareness* only recovers a few explanations — 7% of the File Drawer (FD) (4pp/57pp)
- *Engagement* makes 6pp of results available, reducing the FD by 10.5% (6pp/57pp)
- It also uncovers explanations for 9pp — an additional 16.5% of the FD (9pp/57pp)
- Taken together, this explains between a quarter and a third of the FD before the intervention (28% = 16pp/57pp)
]

---

## Contributions

We develop an approach to **standardize and record hypotheses** from the AEA Registry, and run an RCT on study authors to answer:

**Q2. What prevents researchers from reporting missing pre-registered results?**
  - Lack of *awareness*?
  - Lack of *engagement*?
  - Lack of *resources*?

<div style="margin-top: -15px; visibility: hidden; font-size: 24px;">.hi-num[Our stronger intervention helps explain between a quarter and a third of the File Drawer (28%).]</div>

---
count: false
class: contrib

## Contributions

We develop an approach to **standardize and record hypotheses** from the AEA Registry, and run an RCT on study authors to answer:

**Q2. What prevents researchers from reporting missing pre-registered results?**
  - Lack of *awareness*? .hi-num[No effect on Available]
  - Lack of *engagement*? .hi-num[Yes]
  - Lack of *resources*? .hi-num[No additional effect]

<div style="margin-top: -15px; font-size: 24px;">.hi-num[Our stronger intervention helps explain between a quarter and a third of the File Drawer (28%).]</div>

---

# Lessons for Future Registrations

(Collect Priors)

---

## Most Important (Pseudo) Equation to Remember

<br/>

.eq-huge[
`$$H \;\approx\; P \;\times\; I \;\times\; A \;\times\; O \;\times\; (\text{Contrasts}+) \;\times\; \text{Het}$$`
]

- **The number of hypotheses (H)** in most RCTs in Economics grows very fast.

<br/>

- Is this not obvious?

---

## Not Obvious at All

.center[<img src="figs/hist_priors_num_hyp.png" style="max-width: 95%; max-height: 78vh; object-fit: contain; margin-top: -2em;"/>]

- 85% of forecasts are below the actual median estimated from AEA Registry data.

---

## Key Takeaways From the Data

<div style="font-size: 1.5em; line-height: 2.2;">
<p>Overall theme: <b>reduce quantity, increase clarity</b> of your registrations.</p>
<ul>
<li>Aim for simple clear contrasts</li>
<li>Remove ambiguity on what you are going to test (and what not)</li>
<li>Be aware of the combinatorial nature of most hypotheses</li>
<li>Be particularly clear and detailed when (or if) pre-registering heterogeneity</li>
</ul>
</div>

---