class: center, middle, inverse, title-slide # Accelerating Computational Reproducibility in Economics ## Computational Social Science Forum, BIDS ### Fernando Hoces, Katie Hoeberling BITSS ### 12 October 2020 |
slides
--- count: false <style> .center2 { margin: 0; position: absolute; top: 50%; left: 50%; -ms-transform: translate(-50%, -50%); transform: translate(-50%, -50%); } pre.sourceCode { max-height: 200px; overflow-y: auto; } /* .remark-slide-number { position: inherit; } .remark-slide-number .progress-bar-container { position: absolute; bottom: 0; height: 4px; display: block; left: 0; right: 0; } .remark-slide-number .progress-bar { height: 100%; background-color: blue; } */ </style> <style type="text/css"> # CSS for including pauses in printed PDF output (see bottom of lecture) @media print { .has-continuation { display: block !important; } } </style> # Today's Presentation </br> .font130[ 1. [BITSS](#about-bitss) 2. [Why Reproducibility](#motivation) 3. [ACRE Guidelines](#guidelines) 4. [ACRE Platform](#platform) ] --- count: false # Table of Contents </br> .font130[ 1. [**BITSS**](#about-bitss) 2. [Why Reproducibility](#motivation) 3. [ACRE Guidelines](#guidelines) 4. [ACRE Platform](#platform) ] --- background-image: url("Images/BITSSlogo.png"), url(Images/cega.png) background-size: contain, 200px background-position: 50% 100% , 0% 100% name: about-bitss # About Us ### [BITSS](https://bitss.org) .pull-left[ The Berkeley Initiative for Transparency in the Social Sciences works to improve </br>the credibility of science by </br> advancing transparency, </br> reproducibility, rigor, and </br> ethics in research. ### ACRE Aleks Bogdanoski Albert Chae Joel Ferguson Fernando Hoces ] .pull-right[ .right[ We are part of the Center for Effective Global Action ([CEGA](https://cega.berkeley.edu/)). </br></br></br></br></br></br></br> Katie Hoeberling Edward Miguel (PI) Emma Ng Lars Vilhuber (Co-PI) ]] --- count: false # Table of Contents </br> .font130[ 1. [BITSS](#about-bitss) 2. [**Why Reproducibility**](#motivation) 3. [ACRE Guidelines](#guidelines) 4. [ACRE Platform](#platform) ] --- name: motivation # Motivation 1: "Reproducibility Crisis" </br></br> .font130[ | Replication in Social Sciences<br>(same method, different sample) | Reproduction in Economics<br>(same data and methods) | |------------------------------------------------------------------- |------------------------------------------------------ | | OSC ([2015](https://docs.google.com/document/d/1mm_4HZnEz_2Bh8XuiS2tpqCH08MFPyqUhi1baKPqR8Y/edit#heading=h.7vqf2cziid7z)): 30%-60% | Chang & Li ([2015](https://www.nowpublishers.com/article/Details/CFR-0053)): 43% | | Camerer et. al. ([2016](https://science.sciencemag.org/content/351/6280/1433)): ~60% | Gertler et. al. ([2017](https://www.nature.com/articles/d41586-018-02108-9)): 14% | | Nosek & Camerer et. al. ([2018](https://www.nature.com/articles/s41562-018-0399-z)): ~60% | Kingi et. al. ([2018](https://hautahi.com/static/docs/Replication_aejae.pdf)): 43% | | Klein et. al. ([2018](https://journals.sagepub.com/doi/10.1177/2515245918810225)): 50% | Wood et. al. ([2018](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0209416#abstract0)): 25% | ] --- # .font90[M2: More Inclusive Concept Scholarly Output] .font120[ <blockquote> Clarebout Principle: </br></br> “An article about computational science in a scientific publication is not the scholarship itself, it’s merely scholarship advertisement. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.” </br></br> .right[ <cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite> ] </blockquote> ] --- # .font90[M2: More Inclusive Concept Scholarly Output] .font120[ <blockquote> Clarebout Principle: </br></br> “An <b>article</b> about computational science in a scientific publication is not the scholarship itself, <b>it’s merely scholarship advertisement</b>. The actual <b>scholarship is the complete software development environment and the complete set of instructions which generated the figures</b>.” </br></br> .right[ <cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite> ] </blockquote> ] --- # .font90[M2: More Inclusive Concept Scholarly Output] .font130[ Potential benefits of following the Clarebout Principle Well discussed potential positive effects on: - Pedagogy - Incremental generation of knowledge ] -- .font130[ Under discussed: - Possible positive effect on diversity, equity and inclusion: no connections or language skills ("appropriate politeness") required to obtain materials ] ??? An under discussed benefit of this principle is that increases access to knowledge for students who are outside of elite schools (no connections or "appropriate politeness" required to obtain materials). --- # M3: Prevent Loss of Knowledge Every semester, graduate students around **the world** take an Empirical/Applied [`...`] Economics course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. -- | Stage | New Knowledge | |:-----------------: |:------------------------------------------------------------: | | </br></br> | | | </br></br> | | | </br></br> | | | </br></br> | </br> | --- count:false # M3: Prevent Loss of Knowledge Every semester, graduate students around **the world** take an Empirical/Applied [`...`] Economics course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. | Stage | New Knowledge | |:-----------------: |:------------------------------------------------------------: | | Scope </br> (select and verify) | Data and code exist? | | </br></br> | | | </br></br> | | | </br></br> | </br> | --- count:false # M3: Prevent Loss of Knowledge Every semester, graduate students around **the world** take an Empirical/Applied [`...`] Economics course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. | Stage | New Knowledge | |:-----------------: |:------------------------------------------------------------: | | Scope </br> (select and verify) | Data and code exist? | | Assess | Degree of reproducibility for <br>specific part of the paper | | </br></br> | | | </br></br> | </br> | --- count:false # M3: Prevent Loss of Knowledge Every semester, graduate students around **the world** take an Empirical/Applied [`...`] Economics course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. | Stage | New Knowledge | |:-----------------: |:------------------------------------------------------------: | | Scope </br> (select and verify) | Data and code exist? | | Assess | Degree of reproducibility for <br>specific part of the paper | | Improve | E.g. fixed paths, libraries,<br>added missing files, etc. | | </br></br> | </br> | --- count:false # M3: Prevent Loss of Knowledge Every semester, graduate students around **the world** take an Empirical/Applied [`...`] Economics course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. | Stage | New Knowledge | |:-----------------: |:------------------------------------------------------------: | | Scope </br> (select and verify) | Data and code exist? | | Assess | Degree of reproducibility for <br>specific part of the paper | | Improve | E.g. fixed paths, libraries,<br>added missing files, etc. | | Test robustness | Results are robust to additional <br>specifications | --- # M3: Prevent Loss of Knowledge </br> .font120[ - In each of the previous steps there was valuable new knowledge - This knowledge is not usually disseminated - End of semester report, or presentation that is not published. - Language across courses is not standard (reproduction? replication?) - Current attempts to record this are closer to a full paper format (see [Replication Wiki](https://replication.uni-goettingen.de/)) - As a result there is a large amount of missed knowledge. - Each new semester, new students repeat similar exercises and have to reinvent the wheel. - Lost opportunity for building on top of previous exercises - Lost opportunity for aggregating the new knowledge. ] --- count: false # Table of Contents </br> .font130[ 1. [BITSS](#about-bitss) 2. [Why Reproducibility](#motivation) 3. [**ACRE Guidelines**](#guidelines) 4. [ACRE Platform](#platform) ] --- name: guidelines # Context for ACRE - American Economics Association (AEA) creates first data policy in 2006. - Must publish some data (waivers available) -- - AEA updates [policy in 2019](https://www.aeaweb.org/journals/policies/data-code/). - Must post all data and code. Publication is conditional on verifying reproducibility (if confidential: must document extensively) - A new requirement is to post all cleaning code, even for data that is not public - See the [AEA Data Editor Website](https://aeadataeditor.github.io) for more information -- - We should expect high levels of computational reproducibility after 2019 (AEA Journals). -- - We should not demand 100% reproducibility before, but we could identify the gaps and try to improve some. --- # Beyond Binary Judgments Reproductions can easily gravitate towards adversarial exchanges. - Early career researcher (ECR) have incentives to emphasize unsuccessful reproductions - Original authors have a more senior position and can use it to deter in-depth reproductions from ECRs. - The media also focuses on eye-catching headlines -- ### Our approach: We do not want to say > "Paper X is (ir)reproducible" -- We do want to say > "Result Y in paper X has a high/low level of reproducibility according to several reproduction attempts. Moreover, improvements have been made to the original reproduction package, increasing its reproducibility to a higher level" --- count:false # Beyond Binary Judgments Reproductions can easily gravitate towards adversarial exchanges. - Early career researcher (ECR) have incentives to emphasize unsuccessful reproductions - Original authors have a more senior position and can use it to deter in-depth reproductions from ECRs. - The media also focuses on eye-catching headlines ### Our approach: We do not want to say > "Paper X is (ir)reproducible" We do want to say > "Result Y in paper X has a high/low **level** of reproducibility according to **several** reproduction attempts. Moreover, **improvements** have been made to the original reproduction package, **increasing** its reproducibility to a higher level" --- count: true background-image: url(Images/paper-claims.svg) background-size: 610px background-position: 100% 0% # Our Framework .pull-left[ .font100[ Each **reproduction attempt** </br> is centered around scientific </br> **claims** One paper can contain several claims. Each claim may be </br> supported by various </br> **display items**: tables, </br> figures & inline results. A reproduction attempt is </br> at the claim level, and </br> reproducers must record </br> their **specifications** of </br> interest. ] DI: Display Item S: Specificaiton ] .pull-right[ ] --- count: true # .font70[Large part of this exercise is about standardization] .pull-left[ - Computational Reproduction (or Reproduction) - Replication (will not mention this term again!) - Reproduction attempt - Reproduction package - Claim - Display item - Specification - Preferred specification - Raw data - Analysis data ] .pull-right[ - Candidate paper - Declared paper - Reproduction tree - Complete Workflow - Computationally Reproducible from Analytic data (CRA) - Computationally Reproducible from Raw data (CRR) - Reasonable test - Feasible test - Minimal effort ] --- count: false # .font70[Large part of this exercise is about standardization] .pull-left[ - **Computational Reproduction (or Reproduction)** - **Replication (will not mention this term again!) ** - Reproduction attempt - **Reproduction package** - Claim - Display item - Specification - Preferred specification - **Raw data** - **Analysis data** ] .pull-right[ - **Candidate paper** - **Declared paper** - Reproduction tree - Complete Workflow - Computationally Reproducible from Analytic data (CRA) - Computationally Reproducible from Raw data (CRR) - Reasonable test - Feasible test - Minimal effort ] --- background-image: url(Images/stages.svg) background-size: contain names: Stages # Stages --- background-image: url(Images/scope.svg) background-size: contain count:false names: Stages # Stages --- background-image: url(Images/scope.svg) background-size: 300px background-position: 0% 100% names: scoping # Scoping .pull-left[ 1. Select or be assigned a candidate paper 2. Check ACRE Platform for previous entries and verify availability of reproduction package (RP) 3. If no RP, leave a short record, and repeat with a different candidate paper 4. Once RP is found then candidate becomes declared paper 5. Only then: read the paper and select claim(s), display items and specification to reproduce ] .pull-right[ .font90[ >**Box 1:** Summary Report Card for ACRE Paper Entry > **Title:** Sample Title > **Authors:** Jane Doe & John Doe > **Original Reproduction Package Available:** URL/No > [If "Yes"] > **Additional Reproduction Packages:** Number (eg., 2) > [If "No"] >**Contacted Authors?:** Yes/No > [If "Yes(contacted)"] > **Type of Response:** Categories (6). > **Authors Available for Further Questions for ACRE Reproductions:** Yes/No/Unknown ] ] --- background-image: url(Images/assess.svg) background-size: 700px background-position: 50% 20% # Assessment </br></br></br></br></br></br></br></br> ## Two main parts for assessment: .font130[ 1. Find all the elements behind a display item 2. Score the reproducibility of that display item ] --- background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # .font70[Identify All the Elements Behind a Display Item] Reproducers will be asked to draw a clear connection to the raw data sources mentioned in the paper and the display item under reproduction. -- ### Data sources Connect the data sources in the paper's text with specific raw data files. -- ### Analytic data sets Describe each analytic data file. -- ### Code files Inspect all code files and record all their inputs and outputs. With all the information recorded above, reproducers can use the **ACRE Diagram Builder** to generate a **reproduction tree**. --- background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Reproduction Tree .font90[ ```md table1.tex |___[code] analysis.R |___analysis_data.dta |___[code] final_merge.do |___cleaned_1_2.dta | |___[code] clean_merged_1_2.do | |___merged_1_2.dta | |___[code] merge_1_2.do | |___cleaned_1.dta | | |___[code] clean_raw_1.py | | |___raw_1.dta | |___cleaned_2.dta | |___[code] clean_raw_2.py | |___raw_2.dta |___cleaned_3_4.dta |___[code] clean_merged_3_4.do |___merged_3_4.dta |___[code] merge_3_4.do |___cleaned_3.dta | |___[code] clean_raw_3.py | |___raw_3.dta |___cleaned_4.dta |___[code] clean_raw_4.py |___raw_4.dta ``` ] --- count:true background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font90[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font90[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") * | Availability of materials, and reproducibility | * |------------------------------------------------| * |Analysis| Analysis| | Cleaning| Raw | | * |Code | Data | CRA | Code | Data | CRR | * | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font90[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| * L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font90[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| * L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font90[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | * L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font90[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | * L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font90[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | * L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font90[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | * L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels: Proprietary/Confidential Data .font90[ ```md Levels of Computational Reproducibility with Proprietary/Confidential Data (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| * | | Instr. | | | Instr.| | |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3*: Partial analysis data & code| ✔ ✔ | ✔ - | - | - - | - - | - | L4*: All analysis data & code....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5*: Proof of third party CRA....| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8*: Some instr. for raw data....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9*: All instr. for raw data.....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10*:Proof of third party CRR....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels: Proprietary/Confidential Data .font90[ ```md Levels of Computational Reproducibility with Proprietary/Confidential Data (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| | | Instr. | | | Instr.| | |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | * L3*: Partial analysis data & code| ✔ ✔ | ✔ - | - | - - | - - | - | * L4*: All analysis data & code....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5*: Proof of third party CRA....| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | * L8*: Some instr. for raw data....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | * L9*: All instr. for raw data.....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10*:Proof of third party CRR....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels: Proprietary/Confidential Data .font90[ ```md Levels of Computational Reproducibility with Proprietary/Confidential Data (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| | | Instr. | | | Instr.| | |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3*: Partial analysis data & code| ✔ ✔ | ✔ - | - | - - | - - | - | L4*: All analysis data & code....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | * L5*: Proof of third party CRA....| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8*: Some instr. for raw data....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9*: All instr. for raw data.....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | * L10*:Proof of third party CRR....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- background-image: url(Images/improve.svg) background-size: 700px background-position: 50% 20% # Improvements </br></br></br></br></br></br></br></br> ## Three types of improvements: .font130[ 1. Improvements at the paper level 2. Improvements at the display-item level 3. Specific future improvements ] --- background-image: url(Images/improve.svg) background-size: 300px background-position: 0% 100% # Improvements: Paper-level .font140[ - Use version control software (Git/Github). - Improve documentation: comments, indentations, object names, etc. - Re-organize the reproduction package into a set of folders and sub-folders that follow [standardized best practices](https://www.projecttier.org/tier-protocol/specifications/#overview-of-the-documentation), and add a master script that executes all the code in order, with no further modifications. [See AEA's reproduction template](https://github.com/AEADataEditor/replication-template). - Literate programming environment (e.g., Jupyter notebooks, RMarkdown) - Re-write code using a differenet statistical software (ideally open source, like R, Python, or Julia). - Set up a computing capsule (e.g., [Binder](https://mybinder.org/) and [Code Ocean](https://codeocean.com/)). ] --- background-image: url(Images/improve.svg) background-size: 300px background-position: 0% 100% # Improvements: Display item-level </br> .font120[ - Adding missing raw data: files or meta-data - Example: "Add raw temperature and relative humidity data" - Adding missing analytic data files - Example: "Copy the row files from Data folder into new `Analysis\trade cost\Input`" - Adding missing analysis or cleaning code - Example: "Replaced broken Wald bootstrap code with updated code/command" - Debugging code - Example: "was counting each group 4 times in round 1, so fixed that" ] --- background-image: url(Images/robust.svg) background-size: 700px background-position: 50% 20% # Robustness Checks </br></br></br></br></br></br></br></br> ## Two main parts for robustness: .font130[ 1. Increase the number of possible robustness checks 2. Justify the appropriateness of a specific test ] --- background-image: url(Images/robust.svg), url("Images/robustness checks.svg") background-size: 300px, 500px background-position: 0% 100%, 100% 60% # Robustness **Robustness checks:** any possible change in a computational choice, both in data analysis and data cleaning -- .pull-left[ **Reasonable specifications** ([Simonsohn et. al., 2018](https://urisohn.com/sohn_files/wp/wordpress/wp-content/uploads/Paper-Specification-curve-2018-11-02.pdf)): 1. Sensible tests of the research question 2. Expected to be statistically valid, and 3. Not redundant with other specifications in the set. Reproducers will be able to record two types of contributions: - Mapping the universe of robustness checks - Justify a specific robustness check as reasonable ] --- background-image: url(Images/robust.svg) background-size: 300px background-position: 0% 100% # Robustness & Reproducibility .pull-left[ Robustness with level 1 <img src="Images/robustness_lvl0.svg" width="70%" style="display: block; margin: auto;" /> Robustness with levels 5-9 <img src="Images/robustness checks.svg" width="70%" style="display: block; margin: auto;" /> ] .pull-left[ Robustness with levels 2-4 <img src="Images/robustness_lvl2_4.svg" width="70%" style="display: block; margin: auto;" /> Robustness with level 10 <img src="Images/robustness_lvl10.svg" width="70%" style="display: block; margin: auto;" /> ] --- # Promoting a Constructive Exchange #### 1 - Contacting the original author(s) when there is no reproduction package #### 2 - Contacting the original author(s) to request specific missing items of a reproduction package #### 3 - Asking for additional guidance when some materials have been shared #### 4 - Response when the original author has refused to share due to *undisclosed reasons* #### 5 - Response when the original author has refused to share due to legal or ethical restrictions of the data #### 6 - Contacting the original author to share the results of your reproduction exercise #### 7 - Responding to hostile responses from original authors #### Under development: sample responses form authors to reproducers --- # .font80[Example 1: Following up on additional materials] **Template email:** >**Subject:** Clarification for reproduction materials for `[“Title of the paper”]` >Dear Dr. `[Lastname of Corresponding Author]`, > >Thank you for sharing the materials. They have been immensely helpful for my work. > >Unfortunately, I ran into a few issues as I delved into the reproduction exercise, and I think your guidance would be helpful in resolving them. `[Describe the issues and how you have tried to resolve them. Describe whatever files or parts of the data or code are missing. Refer to examples 1 and 2 below for more details]`. > >Thank you in advance for your help. > >Best regards, >`[Reproducer]` --- count:false # .font80[Example 1: Following up on additional materials] **Template email:** >**Subject:** Clarification for reproduction materials for `[“Title of the paper”]` >Dear Dr. `[Lastname of Corresponding Author]`, > >Thank you for sharing the materials. They have been immensely helpful for my work. > >Unfortunately, I ran into a few issues as I delved into the reproduction exercise, and I think your guidance would be helpful in resolving them. **`[Describe the issues and how you have tried to resolve them. Describe whatever files or parts of the data or code are missing. Refer to examples 1 and 2 below for more details]`**. > >Thank you in advance for your help. > >Best regards, >`[Reproducer]` --- # An example of well described issues: .font80[ >Specifically, I am attempting to reproduce [`display item X (e.g., table 1, figure 3)`]. I found that the following components are required to reproduce to reproduce [`display item X `]: ```md display_item_X └───[code] formatting_table1.R ├───output1_part1.txt | └───[code] output_table1.do | └───[data] analysis_data01.csv | └───[code] data_cleaning01.R* | └───[data] UNKNOWN └───output1_part2.txt └───[code] output_table2.do └───[data] analysis_data02.csv └───[code] data_cleaning02.R └───[data] admin_01raw.csv* ``` >I have marked with an asterisk (\*) the items that I could not find in the reproduction materials: data_cleaning01.R and admin_01raw.csv. After accessing these files, I will also be able to identify the name of the raw data set required to obtain output1_part1.txt. This is to let you know that I may need to contact you again if I cannot find this file (labeled as UNKNOWN above) in the reproduction materials. > >I understand that this request will require some work for you or somebody in your research group, but I want to assure you that I will add these missing files to the reproduction package for your paper on the ACRE platform. Doing this will ensure that you will not be asked twice for the same missing file. ] --- count:false # An example of well described issues: .font80[ >Specifically, I am attempting to reproduce [`display item X (e.g., table 1, figure 3)`]. I found that the following components are required to reproduce to reproduce [`display item X `]: ```md display_item_X └───[code] formatting_table1.R ├───output1_part1.txt | └───[code] output_table1.do | └───[data] analysis_data01.csv * | └───[code] data_cleaning01.R* * | └───[data] UNKNOWN └───output1_part2.txt └───[code] output_table2.do └───[data] analysis_data02.csv └───[code] data_cleaning02.R * └───[data] admin_01raw.csv* ``` >I have marked with an asterisk (\*) the items that I could not find in the reproduction materials: **data_cleaning01.R** and **admin_01raw.csv**. After accessing these files, I will also be able to identify the name of the raw data set required to obtain output1_part1.txt. This is to let you know that I may need to contact you again if I cannot find this file (labeled as **UNKNOWN** above) in the reproduction materials. > >I understand that this request will require some work for you or somebody in your research group, but I want to assure you that I will add these missing files to the reproduction package for your paper on the ACRE platform. **Doing this will ensure that you will not be asked twice for the same missing file.** ] --- # Easy to grade: report 1 <object data="Images/03_report_card_PS1.pdf" type="application/pdf" width="850px" height="500px"> <embed src="Images/03_report_card_PS1.pdf"> <p>This browser does not support PDFs. Please download the PDF to view it: <a href="Images/03_report_card_PS1.pdf">Download PDF</a>.</p> </embed> </object> --- # Easy to grade: report 1 .pull-left[ <object data="Images/03_report_card_PS1.pdf" type="application/pdf" width="400px" height="500px"> <embed src="Images/03_report_card_PS1.pdf"> <p>This browser does not support PDFs. Please download the PDF to view it: <a href="Images/03_report_card_PS1.pdf">Download PDF</a>.</p> </embed> </object> ] .pull-left[ <object data="Images/Donaldson_RRRaj_AER.pdf" type="application/pdf" width="400px" height="500px"> <embed src="Images/Donaldson_RRRaj_AER.pdf"> <p>This browser does not support PDFs. Please download the PDF to view it: <a href="Images/Donaldson_RRRaj_AER.pdf">Download PDF</a>.</p> </embed> </object> ] --- # .font80[Suggested timelines for different formats] </br></br> <style type="text/css"> .tg {border-collapse:collapse;border-spacing:0;} .tg td{font-family:Arial, sans-serif;font-size:18px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;border-color:black;} .tg th{font-family:Arial, sans-serif;font-size:18px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;border-color:black;} .tg .tg-baqh{text-align:center;vertical-align:top} .tg .tg-c3ow{border-color:inherit;text-align:center;vertical-align:top} .tg .tg-0pky{border-color:inherit;text-align:left;vertical-align:top} .tg .tg-dvpl{border-color:inherit;text-align:right;vertical-align:top} .tg .tg-0lax{text-align:left;vertical-align:top} </style> <table class="tg"> <tr> <th class="tg-0pky"></th> <th class="tg-c3ow" colspan="2">2 weeks <br> (~10 days)</th> <th class="tg-c3ow" colspan="2">1 month <br> (~20 days)</th> <th class="tg-c3ow" colspan="2">1 semester <br> (~100 days)</th> </tr> <tr> <td class="tg-0pky"></td> <td class="tg-0pky">analysis data</td> <td class="tg-0pky">raw data</td> <td class="tg-0pky">analysis data</td> <td class="tg-0pky">raw data</td> <td class="tg-0pky">analysis data</td> <td class="tg-0pky">raw data</td> </tr> <tr> <td class="tg-0pky">Scoping</td> <td class="tg-c3ow" colspan="2">10% (1 day)</td> <td class="tg-c3ow" colspan="2">5% (1 day)</td> <td class="tg-c3ow" colspan="2">5% (5 days)</td> </tr> <tr> <td class="tg-0pky">Assessment</td> <td class="tg-c3ow" colspan="2">35%</td> <td class="tg-c3ow" colspan="2">25%</td> <td class="tg-c3ow" colspan="2">15%</td> </tr> <tr> <td class="tg-0pky">Improvement</td> <td class="tg-c3ow">25%</td> <td class="tg-c3ow">0%</td> <td class="tg-c3ow" colspan="2">40%</td> <td class="tg-c3ow">20%</td> <td class="tg-c3ow">30%</td> </tr> <tr> <td class="tg-0pky">Robustness</td> <td class="tg-c3ow">25%</td> <td class="tg-c3ow">5%</td> <td class="tg-c3ow" colspan="2">25%</td> <td class="tg-c3ow" colspan="2">25%</td> </tr> </table> --- count: false # Table of Contents </br> .font130[ 1. [BITSS](#about-bitss) 2. [Why Reproducibility](#motivation) 3. [ACRE Guidelines](#guidelines) 4. [**ACRE Platform**](#platform) ] --- background-image: url(Images/Homepage.png) background-size: contain name: platform # [socialsciencereproduction.org](https://www.socialsciencereproduction.org) --- # Ok, I Am Interested. What's Next? You can check the ACRE Guidelines, and also contribute if you want: https://bitss.github.io/ACRE/ <iframe src="https://bitss.github.io/ACRE/" width="100%" height="350px"></iframe> .center[ A beta version of the platform will be online by early November .font120[ [**Sign up here**](https://forms.gle/yZivWcwijCzEhrBU6) ] if you would like to be part of our beta in the fall. ] --- class: center, middle count: false .font180[**Acknowledgements**] </br> -- Arnold Ventures -- Everybody who has participated in the pilots so far: -- Ted Miguel's Graduate Development Economic Course (2019, 2020) - UC Berkeley -- Dina Pomeranz undergraduate thesis for Marc Richter - University of Zurich -- Slides template: [Grant McDermott](https://github.com/uo-ec510-2020-spring/lectures).