Every line of real Python in this course comes from one reproduction: the BSZ step
of fhoces/opa-prop40, folder
bsz-analysis/py/. BSZ is a paper on the wealth and taxes of California
billionaires. Its reproduction has two steps. Step 1 runs shared .sql
files over the raw data (daily Forbes snapshots, SEC filings), from Python and from R,
and checks that the two exports agree. Step 2 takes the paper's public workbook to every
number the R pipeline exports, with a Python twin written to be compared against the R
side function by function.
This page lists where that code appears in the course, module by module and then file by
file. In modules 1 to 5 the excerpts are condensed to a few lines, the economics is left
out, and each slide shows one idiom next to its R twin. Module 7 works on the files
themselves, with drills graded by check.py on a synthetic panel in the same
shape as the real data (the real data are confidential).
| # | Where in the course | BSZ files and functions | Idiom, with its R twin |
|---|---|---|---|
| 1 |
Slides: A Dict of Functions, Three Comprehension Moves, Run the Real Function, Calling It
Concepts: In the wild: real code from one reproduction
Drills: Q6 the pattern on the ride data, Q7 the real score_tab5_cell(), run
|
load_bundle.py: LOADERS, FORM4_INT, main()compute_shortrunseries.py: srs_summary_2025()compute_tab5.py: score_tab5_cell(), copied whole and run |
A dict of functions as a registry (a named list of functions); argv or DEFAULT; a set comprehension with two for clauses (paste0 over rep(each=)); dict merge with ** (c() then unlist()); keyword arguments with defaults; x if cond else 0 (if (cond) x else 0); a dict returned (a named list); **settings as do.call() |
| 2 | Slides: Filter, Stable Sort, and a Summary Row, numpy for R's Vector Idioms, The Explorer's Grid as a DataFrame, Querying the Grid Concepts: In the wild: filter, stable sort, summary row, and numpy Drills: Q6 the pattern on the ride data, Q7 the real explorer grid, rebuilt and queried | figures.py: build_fig_a1()tables.py: build_tab2(), _rsum()compute_pareto.py: compute_pareto_missing(), compute_fig8_laffer()site_exports.py: build_site_grid(), with score_tab5_cell() |
.notna(); kind="stable" (R's order() is stable, pandas' default is not); a one-row frame appended with concat (bind_rows); np.append(x[1:], 0) as c(x[-1], 0); nansum as na.rm = TRUE; which() as flatnonzero; the np.arange float-step trap against seq(); summation order; itertools.product into a list of dicts and one DataFrame (expand.grid() + bind_rows()), then assign, filter, groupby, nsmallest, pivot |
| 3 |
Slides: A Join Without pandas, match() Two Ways
Concepts: In the wild: a join without pandas, and match() two ways
Drill: exercise.py, Q6
|
check_common.py: compare_keyed()compute_shortrunseries.py: srs_panel_columns()tables.py: build_tab2(), build_tab3() (pick()) |
A join on a composite key with dicts and sets: both anti-joins and duplicate keys counted, then the merge(indicator=True) twin; match() as flatnonzero and as set_index().loc[], and how each fails; the "exactly one row" guard |
| 4 | Slides: A Parameter Calibrated, Not Regressed Concepts: In the wild: a parameter calibrated, not regressed Drill: exercise.py, Q6 | compute_pareto.py: compute_pareto_missing() |
A power-law exponent as a ratio of means pinned at an anchor row, matching a spreadsheet cell by cell; .copy() before writing into a slice (R copies on modify); broadcasting as recycling; the regression twin smf.ols("np.log(y) ~ np.log(x)"), and when the two disagree |
| 5 | Slides: One Function Per Stage, A Check That Fails Loudly Concepts: In the wild: one function per stage, and a check that fails loudly Drill: exercise.py, Q7 | run_export.py: run()check_common.py: write_report() |
Inputs, shared step, computations, outputs as separate functions; None defaults resolved inside; the __main__ guard; a check as a dict (name, ok, n, maxdiff, note); PASS/FAIL lines and an exit code (quit(status = 1)) |
| 7 |
Part 1 slides: load_bundle.py: Reading a Big File in Pieces, Exports That Never Change by Accident, The R Twin, check_rtb_ca.py: Align on Keys, Then Compare (plus the bundle_paths.py, NULL-handling, parameter-binding and run_sql.py slides between them)
Concepts: module 7, part 1
Drills: Q1 loader, Q2 SQL runner and export, Q3 answer-key comparison, Q4 R-vs-Python parity
|
bundle_paths.py, load_bundle.py, run_sql.py, check_rtb_ca.py, R/run_sql.R, sql/01_rtb_ca.sql (a copy ships in the module) |
Chunked csv.reader with empty-to-None (read_csv_chunked, dbAppendTable); executemany with ? placeholders; indexes after the load; executescript vs execute; deterministic CSV exports (sort keys, repr floats); merge(indicator=True); tolerance vs exact equality; exit codes |
| 7 |
Part 2 slides: Function by Function, File by File, openpyxl: Reading a Sheet the Way readxl Does, pandas Twins of the R Idioms, The Parity Test
Concepts: Part 2: translating an R pipeline into a Python twin
Drills: Q5 positional sheet reader, Q6 a cell as as.numeric() sees it, Q7 a dplyr pipeline in pandas, Q8 relative-tolerance parity
|
ingest_excel.py, excel_cells.py, workbook_db.py, compute_*.py, tables.py, figures.py, vintage.py, run_export.py, their R twins in R/, tests/testthat/helper-py-parity.R, sql/02_data_sec_agg.sql |
One Python file per R file with the same function names; openpyxl with data_only=True and a frame indexed like the sheet; to_num() as suppressWarnings(as.numeric()); isin, duplicated(subset=), size(), sum() as %in%, duplicated(), n(), sum(na.rm = TRUE); parity at 1e-9 * max(1, |r|) |
Modules 6 and 7 are the two deliberate breaks from the ride-sharing data. Module 6 reads a
different package (aiscen, from another OPA project) and is not part of this map.
The same map the other way round, for a reader who starts from the repository. Each file links to its source in opa-prop40.
File in bsz-analysis/ | What the course takes from it | Modules |
|---|---|---|
py/bundle_paths.py | one place that knows where every file lives, with a useful error when one is missing | 7 |
py/load_bundle.py | the LOADERS registry and main(); the FORM4_INT set comprehension; the chunked loader with explicit NULLs, parameter binding and indexes after the load | 1, 7 |
py/run_sql.py | executescript against execute; sorted, repr-formatted CSV exports; argparse | 7 |
py/check_rtb_ca.py | align on keys, compare with a tolerance, write a report, exit with a code | 7 |
py/check_common.py | compare_keyed(), a join with dicts and sets; write_report(), PASS/FAIL lines and the exit code | 3, 5 |
py/ingest_excel.py, py/excel_cells.py | a sheet read by position with openpyxl; a cell as R's as.numeric() sees it | 7 |
py/workbook_db.py | two sheets into SQLite with to_sql, the shared .sql files run there | 7 |
py/compute_pareto.py | numpy for R's vector idioms (np.append, nansum, flatnonzero, errstate, the seq() grid); the power law calibrated at an anchor row | 2, 4, 7 |
py/compute_shortrunseries.py | dict merge with ** and unlist()-style keys; match() through flatnonzero | 1, 3, 7 |
py/tables.py | a summary row appended with concat; _rsum() and summation order; match() through set_index().loc[]; the pick() guard | 2, 3, 7 |
py/figures.py | .notna() filter and a stable sort | 2, 7 |
py/compute_tab5.py | score_tab5_cell(), the estimating function behind the explorer, copied whole and run: keyword arguments with defaults, a one-line if/else, a dict returned | 1, 2 |
py/site_exports.py | build_site_grid(): every dial setting through itertools.product, one dict per setting, one DataFrame | 2 |
py/vintage.py | literals that vary by workbook vintage, switched by an environment variable | 7 |
py/run_export.py | run(): one function per stage, the __main__ guard; the role of _targets.R | 5, 7 |
R/run_sql.R | the same .sql file from R, for the parity check | 7 |
tests/testthat/helper-py-parity.R | compare_parity_frames(): relative tolerance on numbers, strict on everything else | 7 |
sql/01_rtb_ca.sql, sql/02_data_sec_agg.sql | the queries the Python loads, runs and checks; treated as a black box here and taught in sql-industry-prep, module 6 | 7 |
The short path through the BSZ code is the "In the wild" slide of each module in order, 1 to 5, then module 7. Each of the five slides is one idiom and takes a few minutes; module 7 is the full set of files with the drills. The SQL course's module 6 covers what the two queries compute, and pairs with module 7 here.