
Research Methods Tutorial
A doctoral-level guide to the analysis methods and skills behind each of the five research workstreams. Use this as a tutorial for performing the applied research that aligns with your assigned theme.
Data Quality & Reproducibility
Reconcile counts, missingness, duplicates, linkage, and version rules.
Datasets: Inventory, crosswalk, codebook, audit metrics, all active instruments
What this workstream does
This workstream ensures every analysis rests on a trustworthy, well-documented foundation. You will audit the dataset for completeness and consistency, reconcile duplicate participant records, characterize missing data, and build reproducible pipelines so any finding can be regenerated exactly from the raw data.
Doctoral-Level Analysis Methods
Data Profiling & Audit Metrics
What it is: Computing completeness, uniqueness, validity, and consistency metrics across every variable in the dataset.
When to use: At the start of any analysis and before reporting any count or rate.
Steps:
- Profile each variable: count non-null, distinct, and out-of-range values.
- Compute missingness and duplicate rates per instrument.
- Flag variables failing validity rules and document them in a data dictionary.
- Produce a one-page audit summary table that can be regenerated from code.
Duplicate Detection & Record Linkage
What it is: Identifying records that refer to the same participant using deterministic rules and probabilistic linkage models.
When to use: Whenever participant records span multiple instruments or collection waves.
Steps:
- Block records on stable identifiers (email, name, DOB) to reduce comparison space.
- Apply deterministic exact-match rules first to capture obvious duplicates.
- Use probabilistic linkage (Fellegi-Sunter) with match weights for fuzzy matches.
- Clerically review borderline pairs and document the linkage decision rule.
Missing Data Analysis
What it is: Characterizing why values are missing and how that affects analysis validity.
When to use: Before any model fitting or reporting on variables with missingness.
Steps:
- Visualize the missing-data pattern (e.g., aggr plot, missingness map).
- Run Little's MCAR test to distinguish MCAR from MAR/MNAR.
- Decide on a strategy: complete-case, multiple imputation, or sensitivity analysis.
- Document the assumed missingness mechanism for every key variable.
Data Harmonization & Crosswalks
What it is: Mapping variables from different instruments or time points onto a common, analysis-ready schema.
When to use: When combining data across instruments, sites, or collection rounds.
Steps:
- Build a codebook documenting each source variable and its scale.
- Construct a crosswalk mapping each source variable to a target variable.
- Apply recodes, unit conversions, and category collapses consistently.
- Version the harmonized dataset and record the pipeline that produced it.
Reproducible Workflows
What it is: Literate-programming pipelines that regenerate all outputs from raw data with a single command.
When to use: For every analysis destined for a report, publication, or audit.
Steps:
- Store raw data as read-only and never edit it by hand.
- Write the full pipeline in a version-controlled notebook (Quarto / R Markdown / Jupyter).
- Set a random seed and declare package versions for stability.
- Render the final report from a clean environment to confirm reproducibility.
Skills You Will Build
Applied Steps for Your Project
- Begin with the inventory and codebook to understand every instrument.
- Run a full data profile and record audit metrics in a versioned report.
- Identify and reconcile duplicate records using deterministic then probabilistic linkage.
- Characterize missingness and document the mechanism for each key variable.
- Publish a reproducible pipeline that regenerates your audit from raw data.
Further Reading
- van der Loo & de Jonge (2018), Statistical Data Cleaning with R
- Little & Rubin (2019), Statistical Analysis with Missing Data, 3rd ed.
- Wilson et al. (2017), Good Enough Practices for Scientific Computing

Psychometrics & Well-Being
Evaluate measurement structure, reliability, validity, and comparability.
Datasets: DS3, DS3-v23, DS9
What this workstream does
This workstream asks whether the well-being scales actually measure what they claim to measure, and whether they do so fairly across groups. You will evaluate reliability, uncover the latent structure of the scales, model item-level performance with IRT, test measurement invariance, and fit structural models linking latent constructs to outcomes.
Doctoral-Level Analysis Methods
Classical Test Theory & Reliability
What it is: Estimating how consistently a scale measures its construct across items and occasions.
When to use: Early in scale evaluation, before interpreting composite scores.
Steps:
- Compute internal consistency (alpha, omega) for each scale.
- Estimate test-retest reliability where repeat data exist.
- Inspect item-total correlations and drop items that weaken the scale.
- Report reliability with a confidence interval, not a point estimate.
Factor Analysis (CFA / EFA)
What it is: Testing or discovering the latent factors that explain item covariation.
When to use: To confirm a hypothesized structure (CFA) or explore an unknown one (EFA).
Steps:
- Specify the measurement model from theory (CFA) or extract factors (EFA).
- Evaluate fit: CFI/TLI > .95, RMSEA < .06, SRMR < .08.
- Inspect factor loadings and residual correlations for local dependence.
- Respecify only with theoretical justification and document each change.
Item Response Theory (IRT)
What it is: Modeling the probability of a given response as a function of the latent trait and item parameters.
When to use: When you need item-level precision or to link scales across forms.
Steps:
- Choose a model: 1PL/2PL/3PL for binary, GRM for ordinal Likert items.
- Estimate item difficulty, discrimination, and (if used) guessing.
- Examine item information and test information curves.
- Flag items with low discrimination or DIF for review.
Measurement Invariance & DIF
What it is: Testing whether a scale measures the same construct the same way across groups.
When to use: Before comparing scale scores across demographic or cohort groups.
Steps:
- Test configural, then metric, then scalar invariance in sequence.
- Compare fit across steps; a meaningful drop flags non-invariance.
- Locate non-invariant items via DIF analysis (e.g., lordif, mirt).
- Decide on partial invariance or item removal and report the decision.
Structural Equation Modeling (SEM)
What it is: Estimating a system of relationships among latent and observed variables simultaneously.
When to use: To test theoretical models linking well-being constructs to outcomes.
Steps:
- Specify the measurement model, then add the structural paths.
- Estimate with a robust estimator (e.g., MLR) for non-normal data.
- Evaluate overall and component fit; check path significance.
- Test alternative models and report a model comparison table.
Skills You Will Build
Applied Steps for Your Project
- Start with DS3 / DS3-v23 / DS9 and the scale codebook.
- Compute reliability for every scale and flag weak items.
- Run a CFA for each scale and evaluate fit against thresholds.
- Test measurement invariance across key demographic groups.
- Fit an SEM linking the well-being latent factors to outcomes.
Further Reading
- DeVellis (2016), Scale Development: Theory and Applications, 4th ed.
- Brown (2015), Confirmatory Factor Analysis for Applied Research, 2nd ed.
- Embretson & Reise (2013), Item Response Theory for Psychologists

Engagement & Verified Outcomes
Examine participation depth, hours, planning, persistence, and production.
Datasets: DS1, DS2, DS6, DS7, DS10, verified-producer registry
What this workstream does
This workstream quantifies how deeply participants engage and whether that engagement produces verified outcomes. You will model how long people persist, how their engagement changes over time, whether distinct subgroups follow different trajectories, and whether program exposure causally improves outcomes — all while handling the count-based, nested nature of participation data.
Doctoral-Level Analysis Methods
Survival / Time-to-Event Analysis
What it is: Modeling the time until an event (e.g., dropout) and comparing risk across groups.
When to use: When the outcome is time-to-dropout or time-to-first-verified-output.
Steps:
- Define the event and the time origin; handle right-censoring.
- Estimate Kaplan-Meier survival curves by group.
- Fit a Cox model and check the proportional hazards assumption.
- Report hazard ratios with confidence intervals.
Longitudinal & Multilevel Modeling
What it is: Modeling repeated observations nested within participants, accounting for within-person correlation.
When to use: When you have repeated measures of engagement over time.
Steps:
- Specify fixed and random effects (intercepts and slopes).
- Choose a covariance structure and an appropriate estimator.
- Model time flexibly (linear, polynomial, or spline).
- Interpret fixed effects and variance components.
Trajectory & Growth Mixture Modeling
What it is: Identifying unobserved subgroups that follow distinct trajectories over time.
When to use: When you suspect heterogeneous engagement patterns behind an average trend.
Steps:
- Fit a latent class growth analysis with increasing numbers of classes.
- Compare fit (BIC, entropy) and class sizes to choose a solution.
- Characterize each class by its trajectory shape.
- Validate classes against external covariates and outcomes.
Quasi-Experimental & Causal Designs
What it is: Approximating experimental estimates of a program effect from observational data.
When to use: When randomization is not possible and you must control for confounding.
Steps:
- Estimate propensity scores and check balance after matching.
- Apply difference-in-differences with parallel-trends support.
- Use interrupted time series with a clear intervention point.
- Report sensitivity analyses for hidden confounding.
Count & Dose-Response Models
What it is: Regression for count outcomes and the shape of the dose-response relationship.
When to use: When the outcome is a count of actions or a cumulative dose.
Steps:
- Compare Poisson vs. negative binomial via dispersion tests.
- Use zero-inflated models when excess zeros are present.
- Model the dose-response curve (linear, log, or spline).
- Report incidence rate ratios and predicted counts.
Skills You Will Build
Applied Steps for Your Project
- Define the engagement and verified-outcome variables from DS1, DS2, DS6, DS7, DS10.
- Model persistence with survival analysis and compare groups.
- Fit a multilevel model of engagement trajectories over time.
- Test for distinct trajectory classes with growth mixture modeling.
- Estimate the causal effect of exposure with a quasi-experimental design.
Further Reading
- Singer & Willett (2003), Applied Longitudinal Data Analysis
- Therneau & Grambsch (2000), Modeling Survival Data
- Morgan & Winship (2015), Counterfactuals and Causal Inference, 2nd ed.
Purpose & Leadership Narratives
Analyze how participants describe purpose, principles, leadership, and readiness.
Datasets: DS1, DS5, DS8, DS8 companion responses, DS10
What this workstream does
This workstream makes sense of how participants put purpose and leadership into their own words. You will code narratives thematically, optionally build grounded theory, apply the five-principles framework, ensure coding is trustworthy through double-coding, and scale insight across thousands of responses with computational text analysis.
Doctoral-Level Analysis Methods
Thematic Analysis
What it is: A flexible method for identifying, analyzing, and reporting patterns (themes) within narrative data.
When to use: For most qualitative analyses of open-ended purpose and leadership responses.
Steps:
- Familiarize yourself by reading all responses thoroughly.
- Generate initial codes inductively and/or deductively from the five principles.
- Group codes into candidate themes and review against the dataset.
- Define, name, and write up each theme with supporting excerpts.
Grounded Theory & Constant Comparison
What it is: Building explanatory theory iteratively from the data using constant comparison and memoing.
When to use: When the goal is theory generation rather than description alone.
Steps:
- Open-code the first transcripts and memo reflexively.
- Constantly compare new data to emerging codes and categories.
- Use axial coding to relate categories; selective coding to integrate around a core category.
- Saturate categories and write a theoretical narrative.
Framework Analysis
What it is: Applying a pre-specified framework (e.g., the five principles) to structure and compare data.
When to use: When you have an a priori framework and want systematic, comparable summaries.
Steps:
- Define the framework matrix (themes by participants).
- Index transcripts to the framework categories.
- Chart data into the matrix and summarize per cell.
- Interpret across rows and columns, mapping patterns and outliers.
Qualitative Coding & Intercoder Reliability
What it is: Systematic codebook development and double-coding to demonstrate trustworthy interpretation.
When to use: Whenever you report themes that should be reproducible by another coder.
Steps:
- Draft a codebook with definitions, inclusion/exclusion, and examples.
- Double-code a sample (e.g., 20%) independently.
- Calculate Cohen's kappa and reconcile disagreements.
- Refine the codebook and apply it to the full dataset.
Computational Text Analysis
What it is: Using NLP to surface patterns across large narrative corpora that manual coding cannot reach.
When to use: When the volume of text exceeds practical manual coding.
Steps:
- Clean and tokenize text; remove stopwords and standardize case.
- Fit a topic model (LDA) and inspect top terms per topic.
- Run sentiment analysis and examine embeddings for semantic clusters.
- Triangulate computational findings with manual qualitative themes.
Skills You Will Build
Applied Steps for Your Project
- Pull narrative fields from DS1, DS5, DS8, DS8 companion responses, DS10.
- Read a sample to familiarize and draft an initial codebook.
- Apply thematic and/or framework coding using the five principles.
- Double-code a sample and report intercoder reliability.
- Scale insight with topic modeling across the full corpus.
Further Reading
- Braun & Clarke (2006, 2022), Thematic Analysis: A Practical Guide
- Charmaz (2014), Constructing Grounded Theory, 2nd ed.
- Grimmer, Roberts & Stewart (2022), Text as Data

Implementation Equity & Reach
Assess who is reached, which priorities emerge, and how programs can improve.
Datasets: Audit metrics, geography, institutions, DS10, approved aggregate outputs
What this workstream does
This workstream asks who the program reaches, who it misses, and how to close the gap. You will apply the RE-AIM framework, quantify disparities with equity decomposition, map reach with spatial analysis, disaggregate outcomes through an intersectional lens, and ground the work in participatory evaluation with affected communities.
Doctoral-Level Analysis Methods
RE-AIM Framework
What it is: A planning and evaluation framework spanning Reach, Effectiveness, Adoption, Implementation, and Maintenance.
When to use: To plan or evaluate a program for real-world impact and generalizability.
Steps:
- Define each RE-AIM dimension and its measures for your program.
- Compute Reach as the proportion of the intended population engaged.
- Assess Effectiveness and Adoption across settings and staff.
- Evaluate Implementation fidelity and Maintenance over time.
Health Equity Decomposition
What it is: Quantifying the size of a disparity and how much of it is explained by underlying factors.
When to use: When comparing outcomes across socio-demographic groups.
Steps:
- Compute rate ratios and rate differences for each group.
- Estimate a concentration index and plot the concentration curve.
- Apply Oaxaca-Blinder decomposition to attribute the gap to covariates.
- Report both absolute and relative measures of inequity.
Spatial Analysis & GIS
What it is: Mapping outcomes and detecting geographic clusters of need or reach.
When to use: When geography shapes access or exposure.
Steps:
- Geocode participant and program locations; aggregate to areas.
- Map choropleths of reach and outcomes.
- Test for spatial clustering with Moran's I and Getis-Ord Gi*.
- Use small-area estimation where direct estimates are unstable.
Stratified & Intersectional Analysis
What it is: Disaggregating outcomes by multiple, intersecting identities to reveal layered inequities.
When to use: Before concluding a program is equitable across groups.
Steps:
- Disaggregate outcomes by key demographic strata.
- Examine intersectional subgroups (e.g., race x gender x geography).
- Test for interaction and report stratum-specific effects.
- Co-interpret findings with affected community partners.
Participatory & Community-Based Evaluation
What it is: Engaging affected communities as co-researchers in defining, conducting, and interpreting evaluation.
When to use: When the evaluation must serve and be trusted by the affected community.
Steps:
- Build a community advisory group and agree on evaluation questions.
- Co-develop measures that are meaningful to the community.
- Share preliminary findings with the community for interpretation.
- Co-author recommendations and an action plan for improvement.
Skills You Will Build
Applied Steps for Your Project
- Pull audit metrics, geography, institutions, DS10, and approved aggregate outputs.
- Apply RE-AIM to characterize reach, adoption, and maintenance.
- Quantify disparities with equity decomposition and rate ratios.
- Map reach and clusters with GIS and small-area estimation.
- Co-interpret findings with community partners and recommend improvements.
Further Reading
- Glasgow, Vogt & Boles (1999), Evaluating the Public Health Impact of Health Promotion (RE-AIM)
- O'Donnell et al. (2007), Explorations in Health Inequality (concentration index)
- Wallerstein & Duran (2017), Community-Based Participatory Research, 3rd ed.
