In crisis or need support? Call or text 988 - 988 Suicide & Crisis Lifeline (24/7)

Voice Up®

Strong — sunflower anchor image for the DrPH Research Methods Tutorial
Page Narrator
DRPH APPLIED RESEARCH METHODS

Research Methods Tutorial

A doctoral-level guide to the analysis methods and skills behind each of the five research workstreams. Use this as a tutorial for performing the applied research that aligns with your assigned theme.

1
THEME 1

Data Quality & Reproducibility

Reconcile counts, missingness, duplicates, linkage, and version rules.

Datasets: Inventory, crosswalk, codebook, audit metrics, all active instruments

What this workstream does

This workstream ensures every analysis rests on a trustworthy, well-documented foundation. You will audit the dataset for completeness and consistency, reconcile duplicate participant records, characterize missing data, and build reproducible pipelines so any finding can be regenerated exactly from the raw data.

Doctoral-Level Analysis Methods

Data Profiling & Audit Metrics

What it is: Computing completeness, uniqueness, validity, and consistency metrics across every variable in the dataset.

When to use: At the start of any analysis and before reporting any count or rate.

Steps:

  1. Profile each variable: count non-null, distinct, and out-of-range values.
  2. Compute missingness and duplicate rates per instrument.
  3. Flag variables failing validity rules and document them in a data dictionary.
  4. Produce a one-page audit summary table that can be regenerated from code.
Software:
R: DataExplorer, skimr
Python: pandas, ydata-profiling

Duplicate Detection & Record Linkage

What it is: Identifying records that refer to the same participant using deterministic rules and probabilistic linkage models.

When to use: Whenever participant records span multiple instruments or collection waves.

Steps:

  1. Block records on stable identifiers (email, name, DOB) to reduce comparison space.
  2. Apply deterministic exact-match rules first to capture obvious duplicates.
  3. Use probabilistic linkage (Fellegi-Sunter) with match weights for fuzzy matches.
  4. Clerically review borderline pairs and document the linkage decision rule.
Software:
R: RecordLink, fastLink
Python: recordlinkage

Missing Data Analysis

What it is: Characterizing why values are missing and how that affects analysis validity.

When to use: Before any model fitting or reporting on variables with missingness.

Steps:

  1. Visualize the missing-data pattern (e.g., aggr plot, missingness map).
  2. Run Little's MCAR test to distinguish MCAR from MAR/MNAR.
  3. Decide on a strategy: complete-case, multiple imputation, or sensitivity analysis.
  4. Document the assumed missingness mechanism for every key variable.
Software:
R: mice, naniar
Stata: mi commands

Data Harmonization & Crosswalks

What it is: Mapping variables from different instruments or time points onto a common, analysis-ready schema.

When to use: When combining data across instruments, sites, or collection rounds.

Steps:

  1. Build a codebook documenting each source variable and its scale.
  2. Construct a crosswalk mapping each source variable to a target variable.
  3. Apply recodes, unit conversions, and category collapses consistently.
  4. Version the harmonized dataset and record the pipeline that produced it.
Software:
R: labelled, sjlabelled
General: OpenRefine

Reproducible Workflows

What it is: Literate-programming pipelines that regenerate all outputs from raw data with a single command.

When to use: For every analysis destined for a report, publication, or audit.

Steps:

  1. Store raw data as read-only and never edit it by hand.
  2. Write the full pipeline in a version-controlled notebook (Quarto / R Markdown / Jupyter).
  3. Set a random seed and declare package versions for stability.
  4. Render the final report from a clean environment to confirm reproducibility.
Software:
Quarto / R Markdown
Git + GitHub
renv / pip freeze

Skills You Will Build

SQL & relational data querying
Python (pandas) or R (dplyr/tidyverse) data wrangling
Git version control
Data validation & ETL pipeline design
Codebook & data dictionary authoring
Reproducible reporting (Quarto / R Markdown)

Applied Steps for Your Project

  1. Begin with the inventory and codebook to understand every instrument.
  2. Run a full data profile and record audit metrics in a versioned report.
  3. Identify and reconcile duplicate records using deterministic then probabilistic linkage.
  4. Characterize missingness and document the mechanism for each key variable.
  5. Publish a reproducible pipeline that regenerates your audit from raw data.

Further Reading

  • van der Loo & de Jonge (2018), Statistical Data Cleaning with R
  • Little & Rubin (2019), Statistical Analysis with Missing Data, 3rd ed.
  • Wilson et al. (2017), Good Enough Practices for Scientific Computing
A white goose standing bright against dark water — standing out where it is hard to stand at all
2
THEME 2

Psychometrics & Well-Being

Evaluate measurement structure, reliability, validity, and comparability.

Datasets: DS3, DS3-v23, DS9

What this workstream does

This workstream asks whether the well-being scales actually measure what they claim to measure, and whether they do so fairly across groups. You will evaluate reliability, uncover the latent structure of the scales, model item-level performance with IRT, test measurement invariance, and fit structural models linking latent constructs to outcomes.

Doctoral-Level Analysis Methods

Classical Test Theory & Reliability

What it is: Estimating how consistently a scale measures its construct across items and occasions.

When to use: Early in scale evaluation, before interpreting composite scores.

Steps:

  1. Compute internal consistency (alpha, omega) for each scale.
  2. Estimate test-retest reliability where repeat data exist.
  3. Inspect item-total correlations and drop items that weaken the scale.
  4. Report reliability with a confidence interval, not a point estimate.
Software:
R: psych
Stata: alpha

Factor Analysis (CFA / EFA)

What it is: Testing or discovering the latent factors that explain item covariation.

When to use: To confirm a hypothesized structure (CFA) or explore an unknown one (EFA).

Steps:

  1. Specify the measurement model from theory (CFA) or extract factors (EFA).
  2. Evaluate fit: CFI/TLI > .95, RMSEA < .06, SRMR < .08.
  3. Inspect factor loadings and residual correlations for local dependence.
  4. Respecify only with theoretical justification and document each change.
Software:
R: lavaan, psych
Mplus

Item Response Theory (IRT)

What it is: Modeling the probability of a given response as a function of the latent trait and item parameters.

When to use: When you need item-level precision or to link scales across forms.

Steps:

  1. Choose a model: 1PL/2PL/3PL for binary, GRM for ordinal Likert items.
  2. Estimate item difficulty, discrimination, and (if used) guessing.
  3. Examine item information and test information curves.
  4. Flag items with low discrimination or DIF for review.
Software:
R: mirt, ltm
Stata: irt

Measurement Invariance & DIF

What it is: Testing whether a scale measures the same construct the same way across groups.

When to use: Before comparing scale scores across demographic or cohort groups.

Steps:

  1. Test configural, then metric, then scalar invariance in sequence.
  2. Compare fit across steps; a meaningful drop flags non-invariance.
  3. Locate non-invariant items via DIF analysis (e.g., lordif, mirt).
  4. Decide on partial invariance or item removal and report the decision.
Software:
R: lavaan, mirt, lordif
Mplus

Structural Equation Modeling (SEM)

What it is: Estimating a system of relationships among latent and observed variables simultaneously.

When to use: To test theoretical models linking well-being constructs to outcomes.

Steps:

  1. Specify the measurement model, then add the structural paths.
  2. Estimate with a robust estimator (e.g., MLR) for non-normal data.
  3. Evaluate overall and component fit; check path significance.
  4. Test alternative models and report a model comparison table.
Software:
R: lavaan, semTools
Mplus

Skills You Will Build

Scale development & validation
Latent variable modeling
R (lavaan, psych, mirt) or Mplus
CFA/EFA specification and fit evaluation
Measurement invariance testing
Interpreting IRT parameters

Applied Steps for Your Project

  1. Start with DS3 / DS3-v23 / DS9 and the scale codebook.
  2. Compute reliability for every scale and flag weak items.
  3. Run a CFA for each scale and evaluate fit against thresholds.
  4. Test measurement invariance across key demographic groups.
  5. Fit an SEM linking the well-being latent factors to outcomes.

Further Reading

  • DeVellis (2016), Scale Development: Theory and Applications, 4th ed.
  • Brown (2015), Confirmatory Factor Analysis for Applied Research, 2nd ed.
  • Embretson & Reise (2013), Item Response Theory for Psychologists
A mallard descending toward water with wings wide open — courage in the landing, trust in the next surface
3
THEME 3

Engagement & Verified Outcomes

Examine participation depth, hours, planning, persistence, and production.

Datasets: DS1, DS2, DS6, DS7, DS10, verified-producer registry

What this workstream does

This workstream quantifies how deeply participants engage and whether that engagement produces verified outcomes. You will model how long people persist, how their engagement changes over time, whether distinct subgroups follow different trajectories, and whether program exposure causally improves outcomes — all while handling the count-based, nested nature of participation data.

Doctoral-Level Analysis Methods

Survival / Time-to-Event Analysis

What it is: Modeling the time until an event (e.g., dropout) and comparing risk across groups.

When to use: When the outcome is time-to-dropout or time-to-first-verified-output.

Steps:

  1. Define the event and the time origin; handle right-censoring.
  2. Estimate Kaplan-Meier survival curves by group.
  3. Fit a Cox model and check the proportional hazards assumption.
  4. Report hazard ratios with confidence intervals.
Software:
R: survival, survminer
Stata: st commands

Longitudinal & Multilevel Modeling

What it is: Modeling repeated observations nested within participants, accounting for within-person correlation.

When to use: When you have repeated measures of engagement over time.

Steps:

  1. Specify fixed and random effects (intercepts and slopes).
  2. Choose a covariance structure and an appropriate estimator.
  3. Model time flexibly (linear, polynomial, or spline).
  4. Interpret fixed effects and variance components.
Software:
R: lme4, nlme, geepack
Stata: mixed

Trajectory & Growth Mixture Modeling

What it is: Identifying unobserved subgroups that follow distinct trajectories over time.

When to use: When you suspect heterogeneous engagement patterns behind an average trend.

Steps:

  1. Fit a latent class growth analysis with increasing numbers of classes.
  2. Compare fit (BIC, entropy) and class sizes to choose a solution.
  3. Characterize each class by its trajectory shape.
  4. Validate classes against external covariates and outcomes.
Software:
R: lcmm, tidyLPA
Mplus

Quasi-Experimental & Causal Designs

What it is: Approximating experimental estimates of a program effect from observational data.

When to use: When randomization is not possible and you must control for confounding.

Steps:

  1. Estimate propensity scores and check balance after matching.
  2. Apply difference-in-differences with parallel-trends support.
  3. Use interrupted time series with a clear intervention point.
  4. Report sensitivity analyses for hidden confounding.
Software:
R: MatchIt, cobalt, marginaleffects
Stata: teffects

Count & Dose-Response Models

What it is: Regression for count outcomes and the shape of the dose-response relationship.

When to use: When the outcome is a count of actions or a cumulative dose.

Steps:

  1. Compare Poisson vs. negative binomial via dispersion tests.
  2. Use zero-inflated models when excess zeros are present.
  3. Model the dose-response curve (linear, log, or spline).
  4. Report incidence rate ratios and predicted counts.
Software:
R: MASS, pscl
Stata: nbreg, zinb

Skills You Will Build

Longitudinal data analysis
Survival analysis
Causal inference for observational data
R (survival, lme4, nlme, geepack) or Stata
Propensity score methods
Interpreting hazard ratios and mixed-effects coefficients

Applied Steps for Your Project

  1. Define the engagement and verified-outcome variables from DS1, DS2, DS6, DS7, DS10.
  2. Model persistence with survival analysis and compare groups.
  3. Fit a multilevel model of engagement trajectories over time.
  4. Test for distinct trajectory classes with growth mixture modeling.
  5. Estimate the causal effect of exposure with a quasi-experimental design.

Further Reading

  • Singer & Willett (2003), Applied Longitudinal Data Analysis
  • Therneau & Grambsch (2000), Modeling Survival Data
  • Morgan & Winship (2015), Counterfactuals and Causal Inference, 2nd ed.
4
THEME 4

Purpose & Leadership Narratives

Analyze how participants describe purpose, principles, leadership, and readiness.

Datasets: DS1, DS5, DS8, DS8 companion responses, DS10

What this workstream does

This workstream makes sense of how participants put purpose and leadership into their own words. You will code narratives thematically, optionally build grounded theory, apply the five-principles framework, ensure coding is trustworthy through double-coding, and scale insight across thousands of responses with computational text analysis.

Doctoral-Level Analysis Methods

Thematic Analysis

What it is: A flexible method for identifying, analyzing, and reporting patterns (themes) within narrative data.

When to use: For most qualitative analyses of open-ended purpose and leadership responses.

Steps:

  1. Familiarize yourself by reading all responses thoroughly.
  2. Generate initial codes inductively and/or deductively from the five principles.
  3. Group codes into candidate themes and review against the dataset.
  4. Define, name, and write up each theme with supporting excerpts.
Software:
NVivo, ATLAS.ti, Dedoose
R: RQDA

Grounded Theory & Constant Comparison

What it is: Building explanatory theory iteratively from the data using constant comparison and memoing.

When to use: When the goal is theory generation rather than description alone.

Steps:

  1. Open-code the first transcripts and memo reflexively.
  2. Constantly compare new data to emerging codes and categories.
  3. Use axial coding to relate categories; selective coding to integrate around a core category.
  4. Saturate categories and write a theoretical narrative.
Software:
NVivo, ATLAS.ti, MAXQDA

Framework Analysis

What it is: Applying a pre-specified framework (e.g., the five principles) to structure and compare data.

When to use: When you have an a priori framework and want systematic, comparable summaries.

Steps:

  1. Define the framework matrix (themes by participants).
  2. Index transcripts to the framework categories.
  3. Chart data into the matrix and summarize per cell.
  4. Interpret across rows and columns, mapping patterns and outliers.
Software:
NVivo, Excel framework matrices

Qualitative Coding & Intercoder Reliability

What it is: Systematic codebook development and double-coding to demonstrate trustworthy interpretation.

When to use: Whenever you report themes that should be reproducible by another coder.

Steps:

  1. Draft a codebook with definitions, inclusion/exclusion, and examples.
  2. Double-code a sample (e.g., 20%) independently.
  3. Calculate Cohen's kappa and reconcile disagreements.
  4. Refine the codebook and apply it to the full dataset.
Software:
NVivo, ATLAS.ti, Dedoose
R: irr

Computational Text Analysis

What it is: Using NLP to surface patterns across large narrative corpora that manual coding cannot reach.

When to use: When the volume of text exceeds practical manual coding.

Steps:

  1. Clean and tokenize text; remove stopwords and standardize case.
  2. Fit a topic model (LDA) and inspect top terms per topic.
  3. Run sentiment analysis and examine embeddings for semantic clusters.
  4. Triangulate computational findings with manual qualitative themes.
Software:
Python: spaCy, NLTK, gensim
R: tidytext, stm

Skills You Will Build

Qualitative coding (NVivo, ATLAS.ti, or Dedoose)
Codebook development & intercoder reliability
Mixed-methods integration
NLP (Python: spaCy, NLTK, gensim)
Trustworthiness criteria (credibility, transferability, dependability, confirmability)
Reflexivity and audit trails

Applied Steps for Your Project

  1. Pull narrative fields from DS1, DS5, DS8, DS8 companion responses, DS10.
  2. Read a sample to familiarize and draft an initial codebook.
  3. Apply thematic and/or framework coding using the five principles.
  4. Double-code a sample and report intercoder reliability.
  5. Scale insight with topic modeling across the full corpus.

Further Reading

  • Braun & Clarke (2006, 2022), Thematic Analysis: A Practical Guide
  • Charmaz (2014), Constructing Grounded Theory, 2nd ed.
  • Grimmer, Roberts & Stewart (2022), Text as Data
A northern hawk owl in flight with steady wings — seeing clearly what others miss
5
THEME 5

Implementation Equity & Reach

Assess who is reached, which priorities emerge, and how programs can improve.

Datasets: Audit metrics, geography, institutions, DS10, approved aggregate outputs

What this workstream does

This workstream asks who the program reaches, who it misses, and how to close the gap. You will apply the RE-AIM framework, quantify disparities with equity decomposition, map reach with spatial analysis, disaggregate outcomes through an intersectional lens, and ground the work in participatory evaluation with affected communities.

Doctoral-Level Analysis Methods

RE-AIM Framework

What it is: A planning and evaluation framework spanning Reach, Effectiveness, Adoption, Implementation, and Maintenance.

When to use: To plan or evaluate a program for real-world impact and generalizability.

Steps:

  1. Define each RE-AIM dimension and its measures for your program.
  2. Compute Reach as the proportion of the intended population engaged.
  3. Assess Effectiveness and Adoption across settings and staff.
  4. Evaluate Implementation fidelity and Maintenance over time.
Software:
RE-AIM calculators, Excel
R for custom metrics

Health Equity Decomposition

What it is: Quantifying the size of a disparity and how much of it is explained by underlying factors.

When to use: When comparing outcomes across socio-demographic groups.

Steps:

  1. Compute rate ratios and rate differences for each group.
  2. Estimate a concentration index and plot the concentration curve.
  3. Apply Oaxaca-Blinder decomposition to attribute the gap to covariates.
  4. Report both absolute and relative measures of inequity.
Software:
R: inequal, oaxaca
Stata: oaxaca, conindex

Spatial Analysis & GIS

What it is: Mapping outcomes and detecting geographic clusters of need or reach.

When to use: When geography shapes access or exposure.

Steps:

  1. Geocode participant and program locations; aggregate to areas.
  2. Map choropleths of reach and outcomes.
  3. Test for spatial clustering with Moran's I and Getis-Ord Gi*.
  4. Use small-area estimation where direct estimates are unstable.
Software:
QGIS, ArcGIS
R: sf, leaflet, spdep

Stratified & Intersectional Analysis

What it is: Disaggregating outcomes by multiple, intersecting identities to reveal layered inequities.

When to use: Before concluding a program is equitable across groups.

Steps:

  1. Disaggregate outcomes by key demographic strata.
  2. Examine intersectional subgroups (e.g., race x gender x geography).
  3. Test for interaction and report stratum-specific effects.
  4. Co-interpret findings with affected community partners.
Software:
R: survey, intersectionality packages
Stata: svy

Participatory & Community-Based Evaluation

What it is: Engaging affected communities as co-researchers in defining, conducting, and interpreting evaluation.

When to use: When the evaluation must serve and be trusted by the affected community.

Steps:

  1. Build a community advisory group and agree on evaluation questions.
  2. Co-develop measures that are meaningful to the community.
  3. Share preliminary findings with the community for interpretation.
  4. Co-author recommendations and an action plan for improvement.
Software:
Participatory toolkits (e.g., CBPR frameworks)
Qualitative + GIS tools

Skills You Will Build

GIS (QGIS, ArcGIS, R sf/leaflet)
Health equity measurement
RE-AIM application
Stratified & decomposition analysis
Community-based participatory research
Interpreting concentration indices and spatial statistics

Applied Steps for Your Project

  1. Pull audit metrics, geography, institutions, DS10, and approved aggregate outputs.
  2. Apply RE-AIM to characterize reach, adoption, and maintenance.
  3. Quantify disparities with equity decomposition and rate ratios.
  4. Map reach and clusters with GIS and small-area estimation.
  5. Co-interpret findings with community partners and recommend improvements.

Further Reading

  • Glasgow, Vogt & Boles (1999), Evaluating the Public Health Impact of Health Promotion (RE-AIM)
  • O'Donnell et al. (2007), Explorations in Health Inequality (concentration index)
  • Wallerstein & Duran (2017), Community-Based Participatory Research, 3rd ed.
A cormorant drying its wings after the dive — strength that comes from resting after effort

Voice Up® is a federally registered service mark (U.S. Reg. No. 8,148,220) owned by J Ellington Publishing LLC and operated by Voice Up Publishing Incorporated under permanent license.