Evaluating impact or effects of treatment programs in psychology or education (or social science more generally) involves building a statistical model which represents the data generating process—or, how we believe the results truly came to be. The validity of such a model depends on many factors, some of which include:
- Experimental design: to what extent can we reduce or remove irrelevant noise through experimental design, random assignment, statistical control, or physical control? Multilevel models are often used to statistically control for individual variation over time, or differences due to local, social conditions in classrooms, schools, or clinics.
- Construct validity: to what extent do the data collected or measures used represent true status on what we intend to measure? Ideally, we have theories to describe how our measures should represent what we intend to measure (e.g., constructs such as reading comprehension, linguistic understanding, academic success).
- Diversity, equity, or bias: to what extent might treatment or measurement effects differ across groups? The interventions we implement or the measures we use may function differently for different students. Statistical models can be used to evaluate equity or bias in our studies.
Related work:
Wolbers, K., Dostal, H., Graham, S., Branum-Martin, L., & Holcomb, L. (2022) “Specialized writing instruction for deaf students: a randomized controlled trial.” Exceptional Children 88(2), 185-204. doi: 10.1177/00144029211050849
Lederberg, A., Connor, C.M., Burke, V., Antia, S., & Branum-Martin, L., (2022, July) “Language Arts Instruction in Classrooms with Deaf and Hard of Hearing Students.” In K. Werfel (chair), Literacy and literacy instruction for students who are deaf and hard of hearing: A Tribute to Carol McDonald Connor. Paper presented at the 29th annual meeting of the Society for the Scientific Study of Reading, Newport Beach, CA.
Zettler-Greeley, C.M., Bailet, L.L., Murphy, S.P., DeLucca, T.L., & Branum-Martin, L. (2018) “Efficacy of the Nemours Brightstart! Early Literacy Program: Treatment outcomes from a randomized trial with at-risk prekindergartners.” Early Education and Development 29(6), 873-892. doi: 10.1080/10409289.2018.1475202.
Branum-Martin, L., Sun, C., & Calhoon, M.B. (2018, July) “Measuring treatment impact across different outcomes, settings, and designs.” In L. Branum-Martin (chair), Methods for disentangling complexities in reading interventions. Paper presented at the 25th annual meeting of the Society for the Scientific Study of Reading, Brighton, UK.
Sun, C.Y., Branum-Martin, L., & Calhoon, M.B. (2017, May). Evaluating intervention programs across multiple outcomes: Multivariate latent growth modeling approaches. Poster presented at the 7th Modern Modeling Methods (M3) conference, Connecticut, USA.
Wolbers, K., Dostal, H., Graham, S., Branum-Martin, L., Kilpatrick, J. & Saulsburry, R. (2018) “Strategic and interactive writing instruction: An efficacy study in grades 3-5.” Journal of Educational and Developmental Psychology 8(1), 99-117. doi:10.5539/jedp.v8n1p99