Natural Language Processing for Implementation Research

Led by Kylie Anglin and Brian Wright

Assessing intervention implementation fidelity is critical for interpreting the results of replication studies. A key assumption in replication is the assumption of treatment stability - we assume that an intervention is implemented in the same manner across participants, sites, and studies. Unfortunately, testing this assumption is often expensive, time-consuming, and even infeasible when replications occur in many locations across many time points. Because of these challenges, our team is developing, low-cost, scalable methods of monitoring implementation in field settings using natural language processing.

Semantic Similarity

Learn more in our working paper: Anglin, K.L., Wong, V.C. (2020) Using Semantic Similarity to Assess Adherence and Replicability of Intervention Delivery.

Assessing treatment consistency in replication studies is a particularly vexing challenge in education where treatments are often language-based; for example, when the treatment is a lecture delivered by a teacher, a therapy session delivered by a school psychologist, or a teacher coaching session delivered by an instruction team leader. We propose a new automated method of assessing the replication of treatments which is generalizable to any language-based intervention. The method does not require researchers to observe each treatment session, but treatment sessions must be recorded and transcribed. The transcriptions may then be compared to one another using natural language processing techniques. In particular, we propose applying a method commonly used in the detection of plagiarism: measuring “document similarity.” In the plagiarism case, researchers want to quantify the similarity of two or more texts and need some method of detecting derivative text (or speech), even if the plagiarizer has made inconsequential changes to the text. The primary difference between assessing plagiarism and assessing treatment replication in language-based interventions is the definition of the ideal; in plagiarism, a derivative text is bad, but in a replication, a derivative text is good.