Measuring the Impact of a Civics Program for Middle Schoolers at the New-York Historical
In 2023, I led an evaluation designed to measure the impact of the New York Historical’s Tang Academy for American Democracy, a program designed to introduce 6th graders to different models of democracy and sustain excitement about learning history through middle school.
Client: The New York Historical
My Role: Project Lead
Project timeline: January 2023-August 2023 (7 months)
I designed an evaluation with three phases that built on each other:
Phase 1: Planning
I collaboratively developed a Logic Model (see below) with TAAD program staff to define measurable outcomes of the program. These outcomes served as guideposts to develop instruments for the following phases.
Example questions from the open-ended assessment completed by students in pilot phase.
→
Phase 2: Pilot Evaluation
I collected data about TAAD through two exploratory methods (observations and embedded open-ended assessments). These qualitative approaches helped me refine the standardized instrument for the next phase.
→
Phase 3: Full-Scale Evaluation
Using a pre-post study design, I assessed student outcomes through standardized observations and an assessment embedded as a program activity to look for statistically significant student growth.
Sample of how I reported pre- and post- student assessment findings, noting if and when there was a statistically significant increase in percentage of correct answers.
Findings:
TAAD students demonstrated significant growth in content knowledge about the two democratic models, skills (e.g., public speaking and collaboration), and social-emotional learning and civic engagement.
Program observations showed that students were most engaged when presented with opportunities to connect program content to their own lived experiences.
Recommendations
The evaluation highlighted areas for increased alignment between the intended outcomes of the program and actual student outcomes, particularly those around object inquiry. In reflecting on the findings with N-YHS, I noted outcomes on the Logic Model that were less aligned with the program’s long term intended impact, and which might be replaced with ones that were more closely aligned (e.g., emphasis on curation).
The client should consider adjusting the art making components of the program to support student creativity and interest in bringing their ideas about democracy and civic engagement back home to their communities.