At the Packard Foundation, we endorse real-time evaluation (RTE). In recent years, many program and foundation evaluators have come to use the term “real-time” to mean reporting regularly—and not just at the evaluation’s conclusion. But in truth, real-time evaluation means more than providing regular feedback. It means positioning the evaluation to effectively support learning and inform strategy.
Because RTE is easier said than done, it is critical that evaluators maintain an ongoing dialogue about how they approach this kind of evaluation, and what they are learning about successfully meeting its challenges and demands.
Evaluators who practice RTE need to focus less on their own predetermined reporting timelines and more on the timelines of the grantees or initiatives being evaluated. While regularly scheduled reporting (such as every six months) can be useful and good evaluation practice, its success in informing strategy can be hit or miss. Even if data is provided frequently, the timing may be off or the data may arrive too late.
Real-time evaluations, at least in part, need to build in flexibility, so that when a strategy changes or a critical event occurs, the evaluation can adjust with it. Evaluations that continually revisit what they are measuring—and how and when they are measuring it—may end up with a wealth of data that ultimately is not comparable over time. Consequently, evaluators must balance the need to be methodologically rigorous and consistent with the need to collect data that will be most useful in informing strategy at different points in time.
When quick decisions need to be made, evaluators must develop methods that both allow for quick design, implementation, and analysis—and also provide useful and trustworthy strategy-level data. For that reason, “rapid response methods” emphasize timing, flexibility, and responsiveness. These methods have quick turnaround times and bring evaluation data—in accessible formats—to the table for reflection and use in decision making.
The Packard Foundation’s Preschool subprogram provides an illuminating look at real-time evaluation in practice.