The phrase “You get out of it what you put into it” is certainly true of evaluation. As powerful a tool as evaluation can be, it is often one of the last considerations of festivals in terms of time and budget. Many festival organizers have limited experience planning for and conducting evaluations and don’t know where to start. Others have had negative prior experiences with evaluation, feeling that they did not getting anything useful or insightful out of the effort.
It doesn’t have to be that way! This toolkit is designed to help you plan and implement your festival evaluation, and then report impact to your festival stakeholders. The resources were developed to be mindful of the unique nature of science festivals. In addition to some of the more general resources found within this toolkit, there are several connected and complementary toolkits that provide more detail and specific tools (such as surveys) that you are free to adapt for your purposes or use as-is.
Evaluation is the systematic appraisal of an initiative or program. Evaluation can be used to summarize what happened at your festival, but that’s just the beginning of what good evaluation should do. In addition to celebrating the good things that happened, the best evaluations spur discussion and lead to new insights about your festival. Effective evaluations can also help you plan your future efforts in the direction of your festival’s goals. For example, the North Carolina Science Festival has the goal of producing a high-quality event within a 30 minute drive of every North Carolinian – each year’s evaluation data help to both document our progress and refine strategies for getting there.
It is important to determine that our Festival work matters, but defining what we mean by IMPACT – and then going about measuring it – is both challenging and rewarding! Some organizers think of impact strictly in terms of the number of participants at their festival. Others think of impact as a function of geographic reach and/or the way the festival engages underserved or underrepresented communities. Still other festival organizers see impact as the creation of opportunities for scientists to engage with the public about their research.
Engagement, fun, learning, career awareness and science literacy are some of the many ways festivals define impact. No matter how you define impact for your festival, evaluation is the process that will help you understand your festival’s current impact and assist you with planning for the future.
The evaluation resources in these toolkits will grow substantially in the coming years thanks to the generous support of the National Science Foundation (NSF). In 2014, the NSF funded Collaborative Research: EvalFest (Evaluation Use, Value and Learning through Festivals of Science and Technology) (NSF#1423004) to build a community of practice among festivals to develop and test innovative evaluation methods. EvalFest currently involves 24 U.S. science festivals who are working together to determine what festivals want and need to measure and then devising tools for doing so. As new resources emerge from this community of practice, they will be shared via these toolkits. Because some of the methods are being implemented by all 24 EvalFest festivals, important discoveries about science festivals overall should also come to light.
In addition to this toolkit, which contains general information about festival evaluation, evaluation resources can be found in the following toolkits:
Evaluation Planning: checklists, timelines, and job descriptions related to developing an evaluation plan and hiring field researchers to assist with data collection.
Evaluation Methods: descriptions of the many ways festivals might go about measuring their progress towards their goals, including methods such as surveys and observations. Less traditional methods such as mystery shopping, embedded assessment booths, zip code analysis and social network analysis will also be added.
Evaluation Instruments: copies of actual surveys, recording sheets and other devices used by festivals to capture data during evaluation activities.
Evaluation Reporting: written reports and presentations of science festivals that share evaluative data and are directed at festival collaborators, funders and the community at large.
EvalFest: documents, training materials, presentations, publications and reports specific to the EvalFest project. While anyone is welcome to view and use these resources, they are best understood and useful to those festivals formally involved in the EvalFest project.
This presentation reports data and evaluation findings from the inaugural 2009 San Diego Science Festival.
This survey collects the demographics and psychographics of the Bay Area Science Festival returning attendees.
This one-hour webinar discusses festival evaluation. It provides an overview of festival data collected from 27 festivals and the tools used to col...
This is an example survey form used to gather data about the demographics and psychographics of the 2012 Bay Area Science Festival Partners/Event Host...
This document serves as a training and information guide for intercept surveys. It includes a procedure for identifying participants, sample interacti...
Attendee survey results from the Bay Area Science Festival in 2012.
This instrument is a tracking survey for attendance at festival events.
This example survey collects data about the demographics and psychographics of the Cambridge Science Festival Carnival attendees.
This survey collects data about the demographics and psychographics of the Cambridge Science Festival Attendees.
This is a report of data collected from surveys administered at the 2009 San Diego Science Festival Expo.
This report includes results from the San Diego Science Festival Exhibitor and Performer survey.
This is an example student survey for hands on activities at the Cambridge Science Festival.
This is a short document for Science Festivals with quotes to keep the momentum going through the year.
The 2011 Las Vegas Science Festival brought together over 80 organizations who provided over a hundred exciting free STEM-related programs to thousand...
This instrument is used as a template to collect data using a 'secret shopper' protocol to help assess exhibitor and partner booths at Science Festi...
This survey collects data about the demographics and psychographics of the North Carolina Science Festival Attendees.
Small Talk was a three-year long collaborative project that looked at the benefits for the science communication community in working together on dia...
This is survey that collects data about the demographics and psychographics of the 2012 Seattle Science Festival Luminaries Series attendees.
The fourth annual San Diego Festival of Science & Engineering engaged an estimated 50,000 people in science, technology, engineering and math (STE...
An evaluation tool that can be used to gauge young participants' interest in the Philadelphia Science Festival's Neighborhood Science programs.
The Science Festival Alliance (SFA) has helped to spark and support dozens of independent festival initiatives in the US and abroad. But the SFA got i...
The core questions from the EvalFest festival survey in 2015. Each EvalFest partner festival featured these survey items on their festival instruments...
A timeline to assist in evaluation planning for EvalFest festival partners.
A presentation featuring several innovative evaluation methods from the 2015 EvalFest Annual Meeting in Chapel Hill, NC on January 14 and 15, 2015.
A checklist for creating a legible and engaging layout for your evaluation reports by Stephanie Evergreen.
A checklist for creating engaging data visualizations for evaluation reports by Ann K. Emery and Stephanie Evergreen.
A checklist for evaluation planning for expos for the North Carolina Science Festival.
A sample of an executive summary brochure from the 2013 North Carolina Science Festival's Thorp Science Night.
The community evaluation report from the 2014 Arizona SciTech Festival.
This guide provides tips and suggestions for data collection at the 2015 North Carolina Science Festival.
The attendee survey from the 2014 Philadelphia Science Festival.
An evaluation guide for coordinators from the 2012 North Carolina Science Festival. This guide gives tips and suggestions for administering an attende...
This is a sample job posting to recruit field researchers to conduct evaluation of the 2013 North Carolina Science Festival.