CSTE logo
This website uses cookies to store information on your computer. Some of these cookies are used for visitor analysis, others are essential to making our site function properly and improve the user experience. By using this site, you consent to the placement of these cookies. Click Accept to consent and dismiss this message or Deny to leave this website. Read our Privacy Statement for more.
CSTE Features
Blog Home All Blogs
Search all posts for:   


View all (177) posts »

Participatory Program Evaluation Planning in Support of the Vermont Comprehensive Cancer Control Initiative

Posted By Leanne Shulman, Friday, April 22, 2016
Updated: Friday, April 22, 2016

In the fall of 2014 the Comprehensive Cancer Control (CCC) program in Vermont was in particular need of evaluation expertise. In the next year (2015) the program oversaw the creation of a new five-year cancer plan, the 2020 Vermont Cancer Plan, and the end of the existing external evaluation contract. Financial resources for evaluation were very limited, with no funding available until 2016 to contract out the development of a new evaluation plan, evaluation questions and revised logic model to accompany the new cancer plan.

As described in CSTE’s 2004 white paper “Essential Functions of Chronic Disease Epidemiology in State Health Departments,” evaluation is a Tier 2 Essential Function of chronic disease epidemiology. Chronic disease epidemiologists play a supportive or coordinating role in delivering Tier 2 functions. As the Vermont CCC program does not have dedicated evaluation staff, it is my responsibility to coordinate evaluation of the CCC program in Vermont.

Ideally, I would develop the evaluation plan alongside the 2020 Vermont Cancer Plan rather than contracting out the evaluation the year after the plan was written. However, given my limited (aka non-existent) experience in developing evaluation plans and limited access to internal evaluators within the Vermont Department of Health, I was not in a position to develop a high-quality evaluation plan.

In an effort to gain access to evaluation expertise I applied to the National Association of Chronic Disease Directors (NACDD) Epidemiology Mentorship program with a proposed project of creating an evaluation plan for the CCC program. The mentorship program paired me with Dr. Ericka Welsh of the Kansas Department of Health and Environment, Bureau of Health Promotion, a senior epidemiologist with expertise in evaluation.

With Dr. Welsh’s assistance I coordinated the development of a five-year evaluation plan to accompany the new five-year state cancer plan. We used the Comprehensive Cancer Control Branch Program Evaluation Toolkit (2010) as a framework. A participatory evaluation approach was employed, which began with an invitation to join the Vermont comprehensive cancer control coalition – Vermonters Taking Action Against Cancer (VTAAC) Evaluation Committee. The invitation was sent to the general coalition membership, as well as to specific partners with evaluation experience.

The ensuing Evaluation Committee consists of three staff from the Vermont Department of Health, the VTAAC coordinator, a cancer center communications manager, a quality improvement liaison for a major private insurer, and the coordinator for the Vermont Cancer Survivor Network.

The 2015 VTAAC Evaluation Committee: Sarah Keblin, Ali Johnson, David Cranmer, Sharon Mallory and Leanne Shulman. Not pictured – Sherry Rhynard and Micah Demers. Photo by Ali Johnson

The Evaluation Committee met five times over the course of nine months in 2015 to provide input on each step of the evaluation plan. Under my leadership, the Evaluation Committee revised the CCC initiative logic model, determined the program’s evaluation questions and created the evaluation planning matrix.

Leadership in this case entailed preparing draft documents, which were critiqued during the meeting, and then creating revised versions, which were further edited via email. This process was repeated for each of the pieces of the evaluation plan. I made a significant effort to do as much preparatory work as possible before committee meetings in order to limit the demands placed on the stakeholders. It took a full nine months to complete the evaluation plan. Although there was a great deal of investment in building relationships, creating drafts and rewriting, the ultimate product is stronger as a result of the input of a variety of voices.

The evaluation plan includes both process and outcome evaluation and is designed to measure and improve the effectiveness of the CCC program and VTAAC, to inform future program and coalition development, and to demonstrate accountability to funders.

The participatory approach, with guidance from a senior-level mentor, resulted in an evaluation plan that has buy-in from key VTAAC partners while maintaining the framework required by the funder (CDC). Activities in the plan will be implemented by staff within the Vermont Department of Health, primarily the CCC epidemiologist (me) with assistance from an external contractor who will conduct individual evaluation activities, such as focus groups. The work done by the Evaluation Committee was presented to and approved by the VTAAC Steering Committee in November 2015. The Evaluation Committee has a continued role in overseeing the implementation of the plan throughout the next five years.

If you are interested in hearing more about the participatory evaluation planning process we undertook, please come to my presentation at the CSTE Annual Conference – 11am at the Cross Cutting I – “No Seward’s Folly: Quality Improvement, Collaboration, and Evaluation” session on Wednesday, June 22, 2016.

The 2016-2020 Vermont Comprehensive Cancer Control Initiative Evaluation Plan can be found here: http://healthvermont.gov/pubs/cancerpubs/documents/evaluation_plan_comprehensive_cancer_control_program.pdf.

Leanne Shulman is the epidemiologist and evaluator for the Comprehensive Cancer Control Program in Vermont. She is also the epidemiologist for the Vermont Office of the Chief Medical Examiner. She was a mentee in the 2015 National Association of Chronic Disease Directors (NACDD) Epidemiology Mentorship program. For more information on CSTE activities, join the Cancer Subcommittee.

Photo by Ali Johnson

This post has not been tagged.

Share |
Permalink | Comments (0)