Beyond Metrics: Advancing Evaluation for Your Program’s Meaningful Change
AUGUST 2025
BEYOND METRICS:
Advancing Evaluation for your program’s meaningful Change
Post-Conference Brief, Toolbox, and Next Steps
TABLE OF CONTENTS
- INTRODUCTION 1
- EVALUATION AS BOTH A SCIENCE AND AN INTERVENTION 3
- Utilization-Focused Evaluation to Support and Evaluate Systems Transformation — Michael Quinn Patton, Ph.D.
- BUILDING CAPACITY IN AN EVALUATION TOOLBOX
- It’s Never too Early to Think About Evaluation — Michelle Burd, Ph.D. 6
- Foundational Building Blocks for Evaluation — Sarah Hug, Ph.D. 8
- Intersection of Logic Models and Evaluation Plans — Cindy Ziker, Ph.D. 10
- It’s About People, Isn’t It: Tools to Strengthen Working with Large Teams — Sara Buldoc, Ph.D. 12
- CURRENT LANDSCAPE AND EXAMPLES:
- Changing the DNA of Evaluation as Usual: A Meta-Analysis — Nancy Van Leuven, Ph.D. 15
- HSI Pilot Project: Improving First-to-Second Year Retention — Rod Taylor, MBA 17
- Experiential Learning and Research to Strengthen Agricultural Education — Jeff Bullock, Ph.D. 18
- Empowering Hispanic Students in Agriculture Through Horticulture and Community Engagement — Tim Ray, Ph.D. 19
- Learning to Lead - Next Gen — Brian Palmer, MA 20
- MOVING FORWARD: NEXT STEPS FOR THE FUTURE
- Beyond the Application: Building a Culture of Strategic Grant Development and Impact — Maral Kismetian, MPA, CRA 22
- The Importance of the “H” and “S” in Hispanic serving Institutions (HSI) — Azuri Gonzalez, Ph.D. 24
- Expanded Meta-Analysis — Nancy Van Leuven, Ph.D. 26
- CLOSING DISCUSSION FORUM
- Overview 34
- Executive Summary 36
- Post-Conference Feedback Survey Findings 40
- APPENDICES
- Conference Agenda 42
- Speakers & Collaborators 44
- Practical Evaluation Tools Shared by Speakers 48
SUGGESTED CITATION
Ziker, C., Paredes, P., Freire Vasconez, P., Pheasant, S. (2025, May 16-17). Beyond
Metrics: Advancing Evaluation for Your Program's Meaningful Change Post-Conference
Brief
Published by the Institute of Food and Agriculture, California State University, Fresno
Graphic Design by Samantha L. Bass
FUNDER ACKNOWLEDGEMENT
This work is supported by the Hispanic-Serving Institutions Education Grants Program, project award no. 2022-77040-37628, from the U.S. Department of Agriculture’s National Institute of Food and Agriculture. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and should not be construed to represent any official USDA or U.S. Government determination or policy.
PROJECT TEAM EXTENDS SINCERE GRATITUDE TO:
- Dr. Irma Lawrence and Dr. Decatur Foster for their support and leadership at the inception of this project with USDA NIFA,
- Juan M. Alvarez, our USDA liaison in the Office of Partnerships and Public Engagement (OPPE),
- Maral Kismetian, MPA, CRA, Carmen T. Mendoza MSPH, MS and Dr. Susan M. Pheasant for their insights and inspirations in developing the original proposal and Dr. Nancy Van Leuven for enthusiastically joining to bring the conference to fruition.
INTRODUCTION
The Beyond Metrics: Advancing Evaluation for Your Program’s Meaningful Change virtual conference was hosted on May 15-16, 2025 by the Institute of Food and Agriculture, California State University, Fresno, with an agenda that included presentations by 12 faculty, practitioners, and leading evaluation experts. Themes included foundational evaluation models, language choice in grant-writing, leadership development, institutional sustainability, servingness, systemic transformation within Hispanic Serving Institutions (HSIs), and the evolving landscape of evaluation. Participation included faculty, evaluators, and grant administrators representing over 30 institutions.
See page 48 highlighting practical evaluation tools shared by our speakers for non-evaluation professionals – Valuable additions for your evaluation toolkit!
THE PURPOSE OF THIS BRIEF IS TO:
- Highlight the key points of the presentations, dialogues, and reflections
- Provide an ‘evaluation toolkit’
- Offer examples of program evaluation projects
- Offer recommendations and collective insights from both the presenters and participants
EVALUATION AS BOTH A SCIENCE AND AN INTERVENTION
DR. MICHAEL QUINN PATTON: “UTILIZATION-FOCUSED EVALUATION TO SUPPORT AND EVALUATE SYSTEMS TRANSFORMATION”
Dr. Michael Quinn Patton, founder and CEO of Utilization-Focused Evaluation,” delivered a compelling and reflective keynote entitled “Utilization-focused Evaluation to Support and Evaluate Systems Transformation,” tracing the history and transformation of evaluation over five decades. As a pioneer in utilization-focused and developmental evaluation, Dr. Patton offered ‘a view back and look to where we may be going’ in the field of evaluation, and shared insights on the overarching principle of utilization evaluation, which he described as having a “focus on intended use, by and with intended users, in every aspect of, and at every stage of, an evaluation.”
Framing evaluation as both a science and an intervention, Dr. Patton articulated how evaluation has evolved beyond traditional metrics to incorporating methods that foster meaningful and sustainable transformation and advocated for a systems thinking approach, capable of addressing today’s convergence of global challenges. He described core tools and practices designed for evaluating complex, adaptive systems that can be a catalyst for systemic change.
KEY THEMES
Dr. Patton outlined four generational shifts in the focus of evaluation over the last five decades:
- 1st Generation (Measurement): Evaluation focused on quantifiable indicators of goal attainment, metrics, and standardized testing.
- 2nd Generation (Description): The emphasis shifted to including qualitative insights and case studies; “No numbers without stories, no stories without numbers.”
- 3rd Generation (Judgment): Formative and summative evaluation was introduced, with a focus on merit and worth.
- 4th Generation (Responsive Stakeholder Engagement): Evaluation became more participatory and collaborative, using culturally responsive approaches that value all stakeholders’ perspectives.
“PROJECTS DON’T CHANGE SYSTEMS. PEOPLE, RELATIONSHIPS, AND PRINCIPLES DO.”
– DR. MICHAEL QUINN PATTON
He emphasized the importance of utilization-focused evaluation and stressed the need to begin evaluations with a clear understanding of their purpose and anticipated application, positioning evaluation as a tool for learning and meaningful change, rather than a compliance exercise.
Key themes included how evaluation can shape behaviors, foster insight, and influence organizational culture. Dr. Patton advocated for the use of research-informed theories of change and the development of theories of transformation that reflect the complex, non-linear nature of systems. His presentation described how true systems change and sustainable transformation require holistic approaches that account for interdependencies, feedback loops, and evolving mental models, underscoring the need for coordinated, networked, and multi-dimensional efforts grounded in shared principles such as equity, sustainability, and interconnectedness.
RECOMMENDATIONS
- Embed evaluation early: Involve evaluators during the design phase to shape goals, logic, and adaptive strategies, moving from "back-end assessment" to "front-end influence."
- Promote networked collaboration: Break down silos and foster inter-organizational collaboration through shared theories of transformation.
- Shift toward system-level thinking: Reorient evaluation frameworks to consider relationships, context, and emergent outcomes rather than isolated metrics.
- Use principles as anchors: Develop and apply evaluative principles (like equity and sustainability) that adapt to context and complexity, rather than rigid best practices.
- Support evaluative thinking: Build internal capacity for evaluative thinking, storytelling, and reflective learning to ensure that evaluation remains meaningful.
BUILDING CAPACITY IN AN EVALUATION TOOLBOX
DR. MICHELLE BURD: “IT’S NEVER TOO EARLY TO THINK ABOUT EVALUATION”
Dr. Michelle Burd, Director of Evaluation Education at the Diana Natalicio Institute for Hispanic Student Success at the University of Texas at El Paso, delivered the session “It’s Never Too Early to Think About Evaluation.” Drawing from her extensive experience as an independent consultant and internal evaluator on STEM education initiatives, she introduced practical tools such as logic models and theories of change to guide effective evaluation design.
KEY THEMES
Dr. Burd emphasized crafting explicitly evaluative questions that assess quality and impact. Break-out group discussions offered participants opportunities to interact and think collectively about evaluation design and the information needed to address questions:
- “How well are project activities being implemented?”
- “How valuable are outcomes for students, faculty, institutions, and partners?”
Key themes included the importance of defining the purpose of evaluation (formative, summative, developmental), and the principles of responsive evaluation, with an emphasis on culturally and contextually responsive evaluation approaches that include diverse stakeholders, disaggregated data, and mixed-method designs. These strategies aim to enhance authenticity, reveal unexpected findings, and identify systemic levers for change. Examples of challenges resulting from misaligned metrics were also described.
RECOMMENDATIONS
- Develop actionable evaluation questions: Ensure questions elicit judgments about quality, value, or worth, rather than merely describe outcomes.
- Use mixed-methods approaches: Combine quantitative and qualitative data to capture a broader range of perspectives and unanticipated insights.
- Adopt developmental evaluation when needed: Use this approach for innovative or emergent programs, or in contexts experiencing significant change or uncertainty.
- Engage stakeholders throughout the evaluation process: Involve students, faculty, advisors, and community members in designing, implementing, and interpreting the evaluation to ensure relevance and responsiveness.
- Apply responsive evaluation principles: Disaggregate demographic data and value the lived experiences of underrepresented groups to ensure equity and authenticity.
- Leverage systems thinking: Consider institutional structures, relationships, and change agents to identify intervention points and monitor impact over time.
- Use visual tools effectively: Employ theories of change and logic models to clarify goals, link activities to outcomes, and communicate plans clearly to diverse audiences.
DR. SARAH HUG: “FOUNDATIONAL BUILDING BLOCKS FOR EVALUATION”
Dr. Sarah Hug, director of Colorado Evaluation & Research Consulting, presented the session “Beyond Metrics: Enhancing Evaluation for Meaningful Change” which emphasized the importance of surfacing assumptions and values through theory of change frameworks to support strategic formative and summative evaluation. She defined theory of change as “the ideas and hypotheses (‘theories’) people and organizations have about how change happens,” noting that these theories “can be conscious or unconscious and are based on personal beliefs, assumptions and a necessarily limited, personal perception of reality.”
Participants engaged in peer discussions prompted by thought-provoking questions such as:
- WHAT PROBLEM DOES YOUR PROJECT SOLVE?
- HOW DOES THE INITIATIVE ALIGN WITH YOUR VALUES?
- WITH THE VALUES OF YOUR ORGANIZATION, IF APPLICABLE?
- HOW ARE YOUR ACTIVITIES MEANT TO MAKE CHANGE?
- WHAT IS THE MECHANISM FOR CHANGE?
Dr. Hug highlighted the value of knowledge, skills, and attitudes (KSAs) as key impact indicators in program evaluation, particularly in STEM education, and described how they can be used as a measurement of change through data collected using surveys, learning assessments, rubrics, interview data, and observation protocols. She offered examples of KSAs for undergraduate research and provided an example of a competency framework to illustrate the general characteristics for novice, intermediate, and advanced competency levels (https://www.rpajournal.com/dev/wp-content/uploads/2020/05/The-Assessment-Skills-Framework-RPA.pdf)
Break-out discussion prompts included:
- WHAT ARE THE KEY KSAs TO YOUR PROGRAM?
- HOW DO YOU MEASURE THEM?
- WITH WHOM DO YOU ASSESS IMPACT?
Dr. Hug introduced social network analysis as a tool to assess community-building, collaboration dynamics, and networks. She noted it is particularly valuable for projects intending to visualize change in relationships, and offered a visual example of a collective impact social network map. Discussion questions included “How do connections or collaborations relate to your projects? How might a stronger/more robust network be an outcome for your efforts? And What elements of a network might be worth measuring in your context?”
RECOMMENDATIONS
- Start with the end in mind: Define success early using SMART outcomes.
- Map stakeholders and engage them in designing the evaluation strategy.
- Incorporate both quantitative metrics and narrative data for a holistic story.
- Use adaptive frameworks for long-term grants anticipating iterative growth.
DR. CINDY ZIKER: “INTERSECTION OF LOGIC MODELS AND EVALUATION PLANS”
Dr. Cindy Ziker, Executive Director of Ziker Research, delivered a presentation entitled “Intersection of Logic Models and Evaluation Plans,” which reinforced the value of logic models as strategic instruments that can bridge program design and evaluation. She detailed the core components of logic models, including inputs, activities, outputs, outcomes, assumptions, and contextual factors. She offered a structured approach to incorporating logic models in evaluation plans through the use of a data matrix that describes how evidence will be collected for addressing evaluation questions such as: “To what extent have outcomes been achieved?”
KEY THEMES
Dr. Cindy Ziker emphasized the critical role of logic models as tools for establishing a shared understanding of project expectations and providing a foundation for formative and summative evaluation. She highlighted their value in promoting accountability and enhancing transparency among stakeholders, advocating for co-developing logic models with project leaders, advisory groups, evaluators, and community partners to communicate a clear vision for achieving the project’s impacts. Dr. Ziker demonstrated how logic model elements can be translated into measurable indicators using a data matrix that includes evaluation questions, data sources, and analytical methods to support data collection plans, tracking of progress, and enabling real-time adjustments.
RECOMMENDATIONS
- Tell the project’s story by using the logic model to communicate the project’s vision, processes, and intended impacts to stakeholders and funders.
- Use a data matrix to describe indicators, data sources, data collection, and analytical methods.
- Involve project leadership, advisory teams, evaluators, and community partners in the development of the logic model.
- Utilize publicly available logic model workbooks, evaluation matrices, and templates
(e.g., from AAAS, NIFA, EvaluATE) for structure and inspiration:
- AAAS: https://sstemrec.aaas.org/wp-content/uploads/2025/01/S-STEM-REC_Logic-Model-2025.pdf
- NIFA: https://www.nifa.usda.gov/sites/default/files/resource/Generic%20Logic%20Model%20for%20NIFA%20Reporting.pdf
- Evaluate: https://evalu-ate.org/resource-library/?resource_type=template&search=logic%20model&evaluation_task=evaluation-design
DR. SARA BOLDUC: “IT’S ABOUT PEOPLE, ISN’T IT?”
Dr. Sara Bolduc, President of Sara Bolduc Planning and Evaluation LLC and an independent consultant, presented “It’s About People, Isn’t It?” which centered on the human dynamics at the core of effective evaluation and grant implementation. Her approach prioritizes relationships, communication, and institutional culture when designing, managing, and evaluating HSI initiatives. Drawing from human-centered design, organizational psychology, and team science literature, her presentation highlighted how interpersonal trust, role clarity, and distributed leadership are essential to evaluation plans.
Dr. Bolduc underscored the importance of interdisciplinary collaboration, communication, and shared leadership to tackle complex scientific problems. She noted that successful collaboration hinges on:
- How people communicate
- Resolve conflicts
- Remain aligned on complex goals
She discussed the benefits of using a team science framework that emphasizes collaboration across disciplines to solve complex scientific problems and outlined barriers encountered when working with large science teams. Dr. Bolduc shared insights on the use of an annual Team Survey to assess team members’ understanding of the program’s vision and goals, work roles, communication practices, and guidelines for crediting authorship and intellectual property. She also presented an outline for developing a Team Handbook and highlighted a variety of freely available resources to support evaluation efforts, which are detailed on her blog: STEM Education and Training TIG Week.
RECOMMENDATIONS
- Apply team science practices when evaluation projects involve large teams.
- Create psychological safety within evaluation teams to help surface challenges early.
- Use strategies to promote effective collaboration, such as recognizing contributions in progress reports, rotating leadership roles, and incorporating reflective debriefs into implementation routines.
- Use an annual Team Survey to support the development of a shared vision; build trust, and prevent conflict in interdisciplinary publishing.
- Implement reflective team practice through monthly check-ins.
- Collaborate with team members to develop a Team Handbook that describes onboarding, coordination, and governance roles.
- Ensure visible credit for all contributors in publishing scientific manuscripts, reporting, and public communications.
- Design evaluations with time and flexibility for human complexity, not just compliance.
CURRENT LANDSCAPE & EXAMPLES
DR. NANCY VAN LEUVEN: “CHANGING THE DNA OF EVALUATION AS USUAL: A META-ANALYSIS”
Dr. Van Leuven conducted a meta-analysis of more than 100 USDA-NIFA-HSI funded projects.The first phase reviewed 39 grants from California, Oregon, and Washington; the second added 71 grants from Arizona, New Mexico, and Texas. Using a triangulation of research methods, the study focuses on current practices and potential future designs. The analysis covered how projects supported student academic development, increased retention and graduation rates, expanded access to careers in food and agriculture,and professional growth.
The analysis found that funded projects relied heavily on quantitative over qualitative evaluation methods. The most common tools were surveys (46 projects), enrollment data (32), and performance outcomes (26). Four themes emerged from the review, summarized as PLOT:
- People and Culture
Focuses on student/faculty experiences, career planning, change management, and fostering HSI culture. Example: UC Santa Barbara’s “Equitable Agriculture and Environmental Management” increases underrepresented students in STEM, expands research on agriculture and environmental justice, and develops faculty expertise. - Leadership Development
Centers on action plans, career advancement, and community-building. Example: University of Arizona’s “Building Leadership & Sustainable Food Systems in Cochise County” promotes public knowledge and engagement to protect food supply, enhance productivity, improve environmental quality, and strengthen community well-being. - Organizational Communications and Outreach
Encompasses strategic communications, marketing, public relations, and outreach efforts. Example: Cal Poly Pomona’s “SoCal Farm to Table” program offers students hands-on experience with government, nonprofit, and business partners. - Training
Involves knowledge transfer for students, faculty, and staff, including orientations and degree-linked coursework. Example: CSU Humboldt’s “La Comida Nos Une” is an interdisciplinary minor in sustainable food systems emphasizing cultural diversity.
RECOMMENDATIONS
- Foster Multi-Institutional Collaboration: Move beyond single-department or single-college efforts to multi-institution partnerships that strengthen relationships and encourage innovative, cross-disciplinary proposals.
- Integrate Qualitative Evaluation: Balance quantitative methods with qualitative tools—such as interviews, focus groups, and case studies—to capture the full scope of program impact.
- Enhance Proposal Engagement: Use concise, compelling narratives and memorable project titles to improve reviewer comprehension and participant interest.
- Prioritize Communication and Outreach: Embed strategic communications into project design to increase visibility, promote successes, and strengthen stakeholder support.
ROD TAYLOR: “HSI PILOT PROJECT: IMPROVING FIRST TO SECOND YEAR RETENTION”
Rod Taylor, Dean for Math, Science, and Engineering at Columbia Basin College in Pasco, Washington, highlighted an NSF-funded STEM Academy pilot program aimed at increasing second year retention and closing equity gaps for underrepresented students.The program uses targeted interventions to promote student success in STEM pathways, particularly for bilingual and first-generation students. Key components include: A dual-language STEM orientation course; bilingual academic advising; peer-to-peer mentoring; financial support; and an industry-led STEM colloquia that connects students with regional professionals. Outcomes will inform a future proposal and provide a replicable model.
EVALUATION TOOLS INCLUDED:
- Retention tracking data
- Student self-efficacy
- Belonging
- Engagement surveys and interviews
- Academic progress monitoring (e.g. grades, course completions, interventions)
Key Themes
- Early intervention and real-time academic progress tools for proactive support, preventing student attrition.
- Collaboration of cross department ‘completion coaches’ (advisors) can give-bilingual support to improve student engagement.
- Cohort models can foster community, while linked courses and peer mentoring create a support network that increase students’ sense of belonging.
- The use of dual-language materials reduces barriers for Hispanic students.
RECOMMENDATIONS
- Implement Comprehensive Support Systems: Combine academic, financial, and cultural support elements to address multiple barriers simultaneously.
- Ensure Language Accessibility: Provide bilingual support and materials for Hispanic-serving institutions to reduce cultural and linguistic barriers.
- Prioritize Early Intervention: Develop systems for identifying and supporting struggling students within the first 2-3 weeks of enrollment.
- Establish Cross-Departmental Communication: Create regular meeting structures to prevent institutional silos and improve student service coordination.
- Build Authentic Community: Create cohort models that foster peer accountability and belonging while maintaining flexibility in course scheduling.
- Engage External Evaluators: Invest in third-party evaluators who participate in program implementation and student interaction, not just assessment.
DR. JEFF BULLOCK: “EXPERIENTIAL LEARNING AND RESEARCH TO STRENGTHEN AGRICULTURAL EDUCATION”
Dr. Jeff Bullock, Acting Dean for Math, Science, and Engineering at Wenatchee Valley College in Washington, described a NIFA/USDA funded project that began in 2022. Dr. Bullock described the primary objectives of the grant, which include acquiring essential laboratory equipment to provide students with hands-on experience in molecular biology techniques relevant to pathology. In addition to lab training, the funding offers scholarships to assist students in their transition to four-year institutions and provides summer internships with industry partners and Washington State University researchers. Industry partners provided students with valuable experience conducting tissue and soil sampling and learning molecular biology techniques in the lab. Despite lower-than-expected participation numbers, the program achieved success in preparing students for STEM pathways.
KEY THEMES
Focus: COVID enrollment and the economic downturn impacting Washington’s tree fruit industry. He highlighted the significant and lasting benefits of the grant, particularly its role in enhancing student learning and institutional capacity through the acquisition of permanent laboratory equipment and the development of strong partnerships with Washington State University researchers. Students gained hands-on experience with DNA/RNA extraction and pathogen detection techniques for diseases, such as little cherry disease, equipping them with valuable skills for careers in STEM fields.
Dr. Bullock emphasized the importance of collaboration with a professional third-party evaluator who conducted comprehensive pre- and post-assessments, interviews, and longitudinal tracking of transfer students to evaluate program impact and students’ successful transitions to four-year institutions. He shared plans to broaden the integration of molecular biology techniques beyond agriculture to biology programs in fields such as medical technology, nursing, and other STEM disciplines, underscoring the relevance of these biotechnology skills.
RECOMMENDATIONS
- Equipment investments, such as molecular biology tools and microscopes, provide lasting value beyond the grant period.
- Internships with industry partners and researchers can provide incredibly valuable student learning experiences.
- Professional evaluators provide specialized expertise, robust assessment systems, and help with grant reporting.
DR. TIM RAY: “EMPOWERING HISPANIC STUDENTS IN AGRICULTURE THROUGH HORTICULTURE AND COMMUNITY ENGAGEMENT”
Dr. Tim Ray, Dean of Agricultural Science and Technology at Chemeketa Community College in Salem, Oregon, shared the outcomes learned from Chemeketa Community College’s HSI grant designed to inspire Hispanic and underrepresented students to pursue careers in agriculture. Through strategic use of student ambassadors, faculty advising, and educator engagement, the program has successfully fostered early career pathways and advanced employment preparation for students.
KEY ELEMENTS INCLUDE A PEER-TO-PEER RECRUITMENT MODEL WHICH HAS LED TO A 40% increase in horticulture enrollment by engaging student ambassadors in outreach efforts
He emphasized that ambassadors gain valuable industry experience through trade show participation and nursery partnerships, enhancing both technical and professional skills. Success has been driven by continuous recruitment efforts and innovative strategies aimed at supporting part-time students balancing work and family commitments. A strategy for inspiring entrepreneurship is to offer a non-credit farm finance class for aspiring agriculturalists to learn the basics of farm management. His presentation underscored the value of collaborations with high school agriculture teachers through summer institutes, which have expanded curriculum alignment and increased College Credit Now (CCN) offerings. The program has had a positive impact on students pursuing four-year degrees in agriculture-related fields by strengthening the pipeline from high school to college, developing strong transfer pathways with Oregon State University's College of Agriculture, and providing a direct transfer degree and $25,000 scholarships.
RECOMMENDATIONS
- Use students as primary recruiters: Employ young people as ambassadors to tell their stories and make peer-to-peer connections.
- Offer tuition waivers as incentives: Supplement ambassador wages with tuition vouchers to make positions more attractive.
- Offer teacher stipends for attending trainings: Acknowledge appreciation for teachers’ investment in time and effort during summer training by offering a stipend.
- Continue Ambassador Programs post-grant: Secure departmental funding to maintain student ambassador positions after grant periods end.
BRIAN PALMER: “LEARNING TO LEAD – NEXT GEN”
Brian Palmer, a mathematics faculty member at Hartnell College in Salinas, presented a thoughtful and critical reflection on his experience as the external evaluator for the USDA-funded “Theory of Evaluation: USDA Next Gen Learning to Lead” grant, a five-year, multi-institutional initiative involving Hartnell College, Imperial Valley College, CSU Monterey Bay, and the University of Arizona at Yuma. The project aims to strengthen workforce pathways for students engaged in leafy greens (e.g., lettuce, spinach, kale) agriculture across California and Arizona through a unique, migratory agricultural labor force that moves seasonally between Salinas, CA and Yuma, AZ. The program aligns educational access with labor mobility by connecting two-year colleges and universities through shared resources, transfer scholarships, campus tours, and internship pathways. Mr. Palmer emphasized the importance of evaluating programs holistically using methods that are grounded in context, student experience, and systems-thinking.
Mr. Palmer described features of the “Learning to Lead” program that integrates high impact practices to promote retention, student engagement, and transfer success. These include wrap-around support, long-term paid internships, learning communities, emergency aid, faculty/peer mentorship, and transfer and emergency scholarships. He noted that "transfer internships" at university partners act as bridges between institutions, promoting smooth academic transitions for students. He introduced Goodhardt’s law, which states “when a metric becomes an objective, it ceases to be a good metric” and cautioned against using misaligned metrics that can misrepresent impact if not carefully interpreted. He emphasized the importance of evaluating programs holistically to determine how the program is functioning, using methods, such as interviews, that are grounded in context and students’ experiences.
RECOMMENDATIONS:
- Evaluate programs, not just objectives, by focusing on whether the structure, implementation, and student experience align with intended impact.
- Interrogate metrics using context and apply Goodhardt’s Law to avoid reinforcing faulty assumptions or oversimplified measures of success.
- Build evaluator-director partnerships early during the design stage to allow for alignment between evaluation goals and program theory of change.
- Use qualitative data through interviews, reflections, and contextual data to uncover insights not visible through quantitative data.
MOVING FORWARD: NEXT STEPS FOR THE FUTURE
MARAL KISMETIAN: “BEYOND THE APPLICATION: BUILDING A CULTURE OF STRATEGIC GRANT DEVELOPMENT AND IMPACT”
Maral Kismetian, MPA, CRA, Director of the Grants Development Office at California Polytechnic State University, San Luis Obispo, presented “Beyond the Application: Building a Culture of Strategic Grant Development and Impact,” a timely and forward-looking perspective on how Hispanic-Serving Institutions (HSIs) can navigate and thrive amid recent shifts in federal funding structures. Drawing from over 20 years of experience in research and grant administration, her insights focused on moving beyond transactional grant submissions to a sustained, collaborative, and impact-driven culture of proposal development and evaluation. She offered practical guidance for navigating current shifts in federal funding priorities and regulatory landscapes, especially for HSIs.
KEY THEMES
Ms. Kismetian’s presentation highlighted the need for building a culture of grant development that is strategic, pro-active, resilient, and collaborative. To accomplish this, she advocated for developing a plan, building early-ideation pipelines, cultivating routine touchpoints between faculty and grant administrative staff, and normalizing cross-unit collaboration. She pointed out that federal priorities are shifting toward national security, economic competitiveness, domestic job creation, and regulatory efficiency; and indicated that programs, such as food security and workforce development, serve as opportunities for aligning institutional missions with these funding trends.
RECOMMENDATIONS
She noted how successful proposal development may hinge on translating agency jargon into values-based, locally resonant language. Examples included swapping “advancing equity for marginalized groups” for phrases like “reducing barriers to participation” and “providing targeted mentorship and support so that all students can succeed.”
- Create a culture of grant development by providing time and structured support (e.g., office hours) for early ideation and proposal development.
- Adapt proposals to shifting funding structures and align with new priorities.
- Leverage technical assistance hubs, USDA partnerships, and resources.
- Build coalitions that create shared infrastructure and partnerships among HSIs.
- Conduct meaningful evaluation beyond metrics.
- Develop forward-looking grant strategies that include timelines.
- Prioritize robust, meaningful evaluation frameworks.
DR. AZURI GONZALEZ: “THE IMPORTANCE OF H AND S IN HISPANIC SERVING INSTITUTIONS (HSI)”
Dr. Azuri Gonzalez, Executive Director of the Alliance of Hispanic Serving Research Universities (HSRU) and the University of Texas Regents’ Endowed Distinguished Director for the Diana Natalicio Institute for Hispanic Student Success, focused on how institutional context and student composition are central to evaluation. She emphasized the importance of institutional positionality, the evolving sociopolitical landscape, and frameworks that center intentionality, structure, and outcomes in the service of historically underrepresented students.
Dr. Gonzalez defined the federal designation for an HSI, which requires that 25% of the institution’s student population must be Hispanic or Latino. She introduced concepts like Hispanic ‘servingness’ (Garcia et al., 2019) and becoming a ‘student ready college’ (McNare, 2022), where institutions take on the responsibility of being ready for students, in contrast to a risk-factor/ deficit framing of student readiness for college. She emphasized the importance of tracking changes in institutional context, student attributes, and serving interventions in evaluation and highlighted how the institutional environment affects academic and non-academic outcomes. She underscored the importance of acknowledging intersectionality in regard to student composition and attributes and encouraged evaluators to consider questions such as:
- “What are student attributes, identities and backgrounds?”
- “What enhances your ability to serve?”
RECOMMENDATIONS
Dr. Gonzalez stressed the need for adapting to evolving policy and post secondary landscape”, which requires re-framing without losing mission alignment. She described the HSRU model, which leverages a research-informed lens to achieve its goal of creating a sustainable consortium to exponentially increase the number of faculty and student researchers who can contribute to national research priorities. She concluded by introducing a multi-institutional model for integrating assessment and engaged the audience in a discussion about strategies for promoting the concept of ‘servingness’ and the use of ‘servingness’ frameworks at institutions.
- Utilize servingness and student ready college frameworks to guide program design, evaluation, and institutional change.
- Consider institutional context and the institution’s unique positionality, goals, and approaches to serving students when conducting evaluations.
- Move beyond metrics like enrollment to include student outcomes that reflect student experiences and systemic impact.
EXPANDED META-ANALYSIS
DR. NANCY VAN LEUVEN
Building on earlier research from six states (AZ, CA, NM, OR, TX, WA), this post-conference study examined which evaluation methods are selected to support students, encourage student retention and graduation, offer access to career pathways in food and agriculture, support academic development, and promote faculty and staff professional development. Dr. Van Leuven conducted a comprehensive meta-analysis of 134 USDA-funded programs at Hispanic Serving Institutions (2018–2023), focusing on initiatives supporting Hispanic and other underrepresented students in agriculture, STEM, and related fields.
Through a synthesis of quantitative and qualitative data analyses, the study identified key trends, thematic priorities, and the overall impact of these programs across multiple states. In doing so, it provides both a descriptive overview of program characteristics and an evaluative assessment of their contributions to academic advancement and career preparation.
SUMMARY: GRANTS AWARDED BY STATE
California and Texas account for the highest number of awarded grants, a trend that aligns with their substantial Hispanic-Serving Institution (HSI) populations. These patterns suggest a strategic alignment between program funding and areas of greatest potential impact including student support, academic development, and career access.
EXPANDED META-ANALYSIS
Table 1 illustrates how grant distributions extend across a diverse range of states, with notable concentrations in regions characterized by significant Hispanic student demographics.
| State | Grant Count | Title, Institution, and Cumulative Award Amount for Largest Project In State |
|---|---|---|
| TEXAS | 42 | “BE AWARE 2: BioEnergy And Water for Agriculture Research and Education Network 2” $1,000,000. University of Texas, San Antonio |
| CALIFORNIA | 35 | “No More Silos: Multi-Disciplinary and Data Intensive Training for Careers in Agricultural and Natural Resource Industries and Agencies” 1,000,000. California Polytechnic University, Pomona |
| PUERTO RICO | 15 | “Enhancing Experiential Learning by Assessing Food Quality of Tropical Crops Using Advanced Chemical Sensors and Unmanned Aerial Vehicles” $1,000,000. Recinto Universitario Mayaguez |
| ARIZONA | 11 | “Training the Next Generation of Hispanic Leaders for Managing Resilient Forests” $269,980. Northern Arizona University |
| NEW MEXICO | 10 | “LIFT – Leadership in Forestry Training” $1,000,000. University of New Mexico |
| FLORIDA | 5 | “Building Research, Education, and Leadership for Agriculture and Related Careers” $1,000,000. Florida International University |
| NEW YORK | 4 | “Future Leaders in Bio-Derived Organic Energy Storage Materials” $400,000. City College of New York |
| COLORADO | 3 | “Development of the Industrial Hemp Education, Agriculture, and Research Program” $275,000. Colorado State University, Pueblo |
| WASHINGTON | 3 | “Experiential Learning and Research to Strengthen Agricultural Education” $268,803. Wenatchee Valley College |
| ILLINOIS | 2 | “Growing and Sustaining the Next Generation: A Program to Create New Leaders in Sustainable Nutrition” $275,000. Dominican University |
| OREGON | 1 | “Inspiring Hispanic Students to Pursue Careers in Agriculture and Technology through Experiential Learning” $274,590. Chemeketa Community College |
| NEW JERSEY | 1 | “Bergen Community College Food and Agriculture Science Education Pathways” $996,251.00 Bergen Community College |
| WISCONSIN | 1 | “Alverno College: Establishing a Pipeline of Student Experience and Development in Sustainable Agriculture” $273,786. Alverno College |
| MASSACHUSETTS | 1 | “North Shore Community College: Accredit, Attract, Achieve, Advance” $199,621. North Shore Community College |
EXPANDED META-ANALYSIS
SUMMARY: KEY TRENDS IN EVALUATION METHODS
In this meta-analysis, the most frequently used methods were quantitative – attendance tracking and surveys– selected to generate numerical data while being convenient for respondents. In contrast, qualitative methods, such as focus groups and interviews, provide in-depth, non-numerical insights into participants’ perspectives, including opinions and experiences.
| Evaluation Methods | Total Grants | Method Description |
|---|---|---|
| Surveys | 86 | A self-reporting method to measure attitudes, behaviors, and impressions as data for larger audiences, often through questionnaires |
| Enrollment | 82 | To measure initial interest and participation |
| Attendance | 79 | To measure ongoing participation |
| Performance/Outcomes | 66 | Evaluation of whether a student/program meets the desired results from programs and coursework |
| Student Projects | 52 | A student-centered process to present deeper knowledge, often through a presentation |
| Retention | 51 | Metric of students remaining enrolled |
| Interviews | 40 | Purposeful conversations that are structured to collect data through semi-structured questions |
| Workshops | 40 | A participatory, interactive learning session exploring a specific topic |
| Expert Assessment | 33 | Choosing specialized professionals to evaluate a program's strengths, weaknesses, and improvements |
| Graduation | 29 | Metric of students who successfully complete program |
| Instructor Observations | 26 | To measure student participation and knowledge retention |
| Focus Groups | 21 | Small groups in moderated discussion to gather impressions and specific program details |
| Products/Services | 21 | Measurements of quality and type of educational services, including allocated resources and how students achieved learning outcomes |
| Other | 15 | Other, unclear methods |
| Not Provided | 9 | No identified evaluation methods identified in proposal |
| Demographics | 8 | Statistics to describe a population with factors such as ethnicity, age, race, gender, etc. |
| Case Studies | 8 | Narrative scenarios used as a contextual tool for analysis |
| Social Media Analysis | 8 | Using social platforms to analyze data about patterns in user-generated content and impacts |
| Pilot Testing | 5 | A smaller trial run to identify potential challenges, (such as the study design and data retrieval) that allows for adjustments to larger programs |
Of the grants analyzed, 64.1% employed a mixed-methods approach, integrating both quantitative and qualitative strategies; 28.1% relied exclusively on quantitative methods; 3.9% used only qualitative methods; and in 3.9% of cases, the methodology was unclear.
EXPANDED META-ANALYSIS
SUMMARY: PROJECT METHODOLOGY OVERVIEW
Funded projects employ a diverse array of methodologies to address both student and institutional needs. This diversity reflects adaptability to local conditions while maintaining fidelity to overarching program objectives.
Most instances of representative strategies included
- Peer mentoring and leadership training
- Integration of academic coursework with community college transfer pathways
- Instruction in sustainability, water management, and bioenergy
- Professional development for faculty and staff
- Curriculum redesign and laboratory modernization to enhance learning environments
SUMMARY: PROGRAM TITLE ANALYSIS
Titles predominantly reference experiential learning, leadership development, and community engagement, underscoring the prioritization of applied learning models that facilitate direct connections between academic study and professional pathways.
Of special interest was whether program titles and descriptions included primary goals of student recruitment. The number of grants in each state mentioning student recruitment as primary goal:Texas (10 out of 50); California (8 out of 35); Puerto Rico (4 out of 15); Arizona (3 out of 11); Florida (2 out of 5); New Mexico (2 out of 10); Colorado (1 out of 3); New Jersey (1 out of 1).
EXPANDED META-ANALYSIS
SUMMARY: KEYWORD FREQUENCY ANALYSIS
A keyword frequency analysis of each program reinforces an emphasis on academic preparation, career readiness, and community engagement within agricultural and STEM contexts.
- EXPERIENTIAL LEARNING (14)
- AGRICULTURE (10)
- HISPANIC (7)
- HSI (7)
- LEADERSHIP (5)
- NUTRITION (5)
- EDUCATION (5)
- RECRUITMENT (5)
- RETENTION (4)
- INTERNSHIPS (4)
SUMMARY: THEMATIC PRIORITIES
An analysis of project summaries reveals goals and objectives that demonstrate a dual focus on student-centered educational enhancement and workforce-aligned program development:
- Expanding Hispanic student enrollment and achievement in agriculture/STEM fields
- Developing leadership competencies and career trajectories in food, agriculture, and sustainability sectors
- Enhancing academic programs through experiential learning opportunities
- Addressing regional workforce needs through specialized training
EXPANDED META-ANALYSIS
SUMMARY: OBJECTIVES ANALYSIS
Objectives consistently align with USDA and HSI goals, including:
- Recruit and prepare Hispanic students for agriculture/ STEM careers
- Provide integrated academic and experiential learning experiences
- Support student retention and on-time graduation
- Strengthen faculty capacity for effective mentorship and instruction
- Sustain and enhance institutional academic offerings
SUMMARY: CROSS-COMPARISON OF PROJECT IMPACT
A cross-sectional analysis of project outcomes confirms that most programs directly engage students, support academic persistence, and create clear links between educational success and workforce opportunities.
- 132 SUPPORTED STUDENTS
- 103 ENCOURAGED RETENTION AND GRADUATION
- 115 PROVIDED ACCESS TO CAREERS IN FOOD AND AGRICULTURE
- 84 ALIGNED WITH ACADEMIC DEVELOPMENT
- 49 FOSTERED FACULTY AND STAFF PROFESSIONAL DEVELOPMENT
EXPANDED META-ANALYSIS
SUMMARY: PROGRAM FINDINGS
One strategy – collaboration between institutions of higher learning – seems to be on an upswing. For example, the “Leveraging Interdisciplinary Nutritional Knowledge” (LINK) Program ia a partnership between six Hispanic-Serving Institutions in Southern California: two 4-year universities, California State University (CSU) Long Beach and CSU Fullerton, and four two-year community colleges including Cerritos, Fullerton, Long Beach City, and Santa Ana Colleges. Such wider networks strengthen the program’s focus about Latino nutrition through outreach, mentoring, education, support services, research, and professional internships with community partners.
Collectively, the evaluated programs exhibit a strong institutional commitment to advancing Hispanic student success in agriculture and STEM disciplines. The concentration of grants in high-impact states, coupled with a prevalent focus on experiential learning and leadership development, indicates that these initiatives are successfully facilitating transitions from academic settings to professional careers. Sustained investment in faculty development, curriculum innovation, and program scalability will be critical to maintaining these outcomes and expanding the reach of proven models.
CLOSING DISCUSSION FORUM
OVERVIEW
Participants engaged in open dialogue and reflection, which centered on the evolving nature of evaluation practices within HSI-funded programs and broader collaborative efforts. Participants appreciated the diversity of viewpoints and methodological approaches shared throughout the conference and expressed gratitude for sessions that revisited the historical evolution of evaluation and how the field has adapted over time to meet emerging challenges and opportunities. Participants reinforced the value of evaluation not just as a compliance activity, but as a tool for strategic learning, institutional growth, and field advancement. Attendees expressed interest in collaborative synthesis efforts and agreed on the importance of continuing these conversations beyond the conference, to ensure the long-term impact.
KEY THEMES
- Challenges in Reporting and Evaluation Practice: One participant shared insights from managing multiple USDA-funded projects and expressed concern over the limitations of annual reports, which often emphasize outcomes over deeper qualitative insights. This sparked discussion on the disconnect between rich, nuanced evaluation work being conducted and the restrictive formats of required reporting.
- Navigating Funder Expectations: There was shared uncertainty about the extent to which federal funders, particularly USDA, value or expect robust qualitative evaluations beyond mandated outcomes. Participants noted that while current reporting formats may not fully accommodate deeper evaluations, this work remains vital for internal learning, stakeholder engagement, and future funding proposals. It was reiterated that fulfilling RFP requirements is the minimum, with broader evaluations serving more strategic purposes.
- Sustaining Institutional Knowledge: Participants discussed the risk of losing valuable insights when programs sunset without efforts to disseminate their findings and supported the idea of synthesis as a means to preserve institutional knowledge and prevent redundancy.
OVERVIEW
RECOMMENDATIONS
CROSS-INSTITUTIONAL AND CROSS-FUNDER COLLABORATION:
Participants highlighted opportunities to think beyond USDA and foster broader collaborations across institutions and funding mechanisms. These efforts could help frame evaluations around common goals and generate more impactful, unified narratives.
THE NEED FOR SYNTHESIS AND COLLECTIVE IMPACT:
Dr. Michael Quinn Patton advocated for moving beyond individual project silos toward a network-level synthesis of impacts, lessons learned, and shared challenges. He proposed a field-wide initiative to document and integrate evaluation insights across projects—especially those concluding within the next 12–18 months. Such a synthesis could serve as a transformative tool for advocacy, future funding justification, and field development.
He suggested looking to models like the International Initiative for Impact Evaluation (3ie) for guidance on synthesis methods and proposed a phased, participatory approach using techniques such as Delphi surveys to gather and refine insights. He also recommended leveraging small funding contributions from existing grants to support synthesis work, such as hiring graduate students or developing joint publications.
EXECUTIVE SUMMARY
Pulling together all recommendations from speakers and participations, the results reflect a maturation of evaluation practice from project‑bounded assessment toward system‑level inquiry. The following ten principles emphasize:
- Embedding evaluation at program inception;
- Articulating shared theories of transformation across partners;
- Using adaptable, mixed‑methods designs aligned with complex contexts;
- Centering stakeholder voice; and
- Building durable collaborative infrastructures—within teams, across institutions, and among funders—for collective impact.
1) CORE PRINCIPLES FOR EVALUATION
- Front‑end influence over back‑end audit: Embed evaluation during design to shape program goals, logic, and adaptive strategies—positioning evaluation as a generative design partner.
- System‑level orientation: Model relationships and context to trace how structures and change agents interact to produce outcomes across time.
- Principle‑driven praxis: Anchor equity, sustainability, and responsiveness as guiding principles that ensure values while allowing contextual fit.
- Evaluative thinking as capacity: Cultivate reflective learning, narrative sense‑making, and judgment about merit and worth—not only literacy measures.
2) METHODOLOGICAL GUIDANCE
- Actionable questions: Focus on judgments of merit and worth (effect, contribution, equity) beyond descriptive activity reports.
- Mixed‑methods with narrative integration: Strengthen quantitative indicators with interviews, focus groups, case studies, reflections, and contextual data to highlight mechanisms and unanticipated impacts.
- Developmental innovative evaluation: Enable rapid feedback and adaptation when programs are emergent or uncertain.
- Responsive and disaggregated analysis: Separate by demographic groups and foreground lived experiences to avoid masking disparities and to align with servingness.
- Guard against metric distortion: Apply contextual interpretation and attend to Goodhart’s Law; evaluate systems (implementation quality and experience) and not proxies alone.\
3) DESIGN TOOLS
- Theory of change and logic models: Use as visual reasoning tools to clarify causal narratives and align activities and outcomes; co‑create with leadership, advisors, evaluators, and community partners.
- SMART outcomes and data matrices: Operationalize success early via indicator matrices that specify measures, sources, collection cycles, and analyses.
- Template ecosystems: Leverage public templates (e.g., AAAS, NIFA, EvaluATE) to standardize core elements while preserving contextual adaptability.
4) STAKEHOLDER ENGAGEMENT
- Participatory orientation: Engage students, faculty, advisors, and community stakeholders across design, implementation, interpretation, and dissemination.
- Early identification and wrap‑around supports: Establish early‑term detection and comprehensive supports (academic, financial, cultural) for progression and completion.
- Authentic community: Build cohorts to foster belonging and accountability while maintaining flexibility for students’ lived realities.
- Broaden success criteria: Move beyond enrollment counts toward outcomes that capture experience, agency, and structural change.
5) COLLABORATION
- Networked collaboration: Dissolve organizational silos; cultivate partnerships grounded in shared theories of transformation; seek cross‑funder alignment.
- Team‑science infrastructure: Formalize norms via team handbooks (onboarding, governance, coordination), rotate leadership roles, and conduct monthly reflective check‑ins and annual vision surveys.
- Attribution and recognition: Ensure visible credit for all contributors across publications, reports, and communications.
- Psychological safety by design: Allocate time and procedural flexibility so complexity can surface early for improved learning and course correction.
6) COMMUNICATION AND PROPOSAL DEVELOPMENT
- Strategic communications: Craft concise narratives and memorable project titles; treat communication as a core workstream.
- Culture of grant development: Provide structured ideation time, adapt proposals to shifting priorities, and leverage technical-assistance hubs and federal partnerships.
- Student ambassadors and educator incentives: Employ students as peer recruiters and storytellers (with wages/waivers) and recognize educator participation with stipends.
7) PROGRAM RESOURCES AND EXPERIENTIAL LEARNING
- Sustained capital investments: Procure equipment that extends learning capacity beyond grant periods to create durable institutional assets.
- Internships and industry partnerships: Expand authentic, mentored research and workplace experiences to accelerate skill formation and placement.
8) MULTI - INSTITUTIONAL, CROSS-FUNDER STRUCTURES
- Consortia for shared infrastructure: Coordinate data standards, common indicators, and shared services to reduce duplication and increase comparability.
- Move beyond single‑agency frames: Explore alignments across complementary funders to enable integrated narratives of impact and diversified sustainability.
9) COLLECTIVE IMPACT
- From project silos to network synthesis: Undertake a participatory, phased analysis of impacts and lessons—prioritizing projects concluding within 12–18 months—to produce sector‑level insights.
- Methodological explorations: Consider evidence gap maps, realist synthesis, and Delphi surveys; fund lightweight coordination (e.g., graduate RA support, joint publications).
10) IMPLEMENTATION ROADMAP (TIMELINE)
- Initiate (Months 0–3): Convene cross‑functional design teams; co‑create theory of change and SMART outcomes; map stakeholders; define data matrix; select templates; establish recognition norms and psychological‑safety routines.
- Build (Months 3–9): Pilot mixed‑methods instruments; create early‑alert student supports; launch ambassador program; formalize team handbook; procure durable equipment; negotiate cross‑institution data standards.
- Adapt (Months 9–18): Apply developmental evaluation cycles; run monthly reflective debriefs; publish interim learning briefs; calibrate metrics against Goodhart risks; document qualitative cases illustrating mechanisms and equity impacts.
- Synthesize (Months 12–24): initiate Delphi‑informed, field‑level synthesis; align with cross‑funder priorities; produce meta‑narratives and toolkits; disseminate findings with visible contributor credit.
CONCLUSION
These recommendations create an evaluation paradigm oriented to complexity, equity, and learning. By embedding evaluation at inception, grounding inquiry in principles and systems, integrating mixed‑methods evidence, and scaffolding collaboration from the project team to the broader field, institutions can generate credible judgments of value while strengthening advocacy, informing policy, and advancing the science and practice of evaluation across contexts.
POST-CONFERENCE FEEDBACK SURVEY FINDINGS
A post-conference survey was disseminated to participants via Google forms. Findings indicate that participants of the Beyond Metrics HSI Conference appreciated the opportunity to engage with current best practices in evaluation, particularly logic models and metrics for assessing program success. Key takeaways included the importance of using evaluation as a learning tool, the need to communicate performance to stakeholders, and the challenges shared by other USDA HSI grantees, specifically in regard to meeting enrollment goals in rural contexts.
Attendees valued insights from experienced evaluators, such as Dr. Michael Quinn Patton’s keynote, which offered perspective and inspiration to those newer to the field. Looking ahead, participants indicated interest in deeper exploration of practical evaluation tools, co-creating accessible evaluation narratives, and better understanding USDA's conceptualization of success.
SUGGESTIONS FOR FUTURE EVENTS INCLUDED:
- IMPROVED SCHEDULING (AVOIDING GRADUATION PERIODS)
- RESOURCE-SHARING
- MORE ADVANCE NOTICE
- CONTINUED OPPORTUNITIES FOR INTERACTIVE GROUP DISCUSSIONS AND PEER LEARNING
ADDITIONAL FEEDBACK FROM PARTICIPANTS WHO SENT UNSOLICITED EMAILS TO EVENT ORGANIZERS INCLUDED THE FOLLOWING QUOTES:
I'd like to join others in evaluation and other discussions post conference.
The conference provided a valuable community during this time of grant upheaval.
Thank you so much for a great and inspiring conference.
APPENDICES
- 42-43 CONFERENCE AGENDA
- 44-47 SPEAKERS AND COLLABORATORS
- 48 LINKS
CONFERENCE AGENDA: DAY 1
THURSDAY, MAY 15, 2025
| Session | Time | Speaker/Lead | Title and Topic |
|---|---|---|---|
| Welcome | 9:00-9:10 | Maral Kismetian, MPA, CRA. | WELCOME Introductions in Chat |
| Consultancy Scholarships Opportunity | Susan M. Pheasant, Ph.D. | EVALUATION “OFFICE HOUR” OPPORTUNITY (Grant paid for 30 or 60 minute office hour with speakers) |
|
| Set Stage for Today’s Agenda- Part 1 | 9:00-9:15 | Maral Kismetian, MPA, CRA Nancy Van Leuven, Ph.D. |
BUCKLE UP, OPEN UP (mind), SHARE UP (questions, stories) Who we are,why we wanted to bring all of you together -Storytelling for meaningful change -Telling story of HSIs -Telling story of project impacts |
| Set Stage for Today’s Agenda- Part 2 | 9:20-10:00 | Nancy Van Leuven, Ph.D. | META ANALYSIS OF PROJECTS How projects attracted and supported students; enhanced postsecondary education instruction to encourage retention and graduation; provided opportunities and access to food and agriculture careers; supported academic development and career attainment; and, fostered professional development for faculty/staff |
| Toolbox Building | 10:00-10:30 | Sarah Hug, Ph.D. | FOUNDATIONAL BUILDING BLOCKS FOR EVALUATION -Theory of Change with handout -Awareness Skill knowledge -Social network analysis |
| Toolbox Building | 10:30-11:00 | Sara Bolduc, Ph.D. | IT’S ABOUT PEOPLE, ISN’T IT Tools to strengthen working with large teams within and across disciplines, institutions, and time zones to conduct research, implement projects, and broaden participation |
| Keynote | 11:00-12:00 | Michael Quinn Patton, Ph.D. | SUPPORTING AND EVALUATING FOOD SYSTEMS TRANSFORMATION |
| Break | 12:00-12:10 | Quick comfort break and grab lunch | |
| Toolbox Building | 12:10-12:45 | Cindy Ziker, Ph.D., MPH | INTERSECTION OF LOGIC MODELS |
| Recap and Reflections | 12:00-1:00 | Nancy Van Leuven, Ph.D. | TOUCH POINTS FOR TODAY, WHAT’S COMING UP TOMORROW - FAQs, recordings, and handouts sharable at end |
CONFERENCE AGENDA: DAY 2
FRIDAY, MAY 16, 2025
| Session | Time | Speaker/Lead | Title and Topic |
|---|---|---|---|
| Welcome | 9:00-9:10 | Maral Kismetian, MPA, CRA. & Susan M. Pheasant, Ph.D. | WELCOME BACK Any burning FAQs generated on Day 1 |
| Strategic Positioning | 9:10-10:00 | Azuri Gonzalez, Ed.D. | THE IMPORTANCE OF “H” AND “S” IN HSI |
| Lightning Round #1 | 10:00-10:15 | Rod Taylor | HSI PILOT PROJECT: IMPROVING FIRST-TO-SECOND YEAR RETENTION AND CLOSING EQUITY GAPS THROUGH A STEM ACADEMY |
| Lightning Round #2 | 10:15-10:30 | Jeff Bullock, Ph.D. | EXPERIENTIAL LEARNING AND RESEARCH TO STRENGTHEN AGRICULTURAL EDUCATION |
| Lightning Round #3 | 10:30-10:45 | Tim Ray, Ph.D. | INSPIRING HISPANIC STUDENTS TO PURSUE CAREERS IN AGRICULTURE AND TECHNOLOGY THROUGH EXPERIENTIAL LEARNING |
| Lightning Round #4 | 10:45-11:00 | Brian Palmer | LEARNING TO LEAD - NEXT GEN |
| Toolbox Building | 11:00-12:00 | Michelle Burd, Ph.D. | IT’S NEVER TO EARLY TO THINK ABOUT EVALUATION Parts of an evaluation plan: 1) Identifying a skilled evaluator; purposes of evaluation; actionable evaluation questions; diagramming tools - theories of change and logic models |
| Break | 12:00-12:10 | Quick comfort break and grab lunch | |
| Grant Leadership and Next Steps | 12:10-12:30 | Maral Kismetian, MPA, CRA | CONTINUITY AND CATALYSTS. Opportunities for current/future consideration: Does this conference become a catalyst to forming ________? |
| Concluding Remarks and Gratitudes | 12:00-1:00 | Maral Kismetian, MPA, CRA. & Susan M. Pheasant, Ph.D. | OPEN FORUM: Next steps? Technical assistance hubs? More training on utilization -focused evaluation logic? Logic models + evaluation? Tools? Pain points? Knowing who you are and who you are serving? |
MEET THE SPEAKERS & COLLABORATORS
Susan Pheasant, Ph.D. - Organizing Committee
Director, Institute for Food and Agriculture, California State University, Fresno
Susan is passionate about production agriculture, lifelong learning, and leadership. She eagerly embraces professional and personal opportunities to intertwine those core topics for the benefit of growth and doing good work. She currently serves as the Director for the Institute of Food and Agriculture at Fresno State in support of faculty, staff, and students as they conduct applied research and outreach with ag industry professionals and as “Cowkeeper” for the Pacific Coast Coalition - Dairy Business Innovation Initiative.
Nancy Van Leuven, Ph.D. - Organizing Committee
Nancy, while officially retired faculty from California State University, Fresno, continues to actively share her expertise in marketing and organizational change management with public corporations and private groups as an independent contractor. A former grantwriter, she has also published about health care for rural populations, how social media is embedded in public agencies, and why global initiatives, such as the UNGC Sustainable Development Goals, are facing obstacles with Western audiences and other stakeholders.
Maral Kismetian, MPA, CRA - Organizing Committee
Director of Grants Development, California Polytechnic State University - San Luis Obispo
Maral has extensive grants administration experience in research and sponsored programs operations, including grant management and coordination. She received her certification as a Certified Research Administrator (CRA) from the Research Administrators Certification Council in 2014. In 2021, she was selected to participate in the USDA E. Kika De La Garza fellowship program. She holds a Master’s degree in Public Administration from California State University, Fresno, and a BA degree in Political Science.
Sarah Hug, Ph.D.
Colorado Evaluation & Research Consulting
Dr. Hug is the director of Colorado Evaluation & Research Consulting. She is a social scientist by training, and practices social science research and external evaluation for a large number of federally funded education programs in science, technology, engineering, and math. Her background in education and the learning sciences informs her work studying inclusive excellence in STEM education. She graduated Magna Cum Laude from Purdue University in 1999 and received her PhD in Educational Psychology from the University of Colorado at Boulder in 2007.
Sara Bolduc, Ph.D.
Sara Bolduc Planning and Evaluation LLC
Dr. Bolduc has lived in Hawai’i for more than twenty years and has worked as a research consultant on a range of projects in Hawaiʻi and throughout the Pacific Region since 2009. In 2018, she founded Sara Bolduc Planning and Evaluation LLC (SBPE) and serves as Principal and President. She has expertise in policy and program evaluation, survey design and implementation, facilitation, community outreach, and land use and environmental planning. Dr. Bolduc also teaches courses at the University of Hawaiʻi, in the Department of Urban and Regional Planning.
Michael Quinn Patton, Ph.D.
Utilization- Focused Evaluation
Dr. Michael Quinn Patton is the Founder and CEO of Utilization-Focused Evaluation, an independent organizational development and program evaluation organization. He served 18 years as faculty at the University of Minnesota, including five years as Director of the Minnesota Center for Social Research and 10 years with the Minnesota Extension Service. He has authored numerous books on evaluation, including Blue Marble Evaluation, Principles-Focused Evaluation, Facilitating Evaluation, Developmental Evaluation, and Utilization-Focused Evaluation.
Cindy Ziker, Ph.D., MPH
Ziker Research
Dr. Ziker is the Executive Director and Chief Research Scientist for Ziker Research. Her background includes over a decade of education research and evaluation experience serving the Arizona Department of Education Assessment Section, local high schools and unified school districts, and the Maricopa County Education Service Agency. She holds an MA in Elementary Education, a Ph.D. in Education Psychology, an MS in Public Health Practice, and a Graduate Certificate in Large-Scale Assessment.
Azuri Gonzales, Ed.D.
Alliance of Hispanic Serving Research Universities
Diana Natalicio Institute for Hispanic Student Success
Dr. Gonzales serves as the Executive Director for the Alliance for Hispanic Serving Research Universities and the UT Regents’ Endowed Distinguished Director for the Diana Natalicio Institute for Hispanic Student Success. As a higher education professional, she has dedicated her career to the transformation of higher education in support of faculty community engaged scholarship, community partnerships and impact, student high-impact practices, and the overall alignment of institutional priorities for strategic change.
Rod Taylor, MBA
Columbia Basin Community College
Mr. Taylor is a Math, Science & Engineering Dean at Columbia Basin College based in Pasco, Washington. Previously, he served as the General Education, Academic Transfer, Basic Skills Associate Dean at Bellingham Technical College and has also held positions at Spokane Community College. He received a Bachelor of Arts degree from Whitworth University and an MBA from Gonzaga University.
Jeff Bullock, Ph.D.
Wenatchee Valley College
Dr. Bullock has been a faculty member at Wenatchee Valley College since 2018. In 2023, he was named the acting dean of math, science, and engineering. He holds degrees in chemistry, molecular biology, and biochemistry, along with a doctorate in plant pathology.
Tim Ray, Ph.D.
Chemeketa Community College
Dr. Ray serves as the Dean of Agricultural Science and Technology at Chemeketa Community College in Salem, OR. He is an experienced education specialist with a demonstrated history of working in the education management industry and is skilled in strategic planning, curriculum development, agriculture, public speaking, and research. He earned a Master of Science (M.S.) focused on Agricultural Teacher Education from The Ohio State University.
Michelle Burd, Ph.D.
Burd’s Eye View Center for Evaluation Research Synthesis
Dr. Burd is the Director of Evaluation Education for the Center for Evaluation Research, Synthesis and also serves as Principal and Owner of Burd’s Eye View, a one-woman shop researching and evaluating social programs for nonprofits, government agencies, and private companies to change lives. Her specialties include education (especially STEM), organizational learning, professional development, school improvement, systemic change, and instructional quality, among other social programs. Her academic background includes a BA in Psychology, Maitrise - psychologie genetique, and a Ph.D. in child development and family relationships.
Brian Palmer
Hartnell College
Mr. Palmer is a mathematics faculty member at Hartnell College in Salinas, California. In addition to teaching, he serves as a college mentor on the STEM Internship Program Team as well as a member of the Advisory Research Group Council. He was the Research Associate on the NSF ESTEEM MicroInternship program and the PI on the Innovations Around Interview Exams project, and is the PI on Hartnell’s NASA Murep Aerospace Academy program. His work at Hartnell centers around the interface between the social and classroom experience for STEM students, and how it can be used for more fulfillment, higher engagement, deeper learning, and improved outcomes for STEM students.
PRACTICAL TOOLS FOR YOUR EVALUATION TOOLBOX
SHARED BYOUR CONFERENCE SPEAKERS
These links emerged during the conference and should not be construed as neither an exhaustive list of resources and tools nor as endorsements.
CONSULTANT LINKS
- Utilization-Focused Evaluation
- Blue Marble Evaluation
- Developmental Evaluation — Utilization-Focused Evaluation
- Sarah Hug | LinkedIn
- Ziker Research | Evaluation & Research Consulting Services
- Evaluation | Sara Bolduc Planning and Evaluation LLC | United States
TOOLS
- STEM Education and Training TIG Week: Tools to Enhance the Functionality and Success of Science Teams by Sara Bolduc
- The Assessment Skills Framework RPA
- Goodhart's Law Definition & Examples
- Logic Model Planning Process | NIFA
- Frequently Asked Questions about Logic Models
- Generic Logic Model for NIFA Reporting
- How to Develop a Program Logic Model
- Logic models: A tool for effective program planning, collaboration, and monitoring
- AAAS: S-Stem Resource & Evaluation Center (REC) Developing and Using a Logic Model or Theory of Change in Your NSF S-STEM Grant Proposal (NSF 25-514)
- International Initiative for Impact Evaluation
- EvaluATE Ate Evaluation Resource Hub
ASSOCIATIONS
- American Evaluation Association
- AEA365 – A Tip-a-Day by and for Evaluators
- Arizona Evaluation Network
- Canadian Evaluation Society
- International Organization for Cooperation in Evaluation (IOCE) | Better Evaluation
This HTML version was generated with AI assistance and may contain errors or omissions. The original approved PDF remains the official document of record. An alternative accessible format is available upon request from the document creator in accordance with Title II of the ADA.