UC LEAFUC LEAFUniversity of Cincinnati

UC LEAF

Evaluation

Formative and Summative evaluations will result from data derived from both quantitative and qualitative research methods regarding programming, initiatives, research, and accountability. The multi-method approach for data collection proposed here will allow for both systematic and in-depth assessment of the process, strategies, and outcomes for sustainable change.

Pipeline Development

Our overall objective will be to “grow our own” women STEM faculty by enhanced recruitment, retention, and impact strategies designed to make UC a destination for women STEM faculty. 

Key activities: (a) Bottom-Up: workshops at three career levels and grants program; (b) Top-Down: training workshops, best practices seminars.

Baseline data: One of our important planned activities is to construct in the provost’s office the infrastructure needed to compile all of these data and do retrospective analyses since 2002.

Formative evaluation activities: Throughout the course of the project we will work with the Office of Equal Opportunity and with Institutional Research to create and maintain a database on the recruitment, hiring, retention, and career trajectory of STEM scientists, male and female. This will be used to assess progress toward our quantitative goals. In addition, we will annually survey all STEM scientists (with identifying information) to track publications, awards, professional memberships, etc. Project staff will meet periodically to discuss progress and adjust the work plan accordingly. Intermediate (at least quarterly) progress reports will be discussed by the project team and with the advisory boards.

We will evaluate the workshop series by asking participants to complete pre- and post-series assessments as well as session evaluations. The assessments will focus on questions relevant to the workshop content and series as a whole (i.e., empowerment, engagement, and impact, respectively).          

Periodically, interviews will be conducted with key informants (provostal staff, deans, heads, selected faculty members and selected advisory committee members) to assess the current status of the pipeline, identify success and continued challenges to identify ways in which our bottom-up and top-down strategies have had impact.

Summative evaluation activities: We will analyze hiring and retention data for the institution as whole and by STEM unit. The results will be presented as a flow chart from recruitment to hiring to retention to promotion, by year.

In addition, our external evaluator will conduct a web-based survey of all workshop participants in the 5th year of the project, which will provide an opportunity to assess how the succession of workshop series worked for those individuals who participated in more than one. Finally, our external evaluator will conduct key informant interviews in the 5th year of the project that will document key strengths and weaknesses of our efforts and results and summarize these in the form of lessons learned.

Transform the Climate

Key activities: (a) Bottom-Up: Learning communities and visiting scholars; (b) Top-Down: Mentoring of heads, site visits, shadowing at ADVANCE institutions, and logic models.

Baseline data: We have survey data from men and women STEM scientists at UC from 2009 and 2011. Beginning in the first year of the project, we will make modifications to our surveys to use the Work Environment Survey now used at Michigan State and Purdue.

Formative evaluation activities: All workshops for deans, heads, and faculty will be assessed by participants and via key informant interviews (several per year). 

LEAF staff will maintain logs of all development activities and share these activity reports with the advisory committees, who can help shape our efforts. The external evaluator will also periodically conduct key informant interviews designed to illuminate implementation problems.

Evaluation staff (internal and external) will evaluate and share with deans and provost ideas for the ways in which unit-level logic models can be improved.

Summative evaluation activities: We will conduct annual climate surveys of all STEM scientists (so as to compare responses by gender) through the course of the project.

We will conduct pre- and post-learning community surveys with participants. Women will be asked respond to a series of questions related to both professional issues and work-life balance issues from three perspectives: (a) how they view the problem, (b) how they would answer if their work climates were ideal, and (c) how they think their department heads would answer the questions.

We will maintain a data-base of all change activities initiated by learning communities as a whole or (as will often be the case for mid-career participants) by individual participants. We will determine which projects moved through to fruition and get assessments from the women, from their mentors, and from their unit heads as to the impact that the initiatives had. If the initiatives stalled or had no impact, we will try to learn why.

In year 5, summative evaluations of each unit’s logic model (and outcomes) will be assessed by the external evaluator.

Accountability and Advocacy Council

Key activities: (a) Provide feedback on initiative successes and challenges; and (b) Provide venue for safe, positive, effective communications between participants, program staff and administration.

Baseline data: Most of our assessments here will be qualitative.  However, this will include the tracking of documented changes in policy, records of engagement and activity coordinated across the bottom up and top down processes, and evaluations by participants of the efficacy of the AAC.

Formative evaluation activities: We will also work with our staff to keep track of the agenda issues Dr. Hardcastle deals with in her role of mediating top-down and bottom-up change processes.

In addition to survey data from participants, the evaluation staff will periodically conduct key informant interviews with council members and with relevant members of the advisory councils.

Summative evaluation activities: In year five summative evaluation of the survey data, as well as key-informant interviews regarding the key initiatives of the AAC will be assessed.

Intellectual Merit

The UC initiative stands out among many existing ADVANCE programs for a few reasons. First, like many, we plan reforms that are driven by the senior administration and deans. However, the reforms will be designed and implemented at the unit level and closely monitored by the project team and AAC. Second, like many initiatives, we are using developmental programming and encouraging the women participants to initiate their own efforts to reform culture. Unlike many programs we have a formal mechanism, the Accountability and Advocacy Council, to facilitate both these top-down and bottom-up efforts and help them produce actionable and sustainable change. In addition to the innovative programming, our proposed Social Science Study also offers unique insight to the process and outcomes of the key initiatives using multiple methodologies (e.g. discourse analyses) that have not been seen in previous ADVANCE grants.  This will uniquely contribute to the existing literature concerning trasformation.

Broader Impact

Through conference attendance and publications, including those concerning the significant social science research we propose, our work will influence other ADVANCE programs and higher education in general, as well as contribute to the corpus of research on the underrepresentation of women in the STEM disciplines. Members of our core team have expertise in gender bias, human resources, and policy research. We anticipate our work being presented at community psychology conferences, industrial and organizational psychology conferences, and human resources/management conferences, as well as being published in the corresponding disciplinary journals.  Many of the programs we propose are highly efficient and therefore useful across organizations seeking to promote inclusiveness and accountability.

Results from prior NSF support

Urmila Ghia is Co-PI on a STEP grant (DUE-0756921, 5/08-4/13, $2,000,000) to establish a seamless K-17 program.  A key activity Urmila Ghia is responsible for is the Academic-Year Research Experience for Undergraduates (AY-REU) Program, one of two major Programs for Pathways to Graduate School (the other is an 8-week Summer REU Program for 12 students/year).  The goal of the REU Program is to provide research training on the use of modern technology in conducting and disseminating research. The objective is to encourage talented undergraduates to enroll in graduate school by exposing them to research.The AY-REU program has been offered to 21 students in AY 2009-10 and 22 students in AY 2010-11 (total = 43 students).