Microsoft Word - FY17SummativeP1final[1].docx

Fall

Army Educational Outreach Program

2017 Summative Evaluation Report

PART 1: Executive Summary

May 2018

2017 Summative Evaluation Report | PART 1 Executive Summary | 1 |

1 | AEOP Consortium Contacts

U.S. Army Contacts

Matthew Willis, Ph.D. Andrea Simmons

Director, Laboratory Management Army Educational Outreach Program (AEOP) Director

Office of the Assistant Secretary of the Army Office of the Assistant Secretary of the Army

Acquisition, Logistics, and Technology Acquisition, Logistics, and Technology

matthew.p.willis.civ@mail.mil andrea.e.simmons.ctr@mail.mil

AEOP Cooperative Agreement Manager Battelle Memorial Institute – Lead Organization

Louie Lopez David Burns

AEOP Cooperative Agreement Manager (CAM) Project Director, AEOP CA

U.S. Army Research, Development, and Director of STEM Innovation Networks

Engineering Command (RDECOM) burnsd@battelle.org

louie.r.lopez.civ@mail.mil

Christina Weber

AEOP Deputy CAM

U.S. Army RDECOM

[email protected]

Evaluation Team Contacts – Purdue University

Carla C. Johnson, Ed.D. Toni A. Sondergeld, Ph.D. Janet B. Walton, Ph.D.

Evaluation Director, AEOP CA Assistant Director, AEOP CA Assistant Director, AEOP CA

carlacjohnson@purdue.edu tonisondergeld@metriks.com walton25@purdue.edu

Report AEOP_08_05232018 has been prepared for the AEOP Cooperative Agreement and the U.S. Army by Purdue

University College of Education on behalf of Battelle Memorial Institute (Lead Organization) under award W911 SR-15-

2-0001.

Full versions of the FY17 portfolio evaluation reports as well as individual program evaluation reports are available on

AEOP website: www.usaeop.com (About > Our Impact).

2017 Summative Evaluation Report | PART 1 Executive Summary | 2 |

2 | Table of Contents

AEOP Consortium Contacts Page 1

Table of Contents Page 2

Introduction Page 3

Summary of Findings and Recommendations Page 7

2017 Summative Evaluation Report | PART 1 Executive Summary | 3 |

3 | Introduction

The Army Educational Outreach Program (AEOP) vision is to offer a

collaborative and cohesive portfolio of Army sponsored science,

technology, engineering and mathematics (STEM) programs that

effectively engage, inspire, and attract the next generation of STEM

talent through K-undergraduate programs and expose them to

Department of Defense (DoD) STEM careers. The consortium, formed

by the Army Educational Outreach Program Cooperative Agreement

(AEOP CA), supports the AEOP in this mission by engaging non-profit,

industry, and academic partners with aligned interests, providing a

management structure that collectively markets the portfolio among

members, leveraging available resources, and providing expertise to

ensure the programs provide the greatest return on investment in

achieving the Army’s STEM priorities and objectives toward a STEM

literate citizenry, STEM savvy educators, and sustainable

infrastructure.

In FY17, the AEOP central application tool included 41,553 unique program participants, 32,947 were

youth program participants and 8,714 were adult participants. Adult participants included Army Scientists

and Engineers (S&Es) in various roles, such as mentors, judges, and presenters, as well as teachers

participating in the new RESET program. Of the total participants in 2017, 855 students and 40 teachers

were from 46 DoDEA schools from the Pacific, Europe and the U.S. The number of unique youth program

participants in 2017 (32,947) represented an increase from 2016 (30,973), but was still lower than in 2015

(38,039).

2017 AEOP Participation by Youth and Adults

Youth

Adults

CII

Camp Invention Initiative

1,425

112

CQL

College Qualified Leaders

229

206

eCM

eCYBERMISSION

21,277

3,253

GEMS

Gains in the Education of Mathematics & Science

2,845

510

HSAP

High School Apprenticeship Program

JSHS

Junior Science & Humanities Symposium

5,577

3,555

JSS

Junior Solar Sprint

892

327

REAP

Research & Engineering Apprenticeship Program

118

RESET

Research Experiences for STEM Educators and Teachers

SEAP

Science & Engineering Apprentice Program

113

119

AEOP Priorities

Goal 1: STEM Literate Citizenry.

Broaden, deepen, and diversify the

pool of STEM talent in support of

our defense industry base.

Goal 2: STEM Savvy Educators.

Support and empower educators

with unique Army research and

technology resources.

Goal 3: Sustainable Infrastructure.

Develop and implement a cohesive,

coordinated, and sustainable STEM

education outreach infrastructure

across the Army.

2017 Summative Evaluation Report | PART 1 Executive Summary | 4 |

Unite

358

402

URAP

Undergraduate Research Apprenticeship Program

Total 2016 AEOP Participants

32,947

8,714

In 2017 the AEOP portfolio included participation from various collaborating schools, laboratories,

Army/DoD S&E’s and other collaborating organizations. AEOPs involved participants from 3,476 K-12

schools (compared to 3,607 in FY16), including more than 1,333 Title I schools (an increase from 962 in

FY16). The portfolio of programs also involved 485 colleges/universities (a decrease from 568 in FY16),

including at least 92 HBCUs/MSIs (an increase from 69 in FY16). AEOP programs self-reported 77 Army

and DoD research and development laboratories or Army organizations involved in the work of the

programs. The AEOP worked with 77 Army-funded laboratories at colleges/universities (compared to 74

in FY16). There were 2,307 Army and DoD S&Es who participated in 2017 programming (compared to

1,287 in FY16).

Costs for the individual 2017 AEOP elements as well as the average cost per student for each program

element are detailed in the table below. Apprenticeship program costs ranged from $3,313 per apprentice

(REAP) to $8,186 per apprentice (CQL). The higher cost of CQL reflects the longer duration of the program,

which may take place in the summer or through portions of the academic year (sometimes lasting the

entire year). AEOP competitions ranged in cost from $140 per student (eCM) to $362 per student (JSHS).

GEMS, which is typically a 1-week summer STEM enrichment activity that takes place at Army laboratories,

had an average cost of $459 per student in 2017. While Unite, a 4-6-week summer STEM enrichment

activity for students from historically underserved and under-represented groups that takes place in an

existing University pre-collegiate program, had an average cost of $1,849 per student.

2017 AEOP Costs

Program Type

Program Cost

Cost Per

Participant

Average Stipend Per

Participant

CII

STEM Enrichment Program

(grades K-6)

$337,583

$237

CQL

STEM Apprenticeship Program

(undergraduate/graduate)

$1,874,600

$8,186

$7,620

eCM

STEM Competition (grades 6-9)

$2,980,003

$140

GEMS

STEM Enrichment Program (grades

5-12)

$1,306,404

$459

$311

HSAP

STEM Apprenticeship Program

(grades 9-12)

$230,961

$4,277

$3,044

JSHS

STEM Competition (grades 9-12)

$2,019,112

$362

JSS

STEM Competition (grades 5-8)

$150,000

$168

REAP

STEM Apprenticeship Program

(grades 9-12)

$390,924

$3,313

$2,127

RESET

STEM Educator Program

$141,661

$7,083

Varies by level

SEAP

STEM Apprenticeship Program

(grades 9-12)

$419,955

$3,717

$3,152

2017 Summative Evaluation Report | PART 1 Executive Summary | 5 |

As in previous years (FY14 to FY16), the apprenticeship programs had the highest cost per participant

while the competitions were the least costly of the AEOP elements on a per student basis for FY17. The

variation in costs between programs is largely due to the cost of participant stipends, which are dependent

upon the educational level of the student and duration of the program. Several programs appeared to be

more efficient in FY17 than in FY16 based upon their slightly lower cost per student participant in FY17

(CQL, eCM, JSS, Unite).

In regards to participation of the DoDEA and Army/DoD laboratories, there was a 26% increase in student

participation (855) compared to FY16 (679 students). Teacher participation decreased (FY17 40 teachers

compared to 57 in FY16) from DoDEA schools in the Pacific, Europe and the U.S. that participated in the

AEOP through the GEMS, eCM and JSHS program. Additionally, through the AEOP competition programs

(eCM, JSHS, JSS) and Unite, the AEOP engaged and collaborated with 242 organizations external to schools

and the Army and DoD laboratories (e.g., professional STEM organizations, businesses, Technology

Student Association state delegations, etc.).

Number of 2017 Collaborating Schools, Laboratories, Army/DoD S&Es, and Other Organizations

Program

K-12 Schools

Colleges/Unive

rsities

(represented

by participants

or serving as

host sites)

Army and

DoD

Research

Labs/

Army

Agencies

Army-

Funded

Universit

y Labs

Army and

DoD

Scientists

Engineers

(S&Es)

Other

Collaborating

Organization

Total

Title I

Total

HBCU/

MIs

Camp Invention

(CII)*

College

Qualified

Leaders (CQL)

102

206

eCYBERMISSIO

N (eCM)

776

290

1,204

Gains in the

Education of

Mathematics

and Science

(GEMS)

924

315

281

High School

Apprenticeship

Program (HSAP)

Unite

STEM Enrichment Program (grades

9-12)

$662,000

$1,849

URAP

STEM Apprenticeship Program

(undergraduate)

$246,405

$4,176

$2,924

2017 Summative Evaluation Report | PART 1 Executive Summary | 6 |

Junior Science

and Humanities

Symposium

(JSHS)

1,024

378

112

246

200

Junior Solar

Sprint (JSS)

312

Research and

Engineering

Apprenticeship

Program (REAP)

Research

Experiences for

STEM Educators

(RESET)

Science and

Engineering

Apprentice

Program (SEAP)

119

Unite

149

110

University

Research

Apprenticeship

Program

(URAP)

Total Sites

3,404

1,287

485

163

242

Data for the 2017 AEOP portfolio evaluation were collected and analyzed by Purdue University, the

evaluation arm of the Lead Organization (LO) of the AEOP CA, Battelle Memorial Institute. With the

support of the AEOP CA Consortium Members, Individual Program Administrators (IPAs), and Government

POCs, evaluation studies for the CQL, eCM, GEMS, HSAP, JSHS, JSS, REAP, RESET SEAP, Unite, and URAP

programs as well as an alumni evaluation were completed by the Purdue University team.

The FY17 AEOP program evaluation utilized participant questionnaires, 21

Century Skill Assessments, as

well as focus groups and/or interviews with participants and adults who led educational activities or

supervised research projects (herein called mentors). This report summarizes the 2017 evaluation of the

AEOP portfolio. Eleven individual program evaluation reports are available under separate cover.

Executive summaries for these eleven reports are attached as appendices to this document. This report

includes a program overview, evaluation and assessment strategy, study sample, and evaluation findings.

The final section offers evidence-based recommendations intended to inform decisions for future

program development.

2017 Summative Evaluation Report | PART 1 Executive Summary | 7 |

4 | Summary of Findings and Recommendations

The 2017 AEOP evaluation collected data about participants, their perceptions of program processes,

resources, and activities, and indicators of achievement related to outcomes aligned with AEOP, Federal

guidance and program objectives. A summary of findings and recommendations for FY18 are provided.

2017 Summary of Findings - Near Term

Priority 1: STEM Literate Citizenry

Broaden, deepen, and diversify the pool of STEM talent in support of our Defense Industry Base.

Finding #1

Growth in Overall Participation and Some Program Participation. In FY17, the AEOPs

increased participation overall 6%, from 30,972 in FY16 to a total of 32,947 participants in

STEM programs, STEM competitions, and STEM apprenticeship programs. This increase

reflects outcomes of increased investments in marketing and promoting AEOPs through a

variety of methods at local, state, and national levels and reverses a three-year downward

trend from enrollments of 41,802 in FY14, 38,039 in FY15, and 30,972 in FY16 respectively.

Programs that experienced participation increases in FY17 include: CII 17% growth (1,425

compared to 1,185 in FY16); eCM 3% growth (21,277 compared to 20,607 in FY16); GEMS

15% growth (2,845 compared to 2,427 in FY16); JSHS 5% growth (5,577 compared to 5,300

in FY16); JSS 34% growth (892 compared to 585 in FY16); Unite 21% growth (358 compared

to 282 in FY16); URAP 12% growth (59 compared to 52 in FY16). JSHS and JSS reversed a

downward trend in enrollment in FY17. It is important to note that in previous years, prior

to the implementation of the use of the Cvent online registration system, most of AEOP

program participation data were self-reported.

Finding #2

Decline in Participation for Most Apprenticeship Programs. Despite overall growth in

participation and some growth for programs mentioned in Finding #1, the remaining four

apprenticeship programs stayed the same or experienced a decline in participation for

FY17. CQL 3% decrease (229 compared to 236 in FY16); HSAP 17% decrease (54 compared

to 65 in FY16); REAP 2% decrease (118 compared to 120 in FY16); and SEAP stayed the same

at 113 for FY17 and FY16.

Finding #3

Three-Year High Number of Applications to Participate in AEOPs – However, Placement

Rates Declined in FY17 for some AEOPs. The overall placement rates across AEOPs

decreased from 83% from FY16 to 68% in FY17, despite a three-year high number of

applications submitted to participate in AEOPs. For FY17, there were 46,518 applications,

an increase of 20% over the 37,399 applications received in FY16 and a 4% increase over

the number of applications received in FY15 when 44,632 applications were received. As a

result of the increasing number of applications, apprenticeship programs have

experienced a downward trend in placement rates due to limitations in funding and

availability of placements/mentors. CQL placed 41% of applicants in FY17, as compared to

51% in FY16; HSAP placed 9% of applicants in FY17 as compared to 18% in FY16; REAP

placed 17% of applicants as compared to 25% in FY16; URAP placed 9% of applicants in

2017 Summative Evaluation Report | PART 1 Executive Summary | 8 |

FY17 as compared to 29% in FY16; and SEAP placed 13% of applicants as compared to 16%

in FY16. However, placement rates grew slightly for STEM enrichment activities. GEMS

placement increased from 55% in FY16 to 61% in FY17 and Unite enrollment grew from

41% in FY16 to 45% in FY17. Acceptance rates for STEM enrichment programs increased

in FY17 (61% GEMS, 38% Unite) as compared to FY16 when 55% of GEMS applicants and

41% of Unite applicants were selected for these programs. eCM continued to accept all

applicants in FY17 to participate in the program, as in previous years. The JSHS

competition does have restricted participation due to regional capacities. However, JSHS

increased placements to 65% in FY17 compared to 60% in FY16.

Finding #4

AEOPs Continued to Serve Underserved Populations. The AEOPs continued to prioritize

the participation of students from traditionally underserved groups, per the AEOP

definition: AEOP’s definition of underserved includes at least two of the following: low-

income students; students belonging to race and ethnic minorities that are historically

underrepresented in STEM; students with disabilities; students with English as a second

language; first-generation college students; students in rural, frontier, or other federally

targeted outreach schools; females in certain STEM fields.

As reported by AEOPs, apprenticeship programs included 38% of underserved students in

their total population. CII achieved 100%, while Unite registered 65% and REAP 54%

respectively. The next highest enrollment of underserved students was eCM with 45%.

GEMS and JSS both had 29% participation of underserved students. All remaining programs

had 19% or less underserved participation – with CQL and SEAP having the lowest

percentages at 6%. Others included: HSAP (19%); JSHS (19%); and URAP (8%).

Finding #5

Participants reported engaging in STEM practices significantly more in their AEOP

programs as compared to in their typical school experiences for each program. Evaluation

findings indicated that AEOPs consistently provided opportunities for participants to

engage in authentic STEM activities that are significantly more intensive than those they

experience in their typical school settings.

Finding #6

Participants reported increased STEM competencies, STEM skills, STEM knowledge, STEM

practices, and confidence in STEM after participating in AEOPs. The programs with the

highest level of agreement (some gain to large gain) with growth in 21

Century STEM Skills;

STEM Knowledge; and STEM practices after participation included: CQL, eCM, GEMS, HSAP,

JSHS, REAP, SEAP, Unite, and URAP. Participants from all programs indicated some to a large

gain in their STEM identity after participation. Participants from CQL, eCM NJ&EE, GEMS,

HSAP, REAP, SEAP, Unite, and URAP reported 90% or higher agreement with the statement

“I am more confident in my STEM knowledge, skills, and abilities” after participating in the

AEOPs.

Finding #7

Participants demonstrated increased attainment toward mastery of the 21

Century

Skills across their participation in the AEOPs in the FY17 pilot of the assessment.

Participants from apprenticeship programs (REAP, URAP, HSAP) and STEM enrichment

program Unite demonstrated growth in all areas of the 21

Century Skills Assessment from

baseline (first days of program) to end of program. Participants exhibited growth in

Creativity & Innovation; Critical Thinking & Problem Solving; Communication,

Collaboration, and Social and Cross-Cultural Skills; Information, Media, & Technological

Literacy; Flexibility, Adaptability, Initiative, & Self-Direction; Productivity, Accountability,

Leadership, & Responsibility.

2017 Summative Evaluation Report | PART 1 Executive Summary | 9 |

Finding #8

Participants reported positive attitudes toward Army/DoD STEM Research. AEOP

participants in CQL, eCM NJ&EE, GEMS, HSAP, REAP, SEAP, Unite, and URAP reported 75%

or more agreement with the statements: “DoD researchers advance science and

engineering fields”, “DoD researchers develop new cutting-edge technologies”, “DoD

researchers solve real-world problems”, and “DoD research is valuable to society”.

Programs that reported less than 75% agreement with the statements included JSHS, JSS,

and eCM (regional).

Finding #9

Evaluation findings indicated that the AEOP exposed participants to STEM careers

generally and to Army and DoD STEM careers, and participating in AEOPs increased their

interest in pursuing STEM careers. A majority of participants (range of 53%-97%) in CQL,

eCM National, GEMS, HSAP, JSHS, REAP, SEAP, and Unite reported learning about 3 or more

STEM careers. Fewer students (range of 32%-44%) in eCM Regional, JSS, and URAP had

learned about 3 or more STEM careers. In regards to specific DoD STEM Careers,

participants reported less exposure in FY17 AEOPs than to STEM careers overall. However,

a majority of students (range of 64%-84%) in all programs except for eCM Regional (39%)

were more interested in pursuing STEM careers after their AEOP participation and more

than half of responding apprentices reported interest in DoD STEM careers in FY17 (range

of 66%-87%).

Priority 2: STEM Savvy Educators

Support and empower educators with unique Army research and technology resources.

Finding #1

Adult participants (i.e. mentors, S&E’s, Team Advisors, teachers) reported use of effective

mentoring strategies in varying degrees across the AEOPs in FY17. Strategies to engage

students in authentic STEM activities (range of 82%-94%) were used most frequently, while

strategies to support participants STEM educational and career pathways (range of 47%-

69%) were used the least. A majority of all adults (range of 63%-86%) reported using

strategies to support the needs of diverse students as learners. Further, a large majority of

adults (82%-94%) reported the use of authentic STEM activities.

Finding #2

In FY17, participants continued to be satisfied with the support received from their

mentor/S&E/Team Advisor/teacher. Most apprentices and students in all programs

reported high levels of satisfaction with the mentorship they received and the quality of

instruction they received (range of 62%-84%). The levels of satisfaction for several

programs, CQL, GEMS, REAP, SEAP, and Unite, were somewhat lower than those reported

in FY16, however levels satisfaction in HSAP and URAP were higher than in FY16. Overall,

the percentage of satisfaction with instruction or mentorship in FY17 was very similar to

that reported in FY16 (range of 62%-83%).

Priority 3: Sustainable Infrastructure

Develop and implement a cohesive, coordinated, and sustainable STEM education outreach infrastructure across the

Army.

Finding #1

The primary means of learning about AEOPs and associated opportunities in FY17

continues to be personal connections, school/university connections, past participants,

or someone connected directly with AEOPs. A continued strength of AEOP is the expansive

network of connections to local communities that serves as a continued means of

recruitment for the program. Overwhelmingly, participants and mentors reported that

AEOP social media, AEOP website, and other materials were much less frequently used as

a means for introducing them to the program.

2017 Summative Evaluation Report | PART 1 Executive Summary | 10 |

Finding #2

Despite limited awareness of participants and mentors of the full AEOP and DoD/Army

portfolio of opportunities, FY17 participants reported interest in continuing on to

participate in another AEOP in the future. Some individual programs made progress in

FY17 in increasing awareness of AEOP programs overall. However, participants in some

programs were not aware of other opportunities within AEOP.

Finding #3

Participation in the AEOP evaluation in FY17 improved for apprentices/students in most

programs. Mentor/adult questionnaire completion is still less than desired. Programs

including CQL, eCM NJ&EE, GEMS, HSAP, REAP, SEAP, URAP, Unite all improved

apprentice/student participation in FY17 – reaching 46% to 94% response rate. Mentors

from HSAP, REAP, and URAP also achieved acceptable return rates.

2017 Summary of Findings - Mid to Long Term

Priority 1: STEM Literate Citizenry

Broaden, deepen, and diversify the pool of STEM talent in support of our Defense Industry Base.

Finding #1

AEOP alumni indicated interest in pursuing a STEM degree and career. A majority of

alumni participating in the survey indicated they were both interested in earning a STEM

degree (88%) and pursuing a STEM career (88%).

Finding #2

Alumni are interested in completing additional elective STEM courses and other STEM

opportunities. More than three-quarters of alumni reported interest in taking elective

STEM courses (80%), learning about new things in STEM (83%), and potential STEM

projects/experiments in a university or professional setting (87%).

Finding #3

AEOP Alumni continue to be engaged in STEM. Nearly three-quarters or more of alumni

reported sometimes or frequently engaging in activities such as: learning about new

things in STEM (88%), talking with family and friends about STEM (75%), and solving

math/science puzzles (73%).

Finding #4

Alumni hold positive views toward STEM generally and Army/DoD STEM specifically. Over

80% of AEOP alumni believe that all people can be successful in STEM. Alumni report

agreement with the following statements: I enjoy solving real-world problems (97%); STEM

careers are a good fit with my interests (97%), I feel successful in STEM classes (95%), and I

can use STEM to help improve my community (95%). In regards to Army/DoD STEM attitudes

specifically, 95% percent or more of alumni indicated feeling Army/DoD research is valuable

to society, 97% agree that the Army/DoD solves real-world problems, as well as develops

new, cutting edge technologies.

Finding #5

Alumni report interest in STEM careers generally, as well as with the Army/DoD

specifically. Nearly all alumni reported being interested in pursuing a STEM career (93%)

in general. Three-quarters indicated they were aware of Army/DoD STEM careers (75%),

and 82% of alumni indicated they would be interested in learning more about Army/DoD

STEM careers. Approximately two-thirds (64%) of alumni were interested in pursuing an

Army/DoD STEM career at the present time.

2017 Summative Evaluation Report | PART 1 Executive Summary | 11 |

What AEOP Participants are saying…..

“I have had a fantastic experience [in CQL]. I owe a lot to my mentors who guided me every step of the way.

Thanks to them, I have had the opportunity of publishing scientific articles, giving talks at conferences, and

performing cutting edge research. I recently was accepted into graduate school for Ph.D. studies and I believe

a major part of my acceptance was the experience I gained from this program.” --CQL Apprentice

“[CQL] provides full immersion of students into nonacademic labs to gain further experience in STEM

programs to understand other available professional paths. It is an excellent summer program and I wish I

had participated when I was in college.” --CQL Mentor

“I never really thought of pursuing STEM because it seemed like a job where you had to sit behind a desk and

type on a computer…After doing eCM and talking to the Army officers and everyone, it makes it seem like

there are so many more opportunities.” --eCM-NJ&EE Student

“I think the [eCM] program is so well organized and it’s got so many resources that I can get a group of sixth

grader to stick with a topic and work on a project for almost an entire year… it’s very open-ended and they

get to pick a topic. It’s part of their community. They feel that direct connection. They take ownership for

what they’re doing. They become experts in what they’re doing. They develop patience. They’re collaborating.

They’re managing time. Their developing all of these skills, I think, is invaluable.” --eCM Adult Participant

“[In GEMS], you get to meet people who do the jobs and hear their side of the story about what the [jobs

are like] that you may be considering going in to.” --GEMS Student

Finding #6

35% of AEOP Alumni reported enrollment in a STEM degree program. Engineering was the

highest enrolled field (11%), followed by medicine (7%), life science (6%), physical science

(5%), mathematics or statistics and technology or computer science (2% each respectively),

followed by Earth science and business (1% each respectively) and other 4%.

Priority 2: STEM Savvy Educators

Support and empower educators with unique Army research and technology resources.

Finding #1

Participants reported very positive impacts of their mentors and agreed mentoring is a

valuable aspect of AEOPs. Many alumni also believed their AEOP mentor helped influence

their future academic career decisions (83%), and helped them learn about Army/DoD

careers (78%). While the reported mentoring relationships appeared to be strong, nearly

half indicated they have stayed in touch with their AEOP mentor after the program (43%).

Priority 3: Sustainable Infrastructure

Develop and implement a cohesive, coordinated, and sustainable STEM education outreach infrastructure across the

Army.

Finding #1

Alumni reported strong interest in participating in other AEOPs, though less than 60%

indicated they were familiar with other AEOPs. In fact, 80% of alumni who responded to

the mid to long term evaluation questionnaire indicated interest in future participation.

2017 Summative Evaluation Report | PART 1 Executive Summary | 12 |

“[GEMS] kids are able to learn principles and apply them immediately, as opposed to the typical classroom

setting where they would learn something but not get the direct application. They’re able to conceptualize

how it would happen in the real world as well as what skill sets they could use in a career” --GEMS Mentor

“I am very impressed by every single hard-working researcher in our group. Each of them contributed to my

learning experience and were always willing to address any questions I had. I am especially thankful for my

mentor, who worked closely with every day and helped me discover all the theories behind our optical

pressure sensor research project. He ensured that I always had the opportunity to involve myself in cutting-

edge research and allowed me to make the most of my HSAP experience. I was exposed to areas of the

engineering world, such as resonances, semiconductor fabrication process, and so much more, that I never

knew existed. I am endlessly grateful for AEOP for this eye-opening opportunity, and I am confident that this

experience will lead me to unimaginable paths and direct my future for the better.” --HSAP Apprentice

“I am very happy with the three excellent HSAP students worked in my lab not only for the work they have

done (two manuscripts in preparation with them as co-authors), but also their passions and hard works in

learning. I could see the changes in them within this short 8 weeks of lab experience and all three will pursue

STEM in college (Physics, Engineering and pre-med).” --HSAP Mentor

“I realized how much I love my research project and understanding how the universe [works], so I've decided

to go into theoretical physics instead of computer science. I changed my mind after being inspired by the

Nobel Laureate and speaking with JSHS alumni on Saturday morning.” --N-JSHS Student

“JSHS is a wonderful competition and symposium. It is worthwhile for all students that participate; regardless

if chosen to present or move on to national level.” --JSHS Mentor

“I am very pleased with my JSS experience. It taught me so many things and made me confident in

complicated scenarios. All of the people I met and my mentors helped me out so much. I loved my experience

and had a great time.” --JSS National Student

“For [students in JSS] to have an opportunity to go and do something – fail at it, do well at it, or whatever –

and then be pushed the next year to learn from those experiences, that is what life is all about.” --JSS Mentor

“I am extremely satisfied with [REAP]. I have learned a lot in very little time, and the program has definitely

increased my interest in STEM! I also appreciate and respect the inclusion of minority groups in STEM. As a

Hispanic woman, I know my demographic is vastly underrepresented in STEM careers, and it is very important

that we change that. I very much enjoy being a part of this program and definitely will apply again next year.

Thank you so much for allowing me to participate; REAP has definitely changed my life for the better!” --REAP

Apprentice

“The [REAP] students were all brilliant, and we grew as a group over the summer. The students furthered my

interest in research by helping me understand problems from different points of view.” --REAP Mentor

“My [SEAP] mentor was very helpful throughout the entire process and worked with me through every step.

He helped me understand science concepts relating to my project and taught me various lab skills. Overall, the

program was very beneficial and has allowed me to expand my knowledge in the areas relating to the STEM

field.” --SEAP Apprentice

2017 Summative Evaluation Report | PART 1 Executive Summary | 13 |

“The SEAP program was simple and provided a great opportunity for the student to learn more about

Engineering and research prior to beginning a degree program in Mechanical Engineering. As a PhD

researcher in engineering, I wish I had been afforded a similar opportunity. The program is a great way for

the Army and ERDC to market ourselves to the community, gain summer help from eager students, and

continue to build a pool of recruits for the future.” --SEAP Mentor

“I loved [Unite]. It was the perfect mix of learning and fun. Also, it was amazing to meet so many people also

interested in similar topics. Over all it has helped me narrow down my career choices as it showed me what

interests and skills I have in various forms of engineering.” --Unite Student

“This year my Unite experience was excellent! The curriculum that was designed strongly supported the goals

of the program and there was clear growth and development in the participants from the beginning of the

program to the end of the program. We incorporated several enrichment components that allowed students

to explore their creativity, build teamwork and effective communication skills, improve their math and

problem-solving abilities, and gain valuable knowledge about STEM careers.” --Unite Mentor

“The [URAP] apprenticeship program was one of the most inspiring and challenging experiences I have had as

an engineering student. It will help me to grow as a Mechanical Engineer and also to be a scientist. I learned

that we should not be afraid to be wrong during experiments. Not only is knowledge important, but patience,

imagination, and creativity are important, too. I would love to be in a program like this again.” --URAP

Apprentice

“[URAP] is very beneficial for undergraduate students interesting in exploring STEM research. This also gives

them a chance to think and work independently, as well as in collaboration with other researchers, thereby

preparing them for a future career in STEM research.” --URAP Mentor

Recommendations for FY18 Program Improvement/Growth

While the successes for AEOP detailed above are commendable, there are some areas that remain with

potential for growth and/or improvement. The evaluation team therefore offers the following

recommendations for FY18 and beyond.

AEOP Priority: Broaden, deepen, and diversify the pool of STEM talent in support of our Defense

Industry Base

Increase and broaden participation in selected AEOP programs. In FY17, AEOP experienced a 6% growth

in participation, increasing to nearly 33,000 participants. The positive momentum should be catalyzed

moving in to FY18. It is recommended that additional resources and efforts be expended in regards to

eCM and JSHS specifically. Both of these programs produced growth in participation this year. eCM has

the flexibility within the e-model to grow participation relatively easily. JSHS has large (~3,000 applications

in Cvent alone in FY17) unmet need that could be transformed into additional participation if

infrastructure is in place to accommodate more participants. This is likely just a percentage of the actual

number of applications, as only 72% of the overall total JSHS regions used Cvent for registration in FY17.

Cvent captured only 2,435 of the 5,577 students reported. It is suggested that these programs examine

2017 Summative Evaluation Report | PART 1 Executive Summary | 14 |

strategies that programs such as Unite and JSS have used to produce growth in FY17 (over 20%). AEOPs

should continue to work to grow the percentage and number of underserved students who are

participating in the program. Unite, REAP and JSS can serve as potential models for the consortium of how

to achieve this in a more rapid and impactful manner.

Examine means for increasing infrastructure to grow placement rates in JSHS and apprenticeship

programs. An important first step in examining strategies to increase enrollment for AEOP overall may be

to take a look at the current unmet need and demand for programs within the portfolio that may not have

the infrastructure (personnel and resources) to be able to accommodate additional participants. Any

potential resources that may be redirected in the way of these programs, or from other potential future

proceeds, could be used to translate into increased participation in FY18.

Strengthen programs ability to impact STEM outcomes and awareness of DoD/AEOP. In FY17, most

programs had significant impacts on STEM outcomes and awareness for participants. Two of the AEOPs

that did not produce as large of gains as the others were JSS and eCM NJ&EE. It is recommended that the

AEOP examine the format, delivery, and feedback from these programs in partnership with the other

partners to determine how there could be improvement in this area in FY18 and beyond.

Continue to examine impact of AEOP participation on growth in skills beyond self-reports. The FY17

pilot of the 21

Century Skills Assessment has provided a powerful glimpse into the significant impact that

AEOP participation is having on extended-time programs (more than one-week in duration) including

some of the apprenticeship programs and Unite. In FY18, the AEOP should continue to build upon this

pilot to consider ways to implement a similar measure with other programs that have more sporadic or

intermittent (not multiple-day) frequency. True independently assessed growth (not self-reported by the

participant) in skills provides a more accurate measure of impact on skills and knowledge than self-

reporting. However, self-reports (questionnaires) also serve as a data point that reveals participants

attitudinal and interest data as well as their perceptions of growth in skills and knowledge.

AEOP Priority: Support and empower educators with unique Army research and technology

resources

Continue to focus on strengthening role of adults in mentoring and instruction. In FY17, there was

considerable improvement in the support of mentor use of effective mentoring strategies within and

across AEOPs, in part due to increased focus and investment of AEOP to the findings of the FY16

evaluation. However, there is still room for more growth, as reported use of effective strategies still

remains less than 50% for some programs and strategy areas. This is an area that should continue to be a

key component of the continued efforts to provide more resources, onboarding, and potential mentoring

(of mentors) to continue to make even greater impacts on student desired outcomes of the AEOP.

Grow adult and youth participant awareness through support and innovative programming from

AEOPs. An area of concern that was prevalent across the majority of AEOPs in FY17 was the persistence

of lack of information availability and/or utilization regarding AEOPs, as well as Army/DoD opportunities

and information by mentors and/or adults leading programs. While we are cognizant of the increased

2017 Summative Evaluation Report | PART 1 Executive Summary | 15 |

focus this area has received over the past couple of years, it appears that mentors need more support

and/or resources/programming/speakers/etc. to provide to students in multiple modes so that

participants become more deeply aware of AEOPs, and Army/DoD opportunities. It is recommended that

the AEOP examine best-in-class practices such as what eCM NJ&EE uses and consider scaling-up effective

strategies across the consortium.

AEOP Priority: Develop and implement a cohesive, coordinated, and sustainable STEM education

outreach infrastructure across the Army

Expand reach of AEOP marketing, recruitment strategies beyond current local networks. Over the past

three years of the AEOP evaluation, it has become increasingly clear that the portfolio has a vibrant, grass-

roots network that has served the AEOP very well in the past and currently. It is important to note how

difficult it often is to get deeply rooted within local contexts and communities to be able to produce such

powerful networking and recruitment efforts year after year. However, it is the time to look to go beyond

these networks to reach out to tap underserved populations that have little to no awareness of the

outstanding opportunities that await them within the AEOP. This is not to say that the current AEOP

network be disregarded – it should continue to be nurtured and leveraged. It is recommended that the

consortium work to develop, at a minimum, a targeted plan for outreach and participation for FY18. The

consortium began this work with the integration of Strategic Outreach Partners in FY16 and continued in

FY17. To date 15 organizations have received funding to work with the AEOP to broaden participation

through outreach to underserved communities. Additionally, in FY17 eCM awarded mini-grants to 183

team advisors to support participation in the program. Nearly 100 of those awards were to teachers at

Title One schools. Potentially some of the strategic partners should be filling this role to help expand the

reach of AEOP.