RD10 - Item 307B: Evaluation of the Virginia Department of Health Teen Pregnancy Prevention Initiative


Executive Summary:
Introduction

Growing concern about teen pregnancy and parenting in the United States is due to two primary factors: the sheer number of teens becoming pregnant and having babies and the concomitant adverse economic, health, and social consequences. Despite a 22 percent decline in the rate of teenagers giving birth in the U.S. since 1991, the U.S. teen birth rate is still one of the highest among developed nations.

Phase I of the Virginia Department of Health (VDH) Teen Pregnancy Prevention Initiative (TPPI) began with the appropriation of $600,000 in general funds during the final legislative session of the 1993 General Assembly for the purpose of establishing three pilot teen pregnancy prevention programs in the health districts of Alexandria, Norfolk, and Richmond. These three health districts were selected, in part, because of their consistently higher than average or rising (in contrast to state teen pregnancy rates which after years of increases had begun to decline) teen pregnancy rates. At the start of FY94, each of these three health districts received $200,000 to develop its pilot program. In response to the General Assembly appropriation, the Maternal and Child Health (MCH) Council's Subcommittee on Teen Pregnancy Prevention formed an interagency advisory committee to establish general guidelines for the three pilot teen pregnancy prevention programs. The interagency advisory committee established seven guidelines adopted by the Subcommittee during the early part of FY94. One of those guidelines stipulated, "VDH, in consultation with the interagency advisory committee, would contract with an external program evaluator to conduct program evaluation. A portion of the appropriated funds should be set aside for this purpose."

Through an administrative agreement with the Department of Medical Assistance Services (DMAS), Virginia obtained matching federal dollars in addition to general fund dollars in FY95. This mechanism established an annual base budget of $1.4 million for the VDH TPPI. The General Assembly authorized the additional funds to go towards starting four additional pilot teen pregnancy prevention programs in the health districts of Crater, Eastern Shore, Roanoke, and Portsmouth. This marked the beginning of Phase II of the VDH TPPI. In conformance with the guidelines adopted by the Subcommittee and the charge to VDH to evaluate the programs in order to ensure that the prevention methodologies were successful and transferable to other health districts, evaluation was given priority status. A percentage of the base budget was withheld from the health districts to fund evaluation activities. At this point, however, a year had already elapsed without an evaluative structure in place for three of the seven pilot programs.

Methodology

The ultimate objective of the VDH TPPI is to develop effective and replicable community-based teen pregnancy prevention programs. Evaluation contributes to the attainment of this objective by:

1. Providing data for use in management planning and resource allocation,
2. Measuring performance at each stage of program development,
3. Measuring performance of programs using outcome measurement, and
4. Providing information and feedback to the VDH regarding effective teen pregnancy prevention strategies.

To these ends, VDH and Virginia Commonwealth University Survey and Evaluation Research Laboratory (SERL) staff developed a statewide system of evaluation in FY96. Since that time, there have been several adaptations made to the original plan in order to overcome a variety of obstacles.

There are presently three primary uniform data collection and reporting components, in addition to a local site evaluation component, within the statewide system of evaluation. All components of the statewide system of evaluation are interrelated and feed into the Logic Model for Program Evaluation (Figure 1). A logic model is a succinct series of statements that link together the problems the program is attempting to address, the methods used to address them, and the expected results. The VDH TPPT program evaluation logic model includes the following components:

1. Inputs are program resources such as money, staff time, volunteer time, facilities, supplies, and equipment and are predicted to have an effect on activities or outputs (the services and efforts a program provides to its target audience such as instruction, public awareness, and referrals).
2. Activities are what the program does with the inputs to fulfill its goals and objectives.
3. Outputs are the direct products of program activities such as the number of participants served.
4. Outcomes are client or agency level changes and are predicted to have an effect on impacts (population or system-level changes which include teen pregnancy rates).

Results

This is the seventh annual report to the Governor and the General Assembly on the evaluation results of the VDH TPPI. There are presently three primary uniform data collection and reporting components in addition to a local site evaluation component within the statewide system of evaluation. Proposed local site outcome evaluation activities for FY02 focused on assessing the impact of teen pregnancy prevention programming aimed at modifying knowledge, attitudes, and behaviors. Proposed activities were both formative and summative in nature and utilized qualitative and quantitative measures.

Phase I Programs

Of the Phase I Programs, the Alexandria Health District has experienced the greatest decrease in its teen pregnancy rates over the eight-year period from 1993 - 2001 (an average decrease of 3.6% per year as compared to the average statewide decrease per year of 3.3% over the same period). The Norfolk Health District has maintained an average per year teen pregnancy rate decrease below what has been experienced statewide over the same period (2.6%). The Richmond Health District has the lowest average reduction of its teen pregnancy rate (0.8% per year over the eight-year period) in relation to all seven VDH TPPI program sites. This is considerably less that what has been experienced statewide.

The Alexandria Teen Pregnancy Prevention Program. In 1984, Alexandria officials launched a unified effort to combat the city's continuing teen pregnancy problem. Over a period of five years, a city task force and a steering committee determined the extent and impact of the problem and identified available resources to address the problem. The efforts of these two strategic groups culminated in the establishment of an Adolescent Health Center (AHC) and the Interagency Consortium on Adolescent Pregnancy (ICAP). In 1991, ICAP, which consisted of 13 public and private human services agencies, developed a multi-faceted, multi-agency, comprehensive model for teen pregnancy prevention. This model is now identified as the Alexandria Teen Pregnancy Prevention Program (ATPPP). TCAP has increased its membership since 1991, and now draws members from 24 different agencies throughout the city. In 1999, the Mayor of Alexandria established the Blue Ribbon Task Force on Reducing Teen Pregnancy. To provide a more unified approach, the ICAP and the Mayor's Blue Ribbon Task Force on Reducing Teen Pregnancy were merged. The group, now known as Alexandria Campaign on Adolescent Pregnancy (ACAP), has developed, implemented, and is overseeing a citywide public awareness campaign to prevent adolescent pregnancies and promote positive youth development. ACAP has established a community goal to reduce the teen pregnancy rate by 25% by 2005. The ACAP serves as the coalition that determines program type and provides direction for the ATPPP.

The ATPPP is a multi-faceted program that incorporates elements of prevention, intervention, and education. In addition to conducting a variety of community education activities such as health fairs, presentations, and workshops, the program funded six intervention projects serving at-risk youth between the ages of nine and seventeen in FY02. Five of those projects shared the goal of reducing the incidence of teen pregnancy in school age youth in the City of Alexandria. The primary objective of those projects was to provide youth who were still in school, but at risk for pregnancy or causing a pregnancy, an opportunity to take an active role in envisioning a future for themselves. Strategies included an emphasis on continued education and the ability to self-determine when to become parents, and the development of an internal locus of control to help youth to either successfully abstain from sexual intercourse or use an effective method of birth control. The sixth intervention project had as its primary focus the provision of support and advocacy services to pregnant and parenting teens.

All of the ATPPP intervention projects were established in areas of the city identified by the Virginia Center for Health Statistics as having a high incidence of teen pregnancy and a disproportionately high number of low-income single parent households. Several other teen pregnancy prevention efforts have developed in Alexandria as a result of ATPPP activities. Alexandria Health Department and ACAP continue to provide youth development and pregnancy prevention services to Latino youth of the city. The "Alexandria Latino Youth Enrichment Program" (ALYEP) which is now funded by the Substance Abuse and Mental Health Services Administration (SAMHSA) continued to strive to accomplish the primary goals of reducing the incidence of adolescent pregnancy and substance use/abuse among Hispanic adolescents living in the Arlandria and West End neighborhoods (two of the city's largest Hispanic communities) and to assure access to, and enrollment in, primary health care tor all program participants. However, since the funding source for FY02 was SAMHSA, more emphasis was placed on reducing the incidence of substance use and abuse in the target population. ALYEP achieved these goals by implementing a comprehensive life skills building and experiential learning program to youth 9 - 17 years of age in the identified communities. Topics of discussion include self-esteem, values, goal development, decision-making, communication skills, cultural adaptation, and family and personal relationships. Alcohol and drug use/abuse prevention and sexuality education are integrated into the life skills building program. Program participants also receive assistance with their homework, perform volunteer activities, learn job skills, participate in cultural and recreational activities, and receive counseling about the need for, and how to access, primary health care (including reproductive health care). The program, in its fourth year of operation, is now an integral part of the community and it has the full support and cooperation of the Hispanic families, businesses, and community leaders. The ALYEP program which began in November 1998 with two part-time staff at one site serving 25 Hispanic youth, had two full-time and one part-time staff that served 110 youth at three sites during FY02.

The Norfolk Teen Pregnancy Prevention Program. The Norfolk Teen Pregnancy Prevention Program (NTPPP) is known to the community as Real Alternatives to Pregnancy (AP). The Norfolk Advisory Group for Teenage Pregnancy Prevention serves as the coalition that determines program type and provides direction for RAP. This body is comprised of representatives from local community and youth service agencies and organizations. The Advisory Group was formed in 1993 and presently has over fifty active members.

RAP is a comprehensive community-based program designed for pre-teens, teens, parents, and the Norfolk community at large. The mission of RAP is to bring together various community elements and public and private resources in order to provide programs that change knowledge, skills, attitudes, and behaviors in such a way as to reduce the incidence of teen pregnancy. RAP is the only VDH TPPI program whose strategy for reducing teen pregnancy rates is to create community/systems level change. An expanded Advisory Board and an annual community conference are two of the ways in which community/systems level change is effected. Additionally, interagency networking is integral to all components of RAP. As part of its commitment to interagency networking, RAP puts together an Adolescent Services Directory that lists all community agencies and organizations providing services to Norfolk teens and their families. In order to encourage and support existing community-based teen pregnancy prevention initiatives, RAP offers incentive grants to community agencies and organizations through a Request for Proposals (RFP) process. In addition to the network-enhancing activities and the general community education activities such as health fairs, presentations, and an annual conference, the RAP Program offered five categories of intervention projects during FY02.

The Richmond Teen Pregnancy Prevention Program. The Richmond Better Beginnings Coalition (RBBC) serves as the coalition that determines program type and provides direction for the Richmond Teen Pregnancy Prevention Program (RICTPPP). The RBBC has been active in the metropolitan area since 1990, with membership consistently comprised of a diverse group of providers and community representatives. In 1997 and 1998, the RBBC restructured, creating membership categories as a means of increasing commitment. Active membership consists now of approximately 40 representatives from many of the agencies and organizations involved in teen pregnancy prevention and positive youth development in the city. The RBBC has traditionally excelled in increasing community awareness and providing information regarding adolescent pregnancy and pregnancy prevention. In recent years, the organization has acted as catalyst, researching, introducing, and training providers in innovative prevention programming. The RBBC, in collaboration with the Richmond City Department of Public Health's (RCDPH) Healthy Start Initiative, was responsible for introducing Postponing Sexual Involvement (PSI) into Richmond City Schools by way of the Virginia League for Planned Parenthood. Additionally, the RBBC sponsored training in the Teen Outreach Program (TOP), the majority of which is now funded and implemented by the RICTPPP.

The RICTPPP was redesigned for FY02. With help from the VDH TPPI Coordinator, a more targeted and comprehensive teen pregnancy prevention initiative was developed. In addition to conducting a variety of community education activities such as workshops and the distribution of brochures at the request of community leaders, school personnel, and youth serving organizations, the RICTPPP, in its revised plan, funds three intervention projects.

Phase II Programs

Of the Phase II Programs, the Roanoke Health District has experienced the most significant decrease in its teen pregnancy rates (an average decrease of 6.8% per year over the seven-year period from 1994 - 2001 as compared to the average statewide decrease of 3.5% per year). In fact, the Roanoke Health District has experienced the most significant average decrease in teen pregnancy rates per year of all VDH TPPI programs. The Portsmouth Health District has also experienced a significant decrease, averaging a decline in its teen pregnancy rate of 4.1% per year over the seven-year period as compared to the statewide average decrease of 3.5% per year over the same period. The Eastern Shore Health District has experienced an average teen pregnancy rate reduction of 2.9% per year and the Crater Health District has experienced an average decrease of 1.5% per year over the seven year period, both below the average experienced statewide.

The Crater Teen Pregnancy Prevention Program. The Crater Teen Pregnancy Prevention Initiative (CTPPI) is unique among the VDH TPPI programs due to the diverse and expansive geographic area covered by the health district. Three of its political subdivisions (Emporia, Petersburg and Hopewell) have had some of the highest rates of teen pregnancy in the Commonwealth.

The CTPPI Executive Committee serves as the coalition that determines program type and provides direction for the CTPPI. It is comprised of representatives from the Tri-Cities Crisis Pregnancy Center and agencies and organizations that receive subcontracts through the CTPPI. The CTPPI Coordinator chairs the group. Staff from Resource Mothers, the Comprehensive Health Investment Project (CHIP), Healthy Start, and the Crater Health District attend coalition meetings. Not only does the CTPPI Executive Committee provide direction for CTPPI, but it also makes every effort to share human and material resources, thus enhancing the effectiveness of all. In order to serve the needs of the three targeted communities, much of the funding for the CTPPI is distributed directly to those communities through a Request for Proposals (RFP) process. In FY02 CTPPI funds went to two community-based coalitions and one intervention project.

The Eastern Shore Teen Pregnancy Prevention Program. Established during the latter part of 1994, the Eastern Shore Teen Pregnancy Prevention Program (ESTPPP) is known to the community as "Young Voices for Better Choices". The coalition that determines program type and provides direction for the ESTPPP is made up of the Community Policy and Management Team (CPMT), the District Advisory Board, and members of the general public. In addition to conducting a variety of community education activities such as workshops, conferences, and community awareness surveys, the ESTPPP puts out a Request for Proposals (RFP) each year to solicit intervention projects. Eight projects were funded by the ESTPPP during FY02. The ESTPPP also partnered with several local organizations and agencies in sponsoring and conducting community activities and events.

The Portsmouth Teen Pregnancy Prevention Program. The Portsmouth Better Beginnings Coalition (PBBC) determines program type and provides direction for the Portsmouth Teen Pregnancy Prevention Program (PTPPP). The PBBC was established in 1984 and consists of teens, parents, community and civic leaders, members from the faith community, and professionals from the public and private sector, united to address issues related to teen pregnancy prevention in the city of Portsmouth. The PBBC is no longer staffed, but programs/services are coordinated by Portsmouth Community Health Center's Case Management and Outreach Services.

Throughout the year, the PBBC, PTPPP, and individual project staff participate in a variety of community education opportunities. These include health fairs, small group meetings at a church, annual community events, civic league or club meetings, and school meetings. During these events, information about the PBBC, the individual projects, and the specific teen pregnancy statistics for Portsmouth is shared. Discussion groups have also been held to assist parents in their communication about sex and sexuality with their teenage children. In addition to the above-mentioned activities, the PTPPP and PBBC organize a host of community education activities centered on Let's Talk Month and Teen Pregnancy Prevention Month. Some examples of activities conducted during these two months include: essay and poster contests; the distribution of teen help cards in the public schools and during football games; group sessions in churches or other community sites to discuss parent-child communication, abstinence skills, public speaking skills, teen parenting issues, and male responsibility; and billboards with statements about teen pregnancy and parent-child communication. In addition to these community education activities, the PTPPP also provided funding for four intervention projects during FY02.

The Roanoke Teen Pregnancy Prevention Program. The Better Beginnings Coalition (BBC) of the Roanoke Valley determines program type and provides direction for the Roanoke Teen Pregnancy Prevention Program (ROATPPP). The ROATPPP has been an official subcommittee of the BBC since 1995. The BBC was formed from the 1984 merger of the Prevention Coalition and the Coalition for Strengthening Family Living. The BBC membership consists of representatives from area school systems, social service agencies, health care providers, volunteer organizations, and concerned individuals. The ROATPPP Coordinator provides staff support to the BBC. The BBC exists for three purposes. First, it acts as an information clearinghouse on teen services. Second, it functions as an advocacy group, identifying the needs of Roanoke Valley adolescents, and working to mobilize groups to bring about change. The BBC works with other youth-serving coalitions and agencies to develop a community-based prevention plan in concert with the other community stakeholders. Third, it provides the community with education about causes and ways of dealing with the problem of adolescent pregnancy.

The ROATPPP targets all adolescents in the community, but focuses its projects primarily on those who are at high risk for becoming pregnant or causing a pregnancy. In addition to general community education efforts, the ROATPPP strives to build upon existing community resources. Consequently, its funds are used to expand four existing community projects.

Conclusion

The VDH TPPI grapples continuously with maintaining a balance between local autonomy/ownership and good stewardship and accountability for state and federal funds. According to the guidelines set by the MCH Council's Subcommittee on Teen Pregnancy Prevention, each locality is to form a communitywide coalition that includes representation from public and private organizations. The coalition is to determine program type and direction. An assessment of two of the seven coalitions is starting to paint a picture for the VDH TPPI that these local coalitions have not been as active and directive as originally intended.

Each of the seven VDH TPPI programs varies considerably in both composition and organizational structure. As a result, there is uniqueness and local culture and flavor reflected in the variability of program strategies and approaches across the seven pilot program sites. They range from having a primary focus on public awareness and community education activities (Levels 1 and 2) in order to reach a broad audience to an emphasis on more intensive intervention projects (Levels 3 and 4) targeted at individuals assessed to be at high risk for becoming pregnant or causing a teen pregnancy. The latter approach serves fewer individuals but is generally thought to have more immediate impact.

It is clear from the program descriptions, process and outcome evaluation data, and community teen pregnancy rate data, that some VDH TPPI program sites have been better able to garner the community support needed to mobilize for action than others. These sites include Roanoke, Alexandria, and Portsmouth. It is also clear that some VDH TPPI program sites are having less-than-expected program effects (i.e., lower than average decreases in their teen pregnancy rates). These sites include Crater, Richmond, the Eastern Shore, and Norfolk. The presumed reasons for not meeting expectations vary across the sites. In the Crater Health District, the available funds are clearly insufficient to provide the type of targeted intensive intervention projects needed to serve the diverse and expansive geographic area covered by the health district. Without the existence of additional funds, the CTPPI may need to rethink its strategy and focus its efforts on one community at a time. For the Richmond Health District, combined poor management and inadequate planning has contributed to less-than-optimal program effects. Under new management, the RICTPPP has overhauled its program in the past year and it is anticipated that this change will begin to appear in the evaluation results in the future. For the Eastern Shore, the community-based guidance directing the dissemination of small grants to multiple community-based agencies results in each agency trying to do "too much with too little". The ESTPPP may need to rethink its strategy and move toward investing more funds into fewer replicable "48" level programs. Level "4B" programs are those that carry with them the expectation that participants will attend all or most of a series of five or more intervention/education sessions and use an established and replicable curriculum. Norfolk has chosen to take a combination community/systems level change approach in addition to trying to target at-risk individuals within the community. Research has shown that community/systems level change approaches (such as through public awareness and education efforts) are effective. However, they require persistence over an extended period. Hence, the true outcomes and impacts of Norfolk's efforts may not be known for many more years. Additionally, since Norfolk is attempting to "do it all", it is possible that the program suffers because it too is trying to do too much with too little. For example, in comparison to the more successful VDH TPPI programs, Norfolk's intensive interventions are less intensive in terms of both average number of sessions attended by participants and the duration of those sessions. Norfolk may also want to re-think its strategy of trying to create change at both the community/systems level and at the at-risk individual level.

Future directions for FY03, which include some major restructuring, are outlined. In light of the restructuring effort and the evaluation results, only one recommendation for the VDH TPPI is provided for the upcoming fiscal year.