The following article originally appeared on Youth Today.
Throughout the year, First Lady Laura Bush has traveled the country touting programs that the White House says do everything from steering kids away from drugs and violence to increasing reading skills and building character. The visits are part of Mrs. Bush’s Helping America’s Youth initiative, which seeks to promote programs that have been demonstrated to be effective.
But as the First Lady prepares to convene a national Helping America’s Youth summit in Washington next month, a look at some of the programs she has visited shows that, by and large, they are based on promising ideas, but have little scientific evidence of effectiveness.
The adviser who helped find model programs for the First Lady describes facing the same challenge that confronts the operators, advocates, and funders of youth development programs: “When you look out there, the number of programs that meet strong standards [for evaluations] are just not there,” says G. Reid Lyon, a well-known education researcher.
Mrs. Bush has said in speeches and interviews that the programs she’s visiting are “very effective,” “successful,” and “have some track record.” But while some of the programs have been examined by independent evaluators, most of their evidence consists of self-evaluations, incomplete preliminary data, anecdotes, or a belief that certain activities help kids – the kinds of evidence that youth development advocates have been citing for decades, but are now often told is not good enough.
There is no doubt that the programs do wonderful things for youth and that the First Lady’s campaign has brought valuable attention to youth work. But her visits illustrate a continuing struggle for the youth work field at a time when government and foundation funders are demanding better evaluations.
Many if not most of the programs spotlighted by Mrs. Bush might not meet the evaluation standards increasingly demanded by her husband’s administration, which has cited a lack of evidence of effectiveness in proposing to eliminate or trim youth programs. They almost certainly don’t meet the standards for scientific evidence called for two years ago in a report by the White House Task Force on Disadvantaged Youth.
Yet for anyone looking for insight into what kinds of youth programs the White House prefers, the programs that Mrs. Bush chose to visit are a good place to start.
The First Lady’s Agenda
Much of the news media have described the First Lady’s initiative as “anti-gang,” but it is not. The label comes from the president’s State of the Union speech in February, when he proposed “a three-year initiative to help organizations keep young people out of gangs, and show young men an ideal of manhood that respects women and rejects violence. Taking on gang life will be one part of a broader outreach to at-risk youth.”
The communications-savvy Bush administration might have figured that describing the initiative as anti-gang would make more of an impact on Congress, the media and the public than pitching, say, a “youth development” effort. Congress is considering several pieces of tough anti-gang legislation.
However, Helping America’s Youth is actually a broad umbrella for youth development programs involving responsible fatherhood, healthy marriages, reading skills, character education, mentoring children of prisoners, and teen sexual abstinence, as well as gang prevention.
Although those are mostly non-school issues, the initiative’s roots lie in the First Lady’s interest in education, especially literacy. During her travels around the country last year, Lyon says, Mrs. Bush, a former teacher and librarian, grew more concerned that “even with the best reading program … a lot of the kids were having difficulties focusing on academics, for a wide variety of reasons,” such as living in poverty or having dysfunctional families. “When they get to school, they’re just not ready for learning.”
Mrs. Bush wanted “to take a look at programs that developed good academic skills through better behavior,” he says.
She was especially concerned about at-risk boys. Mrs. Bush has said that concern was inspired by a New York Times Magazine article last year about Ken Thigpen, a troubled young man in Milwaukee who turned his life around and became a responsible father.
Lyon says Mrs. Bush asked him to find programs that helped youth both academically and behaviorally. The idea was that “we would visit sites across the country that … combined good reading instruction – evidence-based reading instruction – with good behavioral support programs that had data with them.”
Lyon, a sometimes controversial advocate of scientifically based reading instruction methods, was then a top official at the National Institute of Child Health and Human Development and had advised the administration before. He is now overseeing the launch of a worldwide chain of for-profit teachers’ colleges for a Texas company, Best Associates.
Officials in federal agencies such as the Justice, Labor, and Health and Human Services departments sent suggestions for programs to highlight, Lyon says. But he ran into the reality that claims about the effectiveness of many youth development programs “are based on anecdotes, philosophical beliefs and untested assumptions.” Lyon saw that few programs aimed at changing youth behavior had been scientifically tested, and almost none could meet the “gold standard” for evaluations, which would include random samples and control groups tracked over a significant period.
Mrs. Bush “was very aware that the evidence was not the same as in other fields where we had been working,” such as education, Lyon says.
It is a continuing dilemma: Except for programs focused primarily on academics, youth-serving programs have not traditionally been geared toward showing the kinds of outcomes that can be measured, say, over the course of a school year or a summer break.
The Final Report of the White House Task Force on Disadvantaged Youth, released in late 2003, said there was little scientific evidence that most federally funded youth programs work. It said programs should be measured by more strict scientific standards or lose federal funding.
“We need to firmly hold programs accountable for results showing that they actually achieve what they were designed to achieve,” the report said.
The Bush administration cited poor evaluations when it proposed cutting the 21st Century Community Learning Centers program by 40 percent in 2004 (which failed). It cited a lack of proven effectiveness to justify cutting TRIO (an educational outreach program for disadvantaged youth) by more than half in its 2006 budget proposal and to justify eliminating such programs as Even Start and GEAR UP (college prep) in the Education Department, and Juvenile Mentoring, Safe Schools and the Juvenile Accountability Block Grant in the Justice Department.
There are, however, scientific studies that demonstrate the effectiveness of certain elements of youth work, like providing close relationships with caring adults (such as mentors) over a long period of time. These findings were analyzed in a 2001 study by the National Academies, Community Programs to Promote Youth Development.
The First Lady’s office chose to highlight programs that “were based upon behavior principles and social development principles and family interaction principles that were known to be effective,” Lyon says.
In talking about her initiative with the PBS NewsHour early this year, Mrs. Bush said she is “visiting these programs around the country that are already successful, that we know they’re successful, that they have some track record.” Her role on each visit, she said, is “to highlight a very effective program.”