The Cranky Analyst

Upward Bound, Science and Advocacy

2012-01-19T07:52:00.002-05:00

The same advocacy group that successfully pushed through a congressional ban on any further randomized control trial (RCT) of Upward Bound now protests that the quality of the one existing RCT is unacceptable. Not coincidentally, this RCT found Upward Bound to be ineffective.

The first and only RCT of Upward Bound did find suggestive evidence that the program helped students who had the lowest college aspirations. The follow-up RCT, which was strangled in its cradle, would have investigated this finding further.

Disclaimer: I was on the Technical Working Group for the second Upward Bound RCT, which met exactly once before it was killed. I didn't know this at the time; when I was never called for a second meeting I thought maybe I had been expelled for talking too much.

NYT Gives a Shout-Out to My Research on Inequality

2011-11-24T13:39:00.001-05:00

The NYT editorial page cites my research with Martha Bailey on growing gaps in college completion. Unfortunately, they did not link to our paper properly - it's here.

Thoughtful Discussion of Our Research on Early Interventions and College Outcomes

2011-11-15T18:31:00.004-05:00

Tim Bartik, an economist at the UpJohn Institute here in Michigan, has a very thoughtful post about our new paper on the relationship between class size in kindergarten and postsecondary outcomes, which I blogged about earlier this fall. In that paper, we calculate that early interventions are no more effective (dollar-f0r-dollar) than later investments in increasing college attendance.

Bartik reports that our work has generated concern among some advocates of early childhood investments, who worry it undermines support for funding for early interventions. Bartik has a sensible response, which speaks to the complementarity of interventions at different ages. A recommended read.

Student Debt

2011-11-14T22:35:00.001-05:00

This short, well-written article puts rising college costs and student debt in perspective.

College Tuition, Student Loans, and Unemployment : The New Yorker

Using randomized trials to make management decisions

2011-11-11T14:16:00.001-05:00

In this piece, Columbia professor Ray Fisman describes how a business uses a randomized trial to determine whether a new personnel policy (allowing employees to work from home) increases productivity and employee satisfaction. Same approach could be taken in a public or non-profit setting to test myriad practices.

Is working from home a good idea? - Slate Magazine

Learning What Works in Teaching

2011-11-03T09:26:00.001-04:00

This is a fabulous article. A successful school leader lays out a roadmap to how schools and academic researchers can combine to use rigorous research methods to learn what works in teaching. A must-read for researchers, school leaders, teachers and those who train teachers.

Studying Teacher Moves : Education Next

Association for Education Finance and Policy, March 15-17 2012, Boston

2011-10-28T13:35:00.001-04:00

Please submit a paper, session, or poster! AEFP members and prospective members are invited to submit proposals for the AEFP 37th Annual Conference on March 15-17 2012. Proposals may address any of a broad range of topics in education finance and policy. Open for proposals: 10/1/2011 Close: 11/8/2011 Submission portal: http://www.aefpweb.org/proposals

Early Investments and Adult Outcomes

2011-10-24T17:02:00.004-04:00

Along with Josh Hyman (a doctoral student at University of Michigan) and Diane Schanzenbach (economist at Northwestern) I have a new paper on the effect of class size in elementary school on adult educational outcomes (NBER link here; ungated link here). The Economist just blogged about it, which makes my economist heart go pitter-patter!

"This paper examines the effect of early childhood investments on college enrollment and degree completion. We use the random assignment in the Project STAR experiment to estimate the effect of smaller classes in primary school on college entry, college choice, and degree completion. We improve on existing work in this area with unusually detailed data on college enrollment spells and the previously unexplored outcome of college degree completion. We ﬁnd that assignment to a small class increases the probability of attending college by 2.7 percentage points, with effects more than twice as large among blacks. Among those with the lowest ex ante probability of attending college, the effect is 11 percentage points. Smaller classes increase the likelihood of earning a college degree by 1.6 percentage points and shift students towards high-earning ﬁelds such as STEM (science, technology, engineering and medicine), business and economics. We conﬁrm the standard ﬁnding that test score effects fade out by middle school, but show that test score effects at the time of the experiment are an excellent predictor of long-term improvements in postsecondary outcomes. We compare the costs and impacts of this intervention with other tools for increasing postsecondary attainment, such as Head Start and ﬁnancial aid, and conclude that early investments are no more cost effective than later investments in boosting adult educational attainment."

Jan 2012 Update: The BLS Monthly Labor Review also has a nice discussion of the paper.

Standard error, please.

2011-10-08T15:43:00.001-04:00

File Under Silver Linings

2011-10-01T11:26:00.001-04:00

In the recent recession, men's earnings have fallen much more than women's. Since 2007, median male earnings have dropped 6% and median female earnings have dropped 0.9%. As a result, the ratio is at an all-time high: women now make 83¢ for every man's dollar.

NYT: Recession Struck Inadvertent Blow for Women's Equality

Statistical Superheroes!

2011-10-01T11:12:00.001-04:00

Very cool use of statistics. Survivors of a sunken battle ship give disparate, 100-mile wide reports of where it sank, and there follow 70 years of unsuccessful searches. Statisticians plot the distribution of reported locations, use theory to determine its central tendency and recommend a small search area. Wreck found. Nerds to the rescue!

Freakonomics: Nazis, Sunken Ships and a 67-year-old Game of Telephone

Low-Hanging Fruit for the Cranky Analyst: Gross Misinterpretation of Statistics

2011-09-28T17:56:00.004-04:00

The National Center for Education Statistics just put out a report on the employment experiences of new teachers. StudentsFirst grabbed the report and wrote a blog about it. They should have slowed down. I suggest that this fine organization hire some UM Ford School grads to help them with their statistical analyses.

The StudentsFirst blog says of the NCES report:

"Teachers making more than $40,000 were 3.7 times more likely to stay in the profession than those making less."

The NCES report says:

"Approximately 93 percent of beginning public school teachers who were earning less than $40,000 in 2008–09 remained teachers in 2009–10, and about 96 percent of beginning public school teachers who were earning $40,000 or more in 2008–09 remained teachers in 2009–10 (table 3). "

Hmmm, is 96% the product of 3.7 and 93%? Let me get my calculator.....(tap, tap, tap). Nope! That would imply that 355% of better-paid teachers stay on the job. There IS about a 3 percentage point difference in the share of low vs. high-earning teachers staying in teaching: 7% of those earning <$40K leave, compared to 4% of those earning >$40K. That's worth writing about!

Good news: a failing charter school is shut down

2010-04-17T23:52:00.001-04:00

The key idea behind charter schools is that good ones should be expanded and bad ones shut down. This is working in East Palo Alto, where the school board has voted to shut down a charter that is the worst-performing school in the district. The school is run by the Stanford Ed School and was founded by Linda Darling-Hammond, a prominent teacher educator. One of the highest-performing schools in the district is also a charter school, run by Aspire. Ironically, the Stanford team partnered with Aspire when it initially launched its charter. The partnership collapsed, reportedly because Aspire was too focused on academics for Stanford. The schools went their own ways, with vastly different results. http://www.nytimes.com/2010/04/16/education/16sfcharter.html

Weil & the "Evidence of Experience."

2010-03-13T14:38:00.018-05:00

Reading Denialism: How Irrational Thinking Hinders Scientific Progress, Harms the Planet, and Threatens Our Lives, by Michael Specter. Fun and enlightening read, I recommend it. I don't agree with everything he says, but it got me thinking, and I certainly value that in a book. His chapter on alternative medicine moved me from thinking to blogging. Specter quotes health guru Andrew Weil on randomized trials:

There is a great movement toward "evidence-based medicine" today, an attempt to weed out ideas and practices not supported by the kind of evidence that doctors like best: results of randomized trials. This way of thinking discounts the evidence of experience.

Weil misses the point here quite quite badly. A randomized trial aggregates the experiences of the dozens or hundreds or thousands of people who participated in the experiment. By valuing the findings of a randomized trial more than the intuition generated by our own narrow worldview, we are paying attention to the experiences of others. Counterquote from Dara O'Briain, Irish standup comic:

Just because science doesn't know everything doesn't mean you can fill up the gaps with whatever fairy tale most appeals to you.

You can see Dara's routine here.

Charter Schools Close the Achievement Gap

2010-01-11T13:35:00.029-05:00

I have been happily working with a killer set of co-authors recently: Josh Angrist, Parag Pathak and Chris Walters of MIT, Tom Kane of Harvard and Atila Abdulkadiroglu of Duke. We are writing a series of papers that evaluate the effect of charter schools on test scores. We are focused on Massachusetts, where we have an excellent working relationship with its very capable office of education research, which has allowed us access to the statewide test score data needed to undertake an evaluation of this nature.

The key empirical challenge in understanding the effect of charter schools is selection bias: kids who go to charter schools are different in both observable and unobservable ways from kids who don't. You can try to control for the visible differences, as does this study from Stanford. But what we can measure in terms of student background is pretty scant in administrative datasets: sex, race, free/reduced lunch eligibility, special-education status, previous test scores. So there remains plenty of risk for omitted-variable bias. Are kids whose parents are highly educated or motivated concentrated at charters? Kids whose test scores were plummeting in the public schools? Kids who were not challenged in the public schools? All of these differences would contaminate any effort to compare the achievement of kids at charters and kids at public schools.

We solve this problem by exploiting the randomized lotteries conducted by over-subscribed charter schools. The lottery approach focuses on students who apply to charters, comparing outcomes for those who lose the lottery to those who win. A mere coin flip (or randomly-generated number) separates the lottery winners and losers, so we can be confident that they are alike in every observable and unobservable way - except for their charter school attendance. This closely approximates the gold standard of a randomized, controlled trial.

We used this approach to study charters schools in Boston, and have just written a paper focused on a KIPP charter in Lynn, a working-class city north of Boston.* A policy report and two academic papers (on Boston and KIPP Lynn) have been born so far of this collaboration. All our results point to large and positive effects of charter schools on student achievement. The effects are larger for kids who most need help: Blacks, Hispanics, those with limited English proficiency, special ed kids, those who have the lowest baseline scores. The effect sizes are huge - kids at charters gain 0.1-0.2 standard deviations each year on their peers at the traditional public schools. How "big" is this? Wicked big, as they say in my hometown of Somerville, a few miles south of Lynn. In Lynn, the Hispanic-White test score gap is 0.5 standard deviations, as is the Black-White test score gap. Interventions that come anywhere close to gains of that size are few and far between. By way of comparison, the STAR experiment found that smaller classes (sustained for four years, from kindergarten through third grade) increased test scores about 0.2 standard deviations.

We are not the first to use the lottery approach; Caroline Hoxby, Jonah Rockoff and co-authors have looked at charter schools in Chicago and New York City using this methodology, and Roland Fryer has used this approach to examine charters in the Harlem Children's Zone. But the KIPP Lynn paper provides the first lottery evidence on a charter school that serves a large Hispanic population (the school is about half Hispanic): the New York City, Chicago and Boston charters are predominantly Black, and enroll fewer Hispanics that the surrounding schools. And it is the first randomized evaluation of a KIPP charter, which are spreading across the nation. The KIPP results are convincing evidence that charters can succeed with students who have the greatest need. A recent UFT report raised the concern that charters are failing (or ignoring) these very populations. At least in Lynn, KIPP is achieving astounding results with these kids.

..................................................................................

*That's the Lynn in the chant sung to generations of children being bounced on their parents' laps: "Ride a horse to Boston, ride a horse to Lynn, better watch out or you're gonna fall IN!"

This is Transparency? This is Data-Driven Education Policy?

2009-10-02T10:11:00.002-04:00

A letter to parents was in my kids’ backpacks this week. It’s about whether teachers at their school and in their classrooms meet the No Child Left Behind Act’s standards for a qualified teacher. Being a data nerd, I was eager to see the statistics for our school, and to compare them to those for schools in the rest of the district, state and nation.

Did I get the statistics that would let me see whether my kids are taught by qualified teachers? No. Here is the letter:

“The Ann Arbor Public Schools is proud of its faculty and staff. Approximately three quarters of our teaching staff have a masters degree or higher. The District operates an extensive Professional Development program and ALL new four-year probationary teachers receive a full three years of mentoring from an experienced Ann Arbor teacher.

The parents of a student in the Ann Arbor Public Schools, under the No Child Left Behind Act of 2001 (NCLB), have the right to know the professional qualifications of their child’s teachers. Under NCLB you are entitled to know the following information about your child’s classroom teachers:

Whether the State of Michigan has certified the teacher for the grades/subjects he or she teaches;
Whether the teacher is teaching under an "emergency permit" or other "provisional status" by which the state certification criteria have been waived;
The teacher's college major and advanced degrees, if any, by subject;
Whether any instructional aides or similar paraprofessionals provide services to your child and, if so, the paraprofessional's qualification.

If you would like to receive this information, please submit written requests to Administrative and Human Resource Services, 2555 South State Street, Ann Arbor, Michigan, 48104, ATTN: NCLB Inquiries. A minimum of ten (10) business days will be required to prepare the response

Thank you very much for your cooperation.”

Written request? Ten business days? This is no way to keep parents informed about schools, the purpose of the NCLB requirement. The process reminds me of a Freedom of Information Act request. As a parent, I would feel like I was bothering/attacking/impugning the district were I to make such a request (and I am no shrinking violet). I am guessing that very few parents ask for these data, which means that this aspect of NCLB is not working.

Why not put information about teacher qualifications on the web, where parents can read it, or at least in the letter itself? I can think of two possible explanations.

1) They have the information readily available. But, they do not want parents to have this information. Maybe because parents might protest the level or distribution of qualified teachers in the district? That explanation does not put the school district in a flattering light.

2) They do not have the information readily available. It is not in machine-readable format, or is lodged in some clunky database that only one person can figure out how to use, so generating school- and classroom-specific reports on teacher qualifications is time-consuming and difficult. This explanation is equally disturbing to the one above, because it means that the district is not using information about teacher qualifications as it deploys teachers across schools, makes hiring decisions, and examines patterns in student achievement! Yikes!

Sure is interesting to see how the intent of Congress actually plays out in school districts.

Progress on Reforming Federal Aid

2009-09-10T21:22:00.003-04:00

While the nation has been storming and heckling and angst-ing over health insurance reform, there has been a productive conversation and quiet, steady progress toward reforming the federal student aid system.

The direction of reform takes is consistent with the recommendations made by the College Board’s Rethinking Student Aid task force, of which I am a member. My work with Judy Scott-Clayton (newly minted Harvard PhD, now an assistant professor at Columbia) has also influenced the conversation.

Developments that make this policy wonk’s heart twitter with happiness:

The IRS has agreed to pass tax data of aid applicants to the Department of Education (ED), thereby allowing for the automatic pre-filling of the dreaded Free Application for Student Aid (FAFSA) with tax return data already collected by the feds. Getting IRS to play nice with ED was a major barrier to student aid reform. Kudos to all who nudged, cajoled and threatened the federal bureaucracy into cooperating with itself!

House and Senate committees are working on bills that remove from the aid formula (almost!) all data items that don’t already reside in the IRS data files. Should they (as I recommend) get rid of all the non-tax data items, this allows for an application-free aid system. Anything short of that and we are stuck with an application, albeit a shortened one.

ED has substantially improved the interface and skip logic of the Web FAFSA. This makes life simpler for those students who fill out the form online. The paper FAFSA is still pretty horrifying, however, and that is the form that matters for those who do not have access to broadband at home (hint: poor people).

This week, Joe Biden chaired a White House Middle Class Task Force meeting at Syracuse University on barriers to college entry and completion. The Task Force also released a report that describes the Obama administration’s goals for restructuring student aid, while the Council of Economic Advisers and National Economic Council released a report on the same topic.

So exciting to see our research and analysis shaping the path of policy in DC. Simplification has been a topic in which conservatives and liberals have found common ground. Knock on wood that this cooperation continues.

Two Perspectives on Harvard in Sunday NYT

2009-09-07T11:00:00.011-04:00

In the New York Times Sunday Book Review, Harvard President Drew Faust has an essay (“The University’s Crisis of Purpose “) on the social mission of the university. She questions whether colleges have tilted too far toward moneymaking and materialism:

As the world indulged in a bubble of false prosperity and excessive materialism, should universities — in their research, teaching and writing — have made greater efforts to expose the patterns of risk and denial? ….Have universities become too captive to the immediate and worldly purposes they serve? Has the market model become the fundamental and defining identity of higher education?

In the same issue of the Times, we learn that Harvard has lent its name to a line of pricey preppy clothing. In the Harvard Yard menswear line, Oxford shirts start at $165, jackets at $495. Message: Harvard can be bought by men who can afford it.

As is frequently true, those who make financial decisions at Harvard are working at cross-purposes with the institution’s academic leadership. The University spends a lot of time and money trying to make itself welcoming to low-income kids. Image is important, and the image projected by this line of clothing is rich, preppy and male. The few bucks ($500K/year for all licensed clothing, including this line) Harvard makes on this licensing deal are far outweighed by the damage it does to its reputation.

Helicopter Parenting. For Real.

2009-08-31T21:00:00.012-04:00

I am guessing this dad was an unhappy adolescent, and is determined to buy his own kid a happy time in high school:

According to a Lake County Sheriff's Office report, Bart Sutherin flew his 14-year-old son to his first day of classes at East Ridge High School in a rented blue-and-white Hughes 300 helicopter this week...According to the Orlando Sentinel, Sutherin told officials that he wanted to ''make a positive impression on the other students'' on his son's first day of 9th grade.

We are not inspired

2009-08-28T15:51:00.013-04:00

The American Association of School Administrators -- the professional organization of top officials who run our schools -- has posted its comments on the Obama administration's Race to the Top competition. Here is the top, very first, #1 aspect of the RTTT that ASSA comments on:

"p. 3 We encourage you to submit comments in advance of the date by which they must be received.”

Pretty innocent request: Get your memos in early if you can. I passed right over it, moved on to the meat of the guidelines. Not the eagle-eyed ASSA, which led its memo with this insightful, reasoned, child-centered, professional response to this outrageous, ham-handed request:

AASA Response:

"The regulations use thirty days as the response period. Why raise the possibility that only early submissions will be read by urging early submission? Why not shorten the comment period? If later submissions are not read or are not being taken into consideration, then the administration will forfeit the future possibility of being believed when claiming that it is actively listening to public comments."

Wowsers. Way to hook your audience. The remainder of the text makes for equally depressing reading. It is somewhat more on topic. But not any more inspiring.

Finally, some common sense on student loan debt

2009-08-19T22:58:00.004-04:00

The College Board has put out a new and very sensible report on student loan debt levels. Given the hyperbolic media coverage of this topic, the bottom line might surprise you: debt loads are not terribly high. The median is $19,000 for those completing a BA.

Does $19,000 sound like a lot of money? Not given the value of a college education. And not given how much we spend on items of far less enduring value. The typical new car loan is for $29,000. That car is valueless after a decade, while human capital pays off over a lifetime. Further, the monthly payment for that car loan would be over $500, while for the student loan it would be about $200.

Mrs. Krabappel is not yet out of a job

2009-08-19T21:33:00.003-04:00

This report on online education is being framed by the media as showing that online education beats face-to-face instruction in the classroom. So should we replace Mrs. Krabappel with a bevy of netbooks and a wireless router?

Not so fast. The research is far too weak to draw the conclusion that teachers can be replaced with online instruction.To their credit, the authors admit this up front:

"The most unexpected finding was that an extensive initial search of the published literature from 1996 through 2006 found no experimental or controlled quasi-experimental studies that both compared the learning effectiveness of online and face-to-face instruction for K–12 students and provided sufficient data for inclusion in a meta-analysis."

What were the limitations in the existing research that led the authors to this gloomy conclusion (which did not come across in the press reports)?

I) Internal validity

Out of 1000+ studies the authors reviewed, 33 were randomized trials, and 13 were comparison-control with decent controls. The rest, ewww. The RCTs did show pretty big positive effects (0.2 SD), however, and we know that research is not democratic. So, what's the problem?

..which leads us to...

II) External Validity

a) Just one of the 33 randomized trials (and four of the 13 comparison-control studies) took place in a K-12 school. The rest were in colleges or training programs for medical professionals.

b) None of the (five) K-12 studies compared face-to-face instruction with online learning, which is the comparison the we all have in mind when we read the media reports. Rather, the studies compared 1) face-to-face instruction with 2) face-to-face-instruction PLUS online learning. No teachers were taken out of the equation for the treatment or control group.

The bottom line (which did not come across in the press reports...) is that this research tells us nothing about whether online learning and face-to-face instruction in K-12 are substitutes in the learning process. It does provide us some evidence, however, that they are complements.