Market theory and Chicago Public high schools

Daniel Kay Hertz has a great post up at his website (that was tweeted by WBEZ and reproduced on Catalyst) regarding the market trends for Chicago Public Schools high schools. The major charts of the piece are reproduced at right: basically that when one looks at the district as a whole it appears at least at face value that students and families are choosing to move from schools with lower overall ACT scores to ones with higher ACT scores.

However, when Hertz breaks the data down between charter and non-charter CPS schools things get murkier. Students in general are choosing "better" non-charter schools as opposed to "worse" non-charter schools. But, within the subset selection of charter schools students and families are statistically choosing randomly.

There are a few issues with the way the analysis was done (as I understand it).

1. Including selective enrollment schools in the pool of data will skew results to make it look like there is not much self selection for students to move to the top schools in CPS. This is, of course, patently false. If a parent were able to just choose that their student go to Northside (with a 29) or Whitney Young (with a 27), they would. That is, of course, impossible though because each has an admissions rate of about 12%.
    
2. Using ACT growth over three years would be a much better measure. Comparing the overall end scores on the ACT doesn't as much measure the quality of education received at the school (for a moment ignoring the side debate of test-as-imperfect-indicator-of-learning) as it does measure where the students end up. This sounds a bit tautological, and it is, but take an example: if a student enters Roberto Clemente (neighborhood CPS HS) at a 12 and ends at an 18 - that is a fantastic amount of growth and speaks to the skills acquired during the first three years of HS. On the other hand, if a student enters ASPIRA at a 17 and improves to an 18 over three years, at least by the measure of the test, not as much has been learned.
   
   So, it would be interesting to do a similar analysis on Hertz' dataset sorting by three year cohort growth from freshman EXPLORE to junior or senior year ACT scores. If the market approach is working well, parents and students would presumably move to the neighborhood and charter schools which provide the most growth.
    
3. High school is (with few exceptions) geographically constraining. Some of the explanation for an increase in "lower performing" schools could be the fact that the nearest reasonable alternatives were also "lower performing" schools. In other words the perceived cost-benefit trade off of an hour long commute each way for marginally better (say, 1-3 points) school might seem unattractive. Which would then result in increased enrollments with less attractive schools (test score wise) but more attractive geographically.*
    
4. The ending ACT data has a long tail and isn't always at the forefront of a parent's decision making process, especially for newly started schools. With 48 new high schools in the last 10 years many parents are choosing to send their students to programs without records or results until three years later. Doing some back of the envelope math here, 48 schools multiplied by 4 classes that will enter each of those buildings before ACT scores are published/publicized would result in 192 cohorts of students/families selecting a school based on imperfect or partial data. So, while it may appear that parents/students who chose charter school X or new neighborhood school X made an irrational choice, for 192 sets of those families judging that choice requires a level of hindsight bias. 

Overall Hertz brings up some interesting analysis though, and at a minimum this is a start to a conversation, and an important one. Assuming that some of the mitigating variables could be explained away or solved in the near future (better access to information for parents etc), if the trends identified above to persist it would be a serious blow to the entire philosophy of market-based portfolio management for large urban districts.

The underlying larger takeaway should be this - not all neighborhood public high schools are awful, not all charter schools are fantastic. Far from it on both sides, those who paint either groups with such a broad brush do a disservice to the most important constituency here - the parents and students who need to make a real, difficult and impactful decision of where to spend their high school years.

---

* Might be interesting to compare increases and decreases in enrollment with a variety of other factors to see if some have a higher correlation with increased enrollment, such as a the aspects of the 5-Essentials survey.

Correction: an earlier version of this post said that WBEZ had reproduced the Hertz piece, they had in fact only tweeted a link. Apologies for the confusion.