Return on Investment: Education, Rankings, and Data

What are the drawbacks of ranking colleges based on economic return, as Money Magazine is doing?

I was asked to answer the question above on the website Quora.com

**********************************************************************************
Before addressing your question it’s important, I think, to put some things in context. Money’s a newcomer to the college rankings game. The US News is the Google they’d like to compete with, but that is not likely to happen anytime soon. Still, how do they get readers’ fingers to press the link to their website? 

This is what the managers and marketers at Money asked the data folks. And if they didn’t, then they weren’t doing their job. Money might have compiled yet another ranking system in which the Ivies and Stanford came in at the top, but that would not help them brand themselves and it would not get them views and buzz, tweets and shares, comments and up votes (or down votes-- there seems to be no better time to be alive to understand the no bad publicly cliché claims market share for insight into the human condition for most people places and things).  Same old same old stays cold. So they had to go about gathering data and a methodology that would make a splash.

I would be shocked if the data guys (and they are mostly guys right now), did not do research on what stories trended best about higher education over the last several years. I have been keeping up with trends after a fashion and here are some of the hottest topics:

Student debt: the amount owed by students for attending colleges and universities now exceeds the debt outstanding during the real estate boom. It’s well over a trillion dollars.

The use of big data to track outcomes in business and now education. The "Big Data" meme shouts out that with all the information floating around schools, families and students could learn far more about education than they did previously. Many schools have implemented big data to help them make budget enrollment, academic and others decisions.

Budget cuts: many colleges and universities, even those with huge endowments, have had to cut back after the recession and the stagnation of salaries for the middle class. In addition, the schools have suffered from federal cuts back to research. Schools have, essentially, priced themselves out of the market for the middle class. The huge spending sprees schools went on when money was flowing in have now become problematic, although schools continue to build and expand in ways that don’t seem to make a lot of economic sense. Some colleges and universities have started to wake up from a decade long binge with more than just a hangover. They have way bigger problems and many are going to have to change the way they do things to survive.  They need a 12-step program of their own but most have not yet admitted they have a problem and that is a problem indeed. They may need an intervention but my guess is that many will keep doing what they do until they run out of cash or students or both
Full paying students have become a rare species and parents and students are concerned about paying so much without adequate information about what an education means in terms of future earning potential. For the best summary of these issues, read the book College (Un)Bound by Jeffrey Selingo. There are more data points gathered here than any other place I have seen.) 
  
International students: a number of schools have had to look abroad to get full payers. There are now some schools that enroll more than 1000 new students a year from China. Virtually all of them are not eligible for any financial aid and most tend to focus on majors and areas of study that relate to business, engineering, and the sciences.

The rise of new jobs. Many of the top jobs available for recent grads did not exist a decade ago. And many of these jobs look for certain skills such a coding, that are not often a part of any required college curriculum. 

The values of the liberal arts: the number of student majoring in liberal arts fields has dropped significantly. The option for jobs for students with liberal arts majors who have not done very well academically has caused many to question the pragmatic usefulness of such a degree. 

MOOCs and on-line education. The ready availability of on-line classes taught by star profs at top schools means that many schools might fold in the near future if credit starts being awarded. Should students pay 60,000 a year for classes that may be similar for what they can find on-line for free?

I am sure I have left out a number of other stories that came to the top of searches of what people want to hear and read about in higher education, but I would be schlock if the ones above were not looked at by the people coming up with ways to create an strategy for designing a to create a survey that “speaks” to these issues.

My point in bringing all this up? Virtually each of the above issues has some connection to return on investment. Money, therefore, is responding to market forces.  They have understood that the way people now evaluate colleges and universities has changed since US News first designed a way to rank.

Obviously some see the ROI as a poor way to rate the value of an education in terms of how it changes the way people think, live, and choose to work. I think many of these critics raise the issue but I am not sure that aside from decrying it, I have seen little from them that suggests alternatives. At a time when cost, debt, and academic preparedness are all in the minds of most parents and many students, I think this ranking could be useful in a couple of ways. At the same time, it could also lead to schools making changes that would hurt some students and the quality of the education offered to students.

Nate Silver, in his wonderful book, The Signal and the Noise, points out that all data is to some degree “out of context”. There is an infinite amount of data out there and what the data gurus summon by their questions will shape things they think and interpret and ultimately shape the conclusions we reach. As a result, a lot of what gets published as conclusive is anything but. If I would apply his Bayesian approach to Money I would certainly some crucial things have been left out. Still, it’s also important to acknowledge that the rankings they have created are not simply based on ROI alone. Here is how they describe their approach:

"To find out which of the nation’s roughly 1,500 four-year colleges offer the most bang for your tuition buck, MONEY screened out those with a below-average graduation rate and then ranked the 665 that remained on 18 factors in three categories: educational quality, affordability, and alumni earnings provided by Salary Comparison, Salary Survey, Search Wages. (Enrollment, acceptance rate, and H.S. GPA data is for the 2012-2013 academic year, the latest that was available from the Department of Education.) Included: a “value added” grade that rated each college in light of the economic and academic profile of its student body and the mix of majors at that school. We then used a statistical technique to turn all the data points into a single score on a five-point scale and rated them accordingly." 

I have not examined in any detail  how this  methodology works and even if I did I’d have to get a Nate Silver or someone who knows data to say whether what Money used might prove useful for those reading the ratings.  The same, however, could be said for the US News methodology. Malcolm Gladwell took it apart in a New Yorker piece a while ago, but this had little affect in hurting their brand. (Slavoj Zizek likes to quote Lacan in situations like this “Je sais bien….mais quand meme…” --I know very well….but even still... For example, I know the US News rankings are anything but scientific, but I will will attend the school I get into because it has the highest US News ranking even if I liked the feeling of  the campus of a lower ranked school.)

Many of the ways US News rates colleges should be questioned. So too with  Money.  But in both cases they do measure something and it may well be that given the current interest in ROI that Money may be useful for parents who can’t afford, literally, to say to their children” “Go forth and follow your bliss”. One of the sad facts that do come from lots of research is that over 40% of students graduating from college have not improved their critical thinking skills. That means students are going off to school and learning many things but thinking may not be one of them. I don’t know, barring exist exams, how ths could be measured and most school would never introduce such exams. It would hurt their brands. 

For the purposes of a thought experiment, let’s say Money starts to climb the ranking of the rankers. Should this happen then some schools might put more money into career services, a typically understaffed and under-funded part of a college. I think this would be a great trend. Of if schools would not want to do this then they could outsource career services to professionals like modernguild.com. In either case, the students would benefit.

 On the other hand, if Money’s stock did rise, then schools might also encourage student away from low pay professions like teaching or ngos or many other fields where the ay is never great. This would be an unintended consequence but I would predict it would happen. Why? With the US News, the number of applications a school receives and the numbers the students present (rank I class and SAT) have changed admission dramatically. Each year schools try to generate many more applications and drive down their acceptance rates. In order t so they have in a few cases, made it much easier to apply. They waive application fees, send document prefilled, and in the case of some selective schools drop essay requirements and the umber of recommendations they will read. To put it simply numbers rather than the ‘fit’ that every admission person talks about to students tends to get rewarded. And now that colleges are cash poor the ability to pay full fees often affects decisions too.  Most of these effects were not at all in the thinking of the designers who came up with the US  News rankings ad my guess is that most at Money have not thought how their success might, to some degree, come at the expense (pun intended) of the students.

If nothing else, however, Babson’s top ranking speaks volumes about the current state of education. It is a great school. It’s not known nearly well enough in the US but globally they get top kids to enroll from around the world. They also have an exceptional program in entrepreneurship. That word buzzes around the Internet at light speed on untold thousands of sites these days. It may well be the most popular meme when it comes to savvy kids these days, more than liberal arts or investment banking or premed. Babson, unsurprisingly, has the ranking displayed prominently on its webpage. They should. They "earned" it. Whether other schools will now try to imitate Babson and massage data to do so is an open question. There are a number of schools who have done this with the US News data. To keep up with the competition occasionally drives schools to make less than ethical decisions. 

At the moment, we seem to worship those who can take an idea, maybe while still in college, and end up billionaires in less than 10 years. It’s pretty hard to think of anything else that compares to this new way of making it big except for Hollywood and the music industry.  Babson represents the Zeitgeist and Money has found a way of recognizing this. It certainly does not tell a great deal about many of the things that make up a college education but it does represent a way of understanding the changes that have taken place in education over the last decade.


Perhaps in the next generation, the Zeitgeist will have shifted.  Michio Kaku has outlined in his newest book one direction we might be headed toward. Those schools that educate people to perform gene therapy to mothers to be to conceive a 7 foot tall geniuses with stunningly good looks and a life expectancy of 140 and a 200 IQ will be the ones most in demand and so those schools who can prepare the scientists to improve our brains will be, shall we say, the mot highly thought of in the world.

Michio Kaku