Celebrity egghead Malcolm Gladwell generated controversy when he opined that “anyone who gives a single dollar to Princeton” has “lost their mind.” Princeton has by far the highest endowment to student ratio in the country, and these ratios and endowments ought to be looked at skeptically.

America’s elite institutions have played an increasingly important role in the financial, geographical and cultural segregation of Americans. It is a self-perpetuating enterprise: those from the Top 1% disproportionately attend these schools, get high-paying jobs in expensive and isolated urban centers, and then give back large amounts to these schools in return. It is an arrangement that serves the interests of many parties, but disserves others.

Ivy League universities have endowments valued in excess of $119 billion, on which they pay no taxes. Estimates are that the tax exemptions on Ivy investment returns cost federal taxpayers in 2014 about $3.4 billion. While these schools grab a good deal of the surrounding land, and thus drive up property values, they do not pay property taxes. In the case of Harvard, for example, this saves them $40 million a year in taxes, which they might offset by paying off local residents at a fraction of the cost. Princeton recently paid out an $18 million settlement to the city rather than fight a challenge to their property-tax exemption, even though they were confident they could win. The reason? “Preserving diversity” in the local community, one where few of the graduates will live. Anyone who has been to New Haven knows well the sharp juxtaposing of Yale’s opulence to the city’s poverty.

There are other ways in which these endowments skew the public weal. Most of the endowments are managed by hedge fund managers, who get paid handsomely for their efforts, often under a “2 and 20” remuneration system that allows the managers to pay lower tax rates. Nationwide, more than 20% of college endowments – roughly a $100 billion - are invested in hedge funds. Meanwhile, Harvard’s investment management office spent about $70 million a year to have an underperforming investment portfolio. On the other end, hedge-fund managers return the favor by giving large (tax-exempt) donations to these universities, most famously John Paulson’s $400 million gift to Harvard.

Giving to educational institutions remains the second largest philanthropic enterprise behind religious institutions, and the gifts increasingly create a system of haves-and-have-nots in the American college system that mirrors larger societal shifts. Smaller colleges (such as my own) scramble for donations and increasingly compete with one another for students in a shrinking demographic pool. These schools, along with community colleges, are the institutions that best service the places where they are located, rather than the “elite” schools that brag about how they draw their students from all over and send them all over, rootless fragments of population that don’t know whom to serve because they don’t have a place to serve.

A report by Jorge Klor de Alva and Mark Schneider highlights the ways in which “the majority of taxpayers are poorly served by the tax-exempt status of large college endowments.” I might add the same is true for most communities as well. The average Princeton student receives about $105,000 a year in Federal, State and Local appropriations and tax subsidies, compared to the $12,300 at Rutgers, $500 at Rider, or $2400 at the local community college. Given that Princeton draws a disproportionate number of its students from the 1% and its graduates are also disproportionately represented among the 1%, these tax policies are causes of economic and social immobility, which one would expect when 17% of all endowment money serves <1% of the college population.

We are at the front end of the attrition that will be caused by this unequal distribution of resources. Colleges are already closing at higher rates, and the rates will increase in the near future as both the number of students and the financial resources shrink. Small communities such as Holland, MI, which depends on Hope College for graduates who can serve the community, for cultural life, and for economic vitality, will find themselves permanently and perhaps irreversibly undone by these closures.

Harvard, it is said, is a hedge fund with a university attached. Hedge funds are largely about leveraging capital, and our economy is about as leveraged as it can get. We operate on borrowed and liquidated wealth (which is to say, not wealth at all), which increasingly flows away from the poorer places in the country and towards the richer ones. In a time where governments are desperately scrambling for funds that attempt to achieve demonstrable public goods, multi-billion dollar university endowments should be on the table.