Originally published in Student Lawyer, March 2003 (Vol. 31, No. 7)

The RANKING GAME

U.S. News and World Report’s annual law school rankings get plenty of attention.
But do they really matter?

by Jane Easter Bahls

The day it all starts over is just around the corner. On April 7, when U.S. News and World Report issues its Best Graduate Schools 2003 issue, law students, deans, faculty, and alums across the country will eagerly turn pages or scan the U.S. News web site to see how their school wound up in the rankings, which compare schools by reputation, selectivity, resources, and job placement success.

Most will be disappointed, if not angry. After all, only five law schools can be in the top five. Still, many schools will make the best of it, issuing press releases to tout their rank as one of the top 10 law schools for tax law or trial advocacy or dispute resolution.

Why? Because even those who despise them admit that the U.S. News rankings make their mark, however flawed and limiting they may be. According to many involved with legal education, April 7 may be the real April Fool’s Day for those who place too much stock in the newsmagazine’s best-selling issue.

“People are using rankings as a substitute for all or part of their own education about the school,” says Barry Currier, deputy consultant on legal education for the American Bar Association, a longtime critic of the U.S. News compilation.

“The rankings are becoming a self-fulfilling prophecy,” Currier complains. “When people sort themselves by ranking, the good get better.”

Since 1990, when U.S. News started ranking law schools and other graduate schools, legal educators have been crying foul. Critics assail the methodology used and the very idea of ranking all 185 ABA-approved schools on a single scale.

“No rating of law schools beyond the simple statement of their accreditation status is attempted or advocated by the official organizations in legal education,” the ABA states in its Standards for Approval of Law Schools. “Qualities that make one kind of school good for one student may not be as important to another…. Prospective law students should consider a variety of factors in making their choice among schools.”

U.S. News has tinkered year by year with the methodology, but it has refused to stop ranking law schools. As the rankings have an increasing effect on the market, some law school administrators have taken steps to improve their standing—not always in ways that help the school or its students. (See “Gaming the Rankings,” page 18.) But because U.S. News rankings rely heavily on a school’s longstanding reputation, it’s nearly impossible for schools in the bottom tiers to get much higher.

For students feeling glum about their school’s dismal showing, it helps to keep it all in perspective.

Just ask Jen Gannon, a second-year student at Seattle University School of Law. She was devastated last year when her school slipped from the third to the fourth tier (only the top 50 schools are assigned a numerical rank), and she wrote about it in the school’s student newspaper. Gannon says her fight-or-flight instinct kicked in—starting with the impulse to transfer out: “The ranking by U.S. News may not be statistically perfect, and it may be the source of controversy in some circles, but it does matter to people. It matters to future law students, future law professors, and, I daresay, future employers.”

After further reflection, Gannon opted to fight. “I decided that I wouldn’t roll over like a submissive puppy and take this ranking that U.S. News placed on my head,” she wrote. “I’m not a fourth-tier girl, and I’ll be damned if I consider this a fourth-tier school.”

Gannon’s article urged her classmates to get the word out to their contacts in law firms, telling them how good the school is. “When I graduate in 2004, I want to hold my head up high and know that I made the right choice in attending Seattle U.,” she wrote.

Unlike Seattle University, American University’s law school advanced in the rankings last year, making it into the top 50. But it wasn’t entirely a cause for the administration to celebrate.

“On the whole, I believe rankings are not helpful,” says dean Claudio Grossman. “They do not contemplate all the criteria that are important.” Grossman ticks off a list of his school’s strengths that aren’t considered factors in the U.S. News rankings: diversity (although there is now a separate diversity index), academic excellence, service, experiential learning, and percentage of resources devoted to pro bono legal work. And yet, he says, many applicants use these rankings as the only factor in choosing a school.

Each year, nearly all deans of the 185 ABA-approved law schools sign a letter sent to everyone who takes the LSAT, warning potential applicants not to put too much stock in commercial rankings. “These ranking systems are inherently flawed because none of them can take your special needs and circumstances into account when comparing law schools,” this year’s letter explains. “The idea that all law schools can be measured by the same yardstick ignores the qualities that make you and the law schools unique, and is unworthy of being an important influence on the choice you are about to make.”

But signing that letter doesn’t stop law deans from making the most of whatever recognition they get. Temple University ranked in the second tier last year but issued a press release touting its No. 1 ranking in trial advocacy, one of eight specialty rankings, for the fourth straight year. DePaul University ranked in tier three but noted that its health law program was listed among the country’s top 10. And Hamline University ranked in tier four but announced that its alternative dispute resolution program was again ranked fifth—in a press release titled “Hamline University School of Law Repeats Prestigious National Ranking from U.S. News & World Report.” The deans of all three schools had signed the letter denouncing law school rankings.

Last year, at least one school refused to cooperate with U.S. News. Texas Southern University’s Thurgood Marshall Law School ignored the magazine’s annual survey. “The Thurgood Marshall Law School backs up its bark with a bite,” then-dean John Brittain declared to Texas Lawyer. “It disagrees with the rankings and, therefore, refuses to participate.” Brittain explained that the school’s special mission to minority students leads to admitting many who aren’t considered top tier in the traditional sense—a factor sure to keep the school in the magazine’s fourth tier.

“Many schools are serving a particular mission and doing it well,” dean John Attanasio of Southern Methodist University School of Law told Texas Lawyer in reaction to Texas Southern’s boycott. “It’s a shame to denigrate schools when they’re performing a very valuable role in society.”

U.S. News isn’t the first entity to rank law schools, although it’s certainly the best known. Starting in 1977, retired California State University professor Jack Gourman self-published a ranking of law schools but was widely criticized for favoring large research universities and for refusing to reveal his methodology. That didn’t stop Princeton Review from publishing The Gourman Report in 1997, but it might be why it’s now out of print. Princeton Review also has ranked law schools on student satisfaction.

Others have ranked law schools on such factors as the number of law review citations and number of articles published by the faculty. Thomas M. Cooley Law School in Lansing, Mich., ranks the “educational effectiveness” of law schools by measuring the “value added by the institution.” This involves comparing the reported entering class profile of each school to its bar results. The more improvement, the better. Accordingly, schools that admit the lowest LSAT scores and undergraduate GPAs and relentlessly teach to the bar would score highest. By this measure, Campbell Law School in North Carolina ranked No. 1 in the nation last year.

The methodology of U.S. News, meticulously revealed in the magazine and web site, considers a broad range of factors. Selectivity—a combination of median LSAT, median GPA, and percentage of applicants accepted—accounts for 25 percent. Placement success accounts for 20 percent. This includes employment rates at graduation and nine months after, plus bar passage rate. Fifteen percent comes from faculty resources, an umbrella category that includes expenditures per student, student/teacher ratio, and total number of volumes and titles in the library.

By far the largest factor, accounting for 40 percent of the total, is the reputation ranking. U.S. News sends a survey to the dean and three faculty members at each school, asking them to rate every other law school on a scale of one to five. The same survey goes to lawyers and judges. The problem critics have long pointed out is that very few people know enough about all 185 ABA-approved law schools to be able to judge their quality.

Indeed, chances are that many of those filling in the reputation survey haven’t even heard of some of the schools on the list, but they rate them anyway. On the other hand, schools with recognizable names—such as those with great football teams—are more likely to get high rankings for their law schools. John Sexton, dean emeritus of New York University School of Law, quipped to the New York Times a few years back that if those ranking law schools were asked about Princeton Law School, they’d put it in the top 20—even though Princeton has no law school.

The age of a law school makes a huge difference, says professor Michael Herz of Yeshiva University’s Cardozo School of Law. Of all the schools accredited in the past 25 years, not one is listed in the U.S. News top 50. Go back 30 years, and Brigham Young University, accredited in 1974, makes the cut at No. 37. The vast majority of law schools accredited within the past 30 years are clustered in the third and fourth tiers. Does that mean these schools are inferior? No, but a national reputation takes a long time to develop.

Then there are the specialty rankings. U.S. News asks a small set of professors in each specialty field to nominate 15 schools with the best programs. Schools with the most nominations get listed in the top 10.

“We made the intellectual property list for the first time a few years ago not because of a sudden increase in quality, but because we got the word out about our program with a generally distributed brochure,” Herz says. “The following year, a few other schools sent around brochures and snuck onto the list and we fell out of the top 10. The year after that, we sent around an even better brochure, and were back on. It’s pretty silly.”

Further, Herz says, if a school has a decent, well-known program in a particular field—maybe 13th or 14th on everyone’s list—it would come out as the best in the country just because more people have heard of it. “It’s easier to be visible than to be good,” he says. “Specialty rankings rate visibility rather than quality.”

But that’s the point, counters Mitchell Berger, a legal editor with Thompson Publishing Group in Washington, D.C., who defended the U.S. News rankings in an article for the Journal of Legal Education in December 2001. “When it comes to finding a job, the reputation is what counts,” he says. “Reputation is based on perception, not reality, and that is as true for institutions as it is for individuals.”

Berger agrees that the rankings may shortchange some excellent schools that are relatively unknown, and inflate the rankings of others. “But when all is said and done, the rankings roughly correspond to the way both practicing attorneys and law teachers perceive certain schools,” he says. “That perception, however unfair, will have a huge influence on a student’s job prospects.”

U.S. News has changed its methodology over the years in response to criticism, including a 1998 study by Stephen Klein, senior research scientist with the consulting firm of Gansk and Associates. The Association of American Law Schools, which commissioned the study, called for the magazine to stop ranking law schools. Robert Morse, the U.S. News director of data research at who created the ranking system in 1990, patiently lists the many ways he has altered the methodology: dropping the practice of putting data on particular factors in rank order and then adding up the ranks in each category; dropping a column on starting salaries of graduates; moving to the same peer assessment survey for academics and practitioners.

But Morse gets a bit testy when people suggest the magazine misleads students. “U.S. News has never said that this is the only way to rate law schools,” he says. “With the data we’re using, these are the results. But we’re not saying these are the only factors.” Indeed, the introductory articles in the Best Graduate Schools issues urge students to consider a wide range of factors they find important, and not to rely solely on the rankings.

“Like any of these rankings we do, I would think people would weigh these rankings with their own criteria,” says Sarah Sklaroff, the magazine’s education and culture editor. “Our web site has gotten more and more popular because you can search for schools by criteria.” With a premium online edition—accessible for $9.95—you can identify any factor and see the schools re-ranked, accordingly. And the articles don’t feature just the top-ranked schools. “We highlight schools that people might not have thought of,” Sklaroff says.

So who’s paying attention? Prospective law students, certainly. “I use rankings as the No. 1 source as to where I would like to go,” says Jessica Morales, a junior at Manhattan College. “I am not currently applying to law schools, but in a year and a half I will be, and I am already researching the rankings.”But not everyone takes the rankings as gospel. “I think we’d be lying if we tried to suggest that we didn’t consider the rankings to some degree or other,” says Chris Stetson, an M.A. candidate at California State University, Los Angeles, who has applied to law schools for admission for fall 2003. “All of us obviously want to attend the best law school possible, and we have to do the research that will provide us with a list of compatible schools.”

Most applicants, he figures, use rankings as at least a general guide. “This is not to suggest that rankings are an accurate or thorough guide in any sense,” Stetson continues in an e-mail message. “But they can be a very useful tool, and a way to explore schools we might not have otherwise considered.” He advocates using rankings as a starting point, then moving on to talk with students, professors, and lawyers. “It may very well be that a ‘lowly ranked’ school fits your interests best,” he says.The ABA’s Currier notes that most applicants choose their school based first on location, then price, then reputation. “It’s not true that students generally pack up and move across the country to the highest-ranked school,” he says.

What about employers? Texas Lawyer hit upon an alternative way to rank the state’s law schools by polling local hiring sources. In December, the magazine surveyed Texas’ top law firms, requesting rankings on factors such as the helpfulness of each school’s career services office and the long-term success of students they have hired. (See “How Employers Rank Law Schools,” below.)

U.S. News puts little emphasis on employers’ opinions when compiling its national law school rankings, and firms themselves have mixed feelings about the magazine’s efforts. Beth Kirch, director of career services at the University of Georgia Law School, says she finds employers taking note of what U.S. News tier a school ranks in, but not being concerned beyond that. “They know that schools fluctuate within their tier,” she says, noting that over the years, Georgia has fluctuated between ranks 26 and 34. “I’ve never seen an employer say, ‘You dropped, so you must be decreasing in quality.’” On the other hand, she’s heard employers say that they interview only at top-tier schools.

“I think employers pay some attention, but the distinctions they’re making don’t matter a lot to employers,” says Cardozo’s Herz. Indeed, he observes, many law firms are moving away from on-campus interviews because of the huge investment of time required. Instead, they consider whatever résumés come over the transom. While it’s true that the very top national law firms recruit only from the very top law schools, he says, regional employers tend to look at regional schools and local employers at local schools. Employers in a school’s own city will know the reputation of the school from experience, not from a magazine ranking.

Leslie Colvin, director of professional recruiting at Gray Cary, a technology law firm in Palo Alto, Calif., says her firm doesn’t put too much stock in rankings. “While we may look at the results of some of these surveys, we select the law schools from which we recruit based on experience and past results.”

That may be some comfort to law students at schools that can’t seem to move out of the bottom tiers, no matter how good they are. But Herz advises students to keep it all in perspective. “Where you went to law school matters the day you graduate, and it matters less and less every day thereafter,” he says. That first job opportunity may come because of your law school’s name, but over the years you’ll build your own reputation.

Currier and others at the ABA note that ABA approval is a hallmark of quality. The differences between ABA-approved schools are less important than whether or not a school has gone through the accreditation process successfully. Currier explains this by way of analogy: “If you’re in the bottom quarter of the Olympic team, you’re still on the Olympic team.”

“I think the rankings are first of all entertainment,” Herz says. “They’re interesting, amusing, and contain little information. They need to be taken with a grain, maybe a truckload, of salt.”

A Way to Change Your School’s Rank

Are you unhappy with your school’s rank on the U.S. News system? Remember, how a school stacks up depends on which criteria are used and how much weight is assigned to each.

To see how this works—and maybe give your school a better rank—play The Ranking Game at www.law.indiana.edu. Created by professor Jeff Stake at Indiana University School of Law-Bloomington, the game explains a wide variety of criteria that could be used to rank a school, from standard ones like LSAT scores of entering students to such factors as low tuition, campus beauty, and Tibetan restaurants within 400 meters.

To play, you choose the criteria that matter to you, then assign a weight to each. Is student/faculty ratio just as important to you as the number of seats in the library, or twice as important? Once you’ve set it up, the program will sort the 185 ABA-approved law schools by those criteria.

“It is obvious that any ranking depends on the criteria used and the weights applied to those criteria,” Stake says. “But it is not always obvious how much a small change in weight will influence the results.”

“It’s a game, so have fun,” he continues. “But do not let the apparent objectivity of the data fool you into thinking that the rankings based on those data are objective. Ranking games played with real data are still just games.”

—Jane Bahls

Gaming the Rankings

While most law school deans diligently sign the annual ABA letter to prospective students admonishing them not to take rankings too seriously, most also consider rankings important enough to figure out how to improve their standing.

Indeed, U.S. News takes some credit for nudging schools toward excellence. “We’ve been a key factor in making schools pay attention to their placement,” says Robert Morse, the U.S. News data guru who designed the rankings system. “For a while, we were the only ones doing school-by-school placement.” Wanting to be able to report a high percentage of students employed, many schools have been tracking graduates’ employment and helping those who need it.

But not everything a school might do to improve rankings helps students. Every year, just before it’s time to rate law schools’ reputations, law deans find their mailboxes stuffed with expensive, full-color brochures from other law schools bent on higher visibility.

For some schools, improving rankings becomes an all-out campaign. Dean David Burcham of Loyola Law School in Los Angeles felt he owed his constituents an explanation in 2001 when the school dropped from the second to the third tier. He sent out a memo explaining that the school’s numbers improved in most categories, and that U.S. News had reduced the size of the second tier, raising the cutoff from 97 to 88.

“Loyola’s primary job has been and always will be to give our students an outstanding legal education,” Burcham wrote. “But we also owe it to our students and alumni to do everything in our power to improve our ranking.” The dean outlined a five-prong strategy: conducting a national public relations campaign, improving delivery of career planning, developing networking opportunities, reducing the size of the entering class to improve admissions numbers and student/faculty ratio, and reviewing curriculum and teaching methods to ensure that students are prepared to pass the bar exam. Last year, Loyola was back in the second tier.

Some law schools go even further, rejecting applicants whose high LSAT scores indicate that they’ll probably go elsewhere anyway, in order to appear more selective. Or, to improve the amount of money spent per student, they increase tuition and return it to students as financial aid. (Low tuition is not one of the factors U.S. News considers.)

In effect, these schools pursue rankings, hoping that excellence will follow. But if they pursue excellence, perhaps eventually the rankings will catch up.

—Jane Bahls

Reining in the Rankings

After years of listening to legal educators grouse about law school rankings, the organization that administers the LSAT is considering a counterattack. According to a 1998 study, the only two factors that really make a difference in the ranking of schools under the U.S. News methodology are perceived reputation and LSAT scores. A working group at the Law School Admission Council is discussing the concept of no longer releasing LSAT scores—even to law schools.

Instead, when a law school requests a student’s score, the council would say only how the student’s score compares to those of other students at the school. For instance, it might say a given student is in the 62nd percentile of applicants, or the 54th percentile of students accepted by that school last year. This would make it impossible to compare schools on the basis of LSAT. Indeed, law school admissions committees would have to find some other way of comparing students.

“It would be revolutionary,” says Robert Morse, director of data research at U.S. News. Morse questions how schools would be able to compare applicants, because an undergraduate GPA of 3.75 from Yale might mean something quite different from the same GPA at a less rigorous school. “It’s also anti-consumer. How can you test prep when you don’t know how they’re going to score it?”

Ed Haggerty, spokesman for the Law School Admission Council, responds that one criterion for any concept its working groups propose is that it would give law school applicants more information than they already get, not less. So the idea would need developing. “It’s not our goal to have students operating in the dark.”

Morse notes that the reason U.S. News puts so much emphasis on LSAT is that law school admissions committees do. “We’re going to continue to rank law schools, and if the admissions standards change, we’ll adapt our methodology.”

—Jane Bahls

How Employers Rank Law Schools

Law students in Texas can get an idea of how their schools rank—in the eyes of hiring VIPs within earshot. In December, for the second year running, Texas Lawyer magazine polled hiring partners, employment directors, recruiting directors, and managing partners at the state’s 25 largest law firms, requesting numerical rankings of individual law schools based on the firms’ experience with the schools and their students. This year’s results were reported in the magazine’s Jan. 6 issue, ranking seven of the state’s nine law schools. (Two schools did not garner enough responses for inclusion.)

The survey touches on many factors U.S. News doesn’t address at all, having to do not with students’ entering credentials but the abilities of graduating students as lawyers and what the hiring process is like.

Texas Lawyer asked employers about how helpful the schools’
career services offices are, the firm’s long-term success with students hired from each law school, and the firm’s success in retaining students hired from each school.

The survey also asked firms to rate each school’s students’ skills, including communication and interpersonal skills, their strategic thinking skills, their analytical and problem-solving skills, and their knowledge and application of legal ethics.

“Rate the students’ abilities to hit the ground running on the first day as new attorneys,” one question asked. “Rate the students’ fit with your firm culture,” asked another. The survey even asked for ratings of students’ abilities to work as a team, their leadership potential, and their knowledge of the firm when interviewed.

Getting ranked dead last a year ago was a bitter pill to swallow, says Reginald Green, assistant dean for career resources at South Texas College of Law. “We more than other law schools take it very seriously,” Green says. “We wanted to find out what law firms wanted from us.”

After some inquiry, South Texas retooled its on-campus interview process to make things run more smoothly. The school also brought in lawyers to teach students what employers want to see.

The efforts paid off. This year, the school rose in the ranking to fifth.

—Jane Bahls