When in the course of human events a man does certain things, we question whether he was a douchebag. History is a slaughterhouse for sacred cows, and so scholars now work to determine if Thomas Jefferson popped his collar or, worse, had a slave pop it for him.

It’s been a rough couple of decades for Tom. All he was missing in John Adams was a top hat and bushy moustache; tack on Sally Hemings and it’s easy to forget there’s a mountain in the shape of his head. He brought it on himself, by leaving so much behind: tens of thousands of letters, homes, DNA. He’s the low-hanging fruit —but he’s a tangelo.

Jefferson had slaves, but do we blame the girls of My Super Sweet 16 for getting convertibles, or the system that handed them the keys? Automatic jerkdom for plantation owners writes off Washington, Madison, Monroe—we would be a nation built by jerks. Jefferson called slavery a great “stain” on the nation, but in the plantation South slaves were the ones who had to get the stains out. It wasn’t unusually awful to own them.

Sleeping with them would be terrible if he did it, but Jefferson was a devastated widower at 39. He made a deathbed promise to his wife never to remarry. He didn’t have options, aside from slaves or flirting with married women, but even then it’s not clear he knocked the powder out of anyone’s wig. File his love life under “lamentable, but human,” like VH1 on a Sunday night.

He practiced dirty politics. Jefferson sponsored whispering campaigns against Washington and Adams, paid journalists to smear opponents and operated from the shadows. But that’s politics, and contemporaries did the same thing to him. They didn’t have the Internet or 527 groups (Durham Boat Veterans for Truth!), so if you wanted to get ahead you got your hands dirty.

It’s not all savory, but it’s explainable. Visit Jefferson’s homes, however, and you still get the sense that something’s off. The trouble with Jefferson is that Monticello has no staircases.

There are stairs to the upper floors, sure. But they look like something from a Tim Burton movie: cramped, dark, steep and possibly leading to an egg sac containing Helena Bonham Carter, who would jump out and lick your face. If you want a regular staircase with a minimal chance of spine damage, you’re out of luck, because Jefferson the architect (who slept on the ground floor) did not like staircases. They cluttered up his design. He squeezed a dumbwaiter into a mantel, but convenient stairs? Nah.

His retreat near Lynchburg, Poplar Forest, is more intimate, just big enough for a few friends and no more than 40 slaves. It’s octagonal, even down to the outhouses—reflecting a love of either symmetry or the dark arts—and in the middle of restoration it’s still clear why some consider it Jefferson’s masterpiece, the culmination of a lifetime of design. The trouble is that it exists. Jefferson faced crippling debt (and an unfinished Monticello) when he started the house in 1806. With cash flow problems, you might balk at adding friezes and a portico to a house no one was supposed to visit. You might pass on a sunken back yard that took slaves two years to dig. But Tom, see, had this brilliant vision …

These aren’t symptoms of douchebaggery. There is none of the proud ignorance or spotlight-seeking that we expect from modern jerks. The trouble is the self-absorption and true belief that we expect from modern cable news hosts. Jefferson the designer had visions for architecture—and government, and proper living—but they were often tunnel visions: the good was not allowed to be the enemy of the perfect. Never mind if Dolley Madison fell on her ass trying to get to a guest room, or slaves had to shuffle dirt around a back yard, or John Adams had to be stabbed in the back 11 or 12 times. The visions were what mattered. For a guy who hated monarchy, he would have loved Burger King: Jefferson wanted things his way.

And he wasn’t above inconveniencing himself. Visitors to Monticello today see a beautiful home, but Tom lived in a perpetually rotting construction site. What would take the people of Flip This House three weeks without slave labor took TJ 54 years, and it was mostly his fault. He wanted a private, ornate house on a mountaintop, far from a river, or his mother, or basically any people he wasn’t married to or in direct ownership of, including anyone skilled enough to build that house. Thirty years of dumpsters in the driveway would be enough for most people, but Jefferson was a slave to his vision. It probably wasn’t so bad on nights when his vision of proper living involved drinking crates of French wine, but even then there’s all that embarrassing, crushing debt for a hangover.

It’s possible that Jefferson was a jerk, a man wrapped up in himself. But the bigger question: Does it really matter?

The University of Virginia was Jefferson’s proudest achievement. He dreamed of an academic village, staffed by the finest teachers in the world—a plantation to grow the next generation of Americans. Jefferson started the project late in life, supervising from his mountaintop with deteriorating health and little chance to revel in success; like so much of his life’s work, he saw it as a gift to the future.

Sitting on the steps of the Rotunda 200 years later, looking out across a green lawn flanked with porticos, thinking about the generations who enjoyed the same view, you feel … civilized. Even with the occasional 18-year-old, popped-collar d-bag drifting past. Whatever Jefferson’s offenses, his visions have survived, and many of those visions were a lot bigger than Thomas Jefferson.

We should all be such jerks. Well, except for owning slaves.