Before we start, what you need to realize is that dinosaurs were definitely large, but not so large. You probably know the numbers: the largest land mammals ever are around 6–8 meters long (19-26 feet), while the largest dinosaurs were … is it 40 meters (131 feet)?
Damn, what a number!
However, numbers can be veeeery misleading. Look at the second-largest land mammal ever, Indricotherium, and one of the largest dinosaurs, Brachiosaurus, here.
The difference seems to be incomparable …
Those are two entirely different body shapes: most of the brachiosaur's length is used up by its enormous neck and tail. To make it fair, I want you to use your two thumbs: place one over the dinosaur’s neck, and the other over the tail (hopefully, you are not reading this from a touchscreen).
And suddenly, enormous becomes quite … normous. Obviously, Brachiosaurus is still larger than Indricotherium, but it's not four times larger like the numbers would suggest. The real, fair difference between the two is roughly the same as the difference between an elephant and a hippo:
Moral of the story: don't let the body shape mislead you.
So here's the answer to the "so large" part of your question: because they weren't.
However, there is still some "true" difference in size to account for. And at least two factors could've contributed to it:
1) Different rules of herbivory.
In the age of mammals, the most effective strategy of herbivory is grazing.
Grasslands are super-effective. The two most productive mammal-dominated ecosystems ever are savannas and (now gone) mammoth steppes: both can feed enormous numbers of huge mammals. With grasses growing at insane rates everywhere, no other food source on Earth can provide for such high mammalian biomasses.
Moral of the story: if you want to grow up big and full, eat grasses.
However, it wasn't always so. In times of dinosaurs, grasses didn't exist. So, the largest animals then were forced to resort to the second-best herbivory strategy: browsing.
Tree foliage doesn't grow like grasses, yet still there's usually a considerable amount of it per area unit, because it overlaps vertically many times.
Dinosaurs that fed from canopies could afford to grow large: for thermoregulation or defense from predators—usual reasons.
Any animal that grows too big inevitably experiences difficulties with food. At present, any herbivore that became too large would likely just move onto grasses. But dinosaurs couldn't. Hence, the only solution that they had was to grow necks even longer to get even more foliage. But if you grow a larger neck, you also need a larger tail (for balance). Then, you also need broader and thicker bones for all those muscles to attach, stronger legs to support the extra tons of weight, and so on and so on.
Effectively, it was a dead loop: dinosaurs became large, then they grew longer necks to support the growing need for food, which in turn made them become even larger, which in turn further increased their need for food. Browsing herbivory was likely the driving force of sauropod size, and in the end, the only limiting factor was probably the height of the highest canopy.
2) Reproductive limitations
This one doesn't really answer the "why sauropods were large?," but the "why mammals aren't that large?".
A typical sauropod was, effectively, a reproductive frog. It laid dozens if not hundreds of small eggs that hatched into very small babies that had little to do with adults: they occupied very different niches and fed on different food. For sauropods, it killed two problems: firstly, it made pregnancies easy and unnoticeable (which is a factor when you weighed 60 metric tons), and secondly, it removed competition for food between adults and babies.
In other words, sauropods could afford to become as large as necessary without worrying much about how it would affect their pregnancy and reproduction.
On the contrary, being a pregnant 60-tonne (66-ton) mammal is a nightmare—of a real and deadly kind.
All (placental) mammals bear relatively large offspring. However, if you weighed 60 tonnes, that would be … what, 2 tonnes (4400 pounds) heavy offspring? Carrying extra 10 kilograms (22 pounds) of weight at the peak of pregnancy is difficult enough for humans, but having to carry 10 extra tonnes (22,000 pounds) is just impossible, unless you are a whale and swim.
Not to mention that it would be a very long pregnancy.
Not to mention that pregnant females require even more food.
Not to mention that the young must be fed, only to grow up to compete with you for the same food later.
Moral of the story: children are expensive … unless you are a frog or a sauropod.
Q: How about livebearing smaller babies?
There are two problems with this. Firstly, it just doesn't happen. There are relatively small newborns in some placental mammals, but nothing like the difference between sauropod adults and babies.
Secondly, if babies are too small, then they become unavailable for social interactions: in fact, they are better to stay away from parents immediately to avoid being stomped on. Social behavior and learning are the backbone of mammalian success. Trying to get rid of it just isn't worth it.
So in the end, dinosaurs that weren't so large were large because they bred like frogs and because their kitchen was … a little underrepresented.