When the user experience begins to suffer. Things continue to make sense contextually until there's too high of a concentration of things to deal with.
Amazon's philosophy is the world's market place, where you can get everything from "a to z." While it's a giant site, it revolves around a common theme with common values, serving a specific type of consumer (someone looking to buy um, anything really). One con of this is it's a very cost intensive operation to run this optimally, which isn't a problem if you can clear being in the red for as many years as Amazon has but still maintain investor interest.
StackExchange separates users by interests not because they won't co-mingle, but they likely feel that people benefit from participating in Q&As related to their interests. When you're consuming social information, it's a lot easier if it's curated (Reminds me of how much I hated Twitter until I started making lists). This is a much more cost effective situation as certain sites can scale up and down, and be dissolved or consolidated without impact on the majority of users.
The Area 51 stack exchange is kind of interesting. It's a single site where a few communities "test." Also, Quora is kind of like SE on a single site. Browse around and see how you feel in contrast over the two. Can't please all of the people all of the time, but you can address a market gap by filling in the niches, even if that niche is a giant site that sells everything. :-)
