Upside of a small community: more interaction between the users that do comment.
Downside of a small community: if someone's a dick, you can't really completely get away from them.
But the question is, can you handle one dick, or do you prefer dealing with hundreds?