Characters being unconvincingly incompetent or foolish. Alien and Aliens are some of my favorite horror movies to this day because the characters behave in a consistent and believable fashion based on who they are and what they know. The space truckers aren't at all equipped to handle the situation but they apply their knowledge as best they can. The marines are totally absorbed in their self image as gung ho badasses and so wildly underestimate their enemy while making foolish but believable decisions.
Basically; I like when the characters behave competently and lose not because the writer's made them carry the idiot ball and behave foolishly, but because the threat overwhelmed their competence and resourcefulness. It makes the threat more legitimately frightening; these peopkle did their best and were still wiped out, so hwo could I expect to do better?