the only problem i had when i was driving a small car (i drive a mid-size car now) was my sense of vulnerability when surrounded by stupid massive lifted trucks bearing down on me. it felt really unsafe. and i live in stupid-giant-truck land, they're not an anomaly here.
eta - plus giant truck owners seem to get off on being scary aggressive drivers.