(like people who get angry at something you say and go to your profile to systematically downvote everything you’ve done, or organized dogpile voting, or …)
I actually saw a system once for dealing with that that I thought had serious potential. If you wanted to downvote someone, it cost you time. Every time you downvoted the system would pause you, rendering you unable to use it for a period of time. On your first downvote it was measured in milliseconds, but with every downvote you cast in a given time period (by default it was the day, I think?) the pause increased exponentially. So by your 20th downvote you were being frozen for a minute and by the time you hit your hundredth you were freezered for a week. (It was, actually, technically speaking, impossible to reach your hundredth as a result.)
The idea behind this was that the community could downvote you to perdition if you were a jackass (since it would be a miniscule freeze time for them), but if you tried to counter that by downvoting everybody who downvoted you, you'd rapidly be frozen out of the community.
Of course the problem with that was that it was based on the naive supposition that people wouldn't coordinate downvoting circles; that you wouldn't be able to arrange brigading and dogpiling. But I still think something interesting could be salvaged from the idea by people smarter than I am. After all the statistics are all there and it should be possible to identify voting circles, sock puppet accounts, and the like from statistical behaviour, no?
Yeah, monetary costs work too. The idea is that it has to cost.
Though with monetary the costs would still have to go up exponentially so that you don't have some deep-pocketed Apartheid Manchild literally nickel and diming a server to death.