Programming Horror

For one thing: don't bother with fancy log destinations. Just log to stderr and let your daemon manager take care of directing that where it needs to go. (systemd made life a lot easier in the Linux world).

Structured logging is overrated since it means you can't just do the above.

Per-module (filterable) logging are quite useful, but must be automatic (use __FILE__ or __name__ whatever your language supports) or you will never actually do it. All semi-reasonable languages support some form of either macros-which-capture-the-current-module-and-location or peek-at-the-caller-module-name-and-location.

One subtle part of logging: never conditionally defer a computation that can fail. Many logging APIs ultimately support something like:

if (log_level >= INFO) // or <= depending on how levels are numbered
    do_log(INFO, message, arguments...)

This is potentially dangerous - if logging of that level is disabled, the code is never tested, and trying to enable logging later might introduce an error when evaluating the arguments or formatting them into the message. Also, if logging of that level is disabled, side-effects might not happen.

To avoid this, do one of:

• never use the if-style deferring, internally or externally. Instead, squelch the I/O only. This can have a significant performance cost (especially at the DEBUG level), which is why the API is made in the first place.
• ensure that your type system can statically verify that runtime errors are impossible in the conditional block. This requires that you are using a sane language and logging library.
• run your testsuite at every log level, ensure 100% coverage of log code, and hope that the inevitable logic bug doesn't have an unexpected dynamic failure.