Might be caused by sediment in your hot water heater (assuming you are using a tank water heater). If you're not familiar, there are tons of articles out there on how to clean it out.
A secondary concern is that you seem to have hot water moving through the system when you don't expect it. Do you have any water circulators on the house on faucets (so you get instant hot water at faucets?). That could explain it. Otherwise, you might have a leak somewhere.