I think you should pause to interrogate the statement “freedom to own slaves.” What do you think ownership is? Who enforces it?
If passing a law that takes away ownership is “authoritarian” in your eyes, what about the enforcement of ownership? Doesn’t the state enforcing property rights also take away certain freedoms? Not just with the obvious example of slavery, but in general.