Setting freezeRotation to false removes all rotation constraints, so you need to put them back. My code is the following for it.
// resume physics control of rotation
rigidBody.freezeRotation = false;
// reapply constraints
rigidBody.constraints = RigidbodyConstraints.FreezeRotationX
| RigidbodyConstraints.FreezeRotationY
| RigidbodyConstraints.FreezePositionZ;