I tested out what would happen if I would give a negative score and it ended up going below zero which made me a little bit confused.
For anyone else that got a little confused by this, you have to save the variable that gets returned by Clamp method.
public void ModifyScore(int scoreModifier)
{
score += scoreModifier;
score = Mathf.Clamp(score, 0, int.MaxValue);
Debug.Log(score);
}