I used a layermask

[Cat_Hill]

I ended up using a layermask in mine figuring I only wanted to test against the terrain for movement purposes.

Also for the person asking about performance, one or two more raycasts per frame are a pretty minor performance cost but if you are performance conscious (e.g. targeting older generation devices) the layermask might improve your performance too.

        private bool InteractWithMovement()
        {
            bool hasHit = Physics.Raycast(GetMouseRay(), out RaycastHit hit, terrainLayerMask);
            if (hasHit)
            {
                if (Input.GetMouseButtonDown(0)) {
                    mover.MoveTo(hit.point);
                }
                return true;
            }
            return false;
        }