I took a bit of a different approach that does one RaycastAll.
void PerformRaycasts() {
Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
RaycastHit[] raycastHits = Physics.RaycastAll(ray, maxRaycastDepth).OrderBy(h => h.distance).ToArray();
foreach (RaycastHit hit in raycastHits) {
if (CheckForEnemy(hit)) return; // Specify layer priorities here (order of checks matters)
if (CheckForWalkable(hit)) return;
}
Cursor.SetCursor(unknownCursor, hotSpot, CursorMode.Auto);
}
This is a bit more efficient and deterministic. The Unity doc says that Physics.Raycast “Casts a ray, from point origin, in direction direction, of length maxDistance, against all colliders in the scene”. It will return a indeterminate collider from among all those that pass the layerMask.