If you are like me and using the course several years after it was created and decided to default to TextMeshPro you will not be able to click through your damage text, and the cursor will switch to UI. Setting layer to Ignore Raycast does not fix this. This issue means there is effectively a large deadzone where you cannot click for move/combat/etc located at the top of every object using the character prefab.
After digging for awhile i finally figured out that for some reason TextMeshPro objects have a checkbox for Raycast Target in the Extra Settings, and this supercedes the Ignore Raycast layer, once you disable that you will no longer have the issue.