No matter, adding a NavMesh agent fixed it majorly - still a bit buggy around uneven terrain but easily mitigated.