Hi,
Pause your game when this problem occurs. Then check the ladder collider. The ladder collider is an edge collider. The player must be touching at least one of the two vertical edges to be able to “climb”. If the gap between the two edges is too large, it might be that the player starts falling down between them. That could result in the behaviour shows in your gif.
If the gap is too large, edit the physics shape of the ladder.
See also:
- Forum User Guides : How to mark a topic as solved