do you know which gameobject triggers the lose collider?
If it’s the baby then try setting the collision detection to ‘continuous’ for the baby’s RigidBody2D component (it solved some weird collision behaviours for me where the ball sometimes glitched inside the paddle)
You could also set the rigidbody's type to kinematic while it is stuck to the paddle. The kinematic mode makes the rigidbody ignore all applied external forces (like gravity).
Just add
GetComponent<RigidBody2D>().isKinematic = true;
GetComponent<RigidBody2D>().isKinematic = false;
Hope this helps