I just want to leave this solution here if somebody else stumbles across the same bug.
The icons for the Waypoints were not showing up in my Scene, no matter how much I turned Gizmo"s on and off. This seems to be a problem with Unity (I was on version 2021.3.19f1) when I updated the project to 2021.3.21f1 it was fixed and I could see those yellow diamonds in my scene.