I have a mac pro with 2x the top FirePro cards in it, and I still get weird movement you’re talking about … it’s not relating to that for me, but rather seems to be a function of the nature of how unreal is performing that action. I noticed that ( and this is for myself ), I only get this effect if I use a mouse ( ala click+drag ). If I use my wacom tablet i do NOT have this experience.
What I found helps is if you pretend mentally that it’s like sculpting clay. You take small precise little slices that adds up to what you want ( be them up/down, whatever ). I found that long broad strokes are all over the place where I didn’t want, where you click + drag, and just expect the tool to “do it”. Instead, you make small strokes, back and forth ( i usually use some sort of elliptical back and forth ). This means you need to adjust your tool setting to reflect whatever style you tend to work with, and let’s not forget be patient.