Hi,
TL;DR: Camera solver can’t work with a fix camera only rotating, you have to check the tripod checkbox in the solve panel.
There was something I didn’t quite understand, I had a really good camera solver: 0.3762, but when I was watching through the camera’s eye, the movements were all jerky, even more the markers positions were not in the right 3D place. Have a look:
Even if I got the order roughly right (The two markers on the snow closer to the camera than the one on the mountain), the distances between markers weren’t right at all (several tens of meters between fore- and back-ground IRL and 3-4m in blender )
So I tried a bunch of stuff, refining markers, change the solver begin and end keyframe, modify the lens’ focal length …
I even took an other video in the fields near my place with roughly the same motion and got through the process again, but the same problem appeared. 
Then I looked in the blender manual (Link) and found that:
Tripod Motion can be used for footage where the camera does not move and only rotates. Such footage can’t be tracked with a generic solver approach, and it is impossible to determine the actual feature points in space due to a lack of information.
Well in my video it is the case, I am standing and doing a quick rotation, so the camera only rotate.
So I did check that tickbox, the solve error went up to 3.0273 but the result was much more stable:
All the markers are on a sphere with the tripod motion ON.
Pfew, at least problem solved ! I can include a model in there 
.
Well now I know I should do panning movement with the camera for the next CGI 
Be sure to tick the tripod box if you have the same configuration as me.
Cheers !