Because I already had automatic weighting applied, some vertices were lost in some groups. You can remove the armature, but those groups still exists and re-used.
I deleted those old vertices groups and tried it again. now everything works as expected.