I had similar problems with the door of ‘my bedroom’… being modular sometimes can be annoying…specially wen you link groups of object that rotate … like my door knob… i spent so many time adjusting the proxy grouping that would be easier to create the door inside the main model and everything is done remove all objects and save my door in another blender file to reuse…
the solution was put all together in one object , add a bone and link the armature and object