I fiddled around and found the most distinct difference when using HDRI Environment on the world and every other lamp completely-off. That is both eye and render off. (Blender 2.78c)
Without portal… 20 samples:
With portal and 20 samples:
It is worth mentioning that I have windows that does not stop the light…
… and that I almost always run with the following settings to reduce noise…
It’s in the Clamp Direct/Indirect under Sampling and Filter Glossy under Light Paths.