Boolean causes a lot of geometry funnies. It’s likely to be what you’re getting all the extra polies from. I mean, nothing wrong with that, it’s just a limitation of the modifier.
Looks good to me! Sure you can tidy the geometry up but remember there is a curve of time spent vs end result.
If you tidied it you’ll have spent more time on it, and I doubt any of us would have seen the difference.
Unless this is going to be used thousands of times in a scene, the performance benefits would also be negligible.
You can use the Remove Doubles tool and the Decimate modifier into planar type with a small angle limit to take out some polygons by merging the ones with a angle smaller than the limit you used. Just make sure you save it before doing it.