Even if you have hundreds of contrib modules loaded into /sites/all, it would not affect performance on a site that only enabled a small number of them because modules don't cause a load on the system when they are not enabled. The main downside to this would be on the usability and loading time of admin/build/modules.
Is this correct? If not, please explain.
Thank you,
Shai