Even if you have hundreds of contrib modules loaded into /sites/all, it would not affect performance on a site that only enabled a small number of them because modules don't cause a load on the system when they are not enabled. The main downside to this would be on the usability and loading time of admin/build/modules.

Is this correct? If not, please explain.

Thank you,

Shai