Anyway, thinking furher about the indexing problem, I manually launched cron.php when iowait was 0,...%. When I did this, iowait went up to 70%, 80% and stayed very high (+50%). Restarting mysql brought it down to 0,...% again.
So I'm thinking both problems may be connected: the broken search indexing and cron.php that causes iowait to go way up - and causing it to stay very high. We also see that there is a high load on mysql.
- Set up a MySQL slow query log on your test machine.
- OPTIMIZE all your tables.
- There is a setting on the search config page that allows you to control the number of nodes indexed per cron run. Try reducing it.
- Try disabling the search module and run cron. If it isn't the search module causing an issue, try disabling other modules .. ad nauseum.
etc. etc.