Topic: TIP: CPU bottleneck
I've been uploading several large (50k row) datasets into a test warehouse hosted on my local Ubuntu VM. They were taking many hours to upload so I thought I'd try boosting the specification of the VM to speed things up.
First I upped the RAM from 1MB to 4MB. This didn't have much if any effect.
Next I added an extra CPU core, giving me 2 in total. This had a very pronounced effect: it more or less halved the import time. Looking at Top (the *nix equivalent of Windows Task Manager) both Apache and Postgres fight for the CPU during an import, so adding the extra core made all the difference.
So if you're looking to speed up your warehouse, it seems that adding an extra CPU core is a useful optimisation. It also highlights the benefit of using a VM; it was super-easy to add extra memory and an extra CPU without having to change actual hardware.
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital