John Ford posted an interesting post on improving the build times of mac minis. This largely turned out to be a hardware solution (spend enough money on solid state drives and more cores and things are likely to improve). But that is expensive. What is interesting to me though is the build times. 2 hours for a dual core set-up?
For a long time the only really thing you can do is pass -jN. -j merely parallelized the process. It was a good idea to have -j2 even on a single core just so that the CPU was busy doing something. That was often the weak point, you just couldn’t compile fast enough.
Of course, now we have dual-tri-quad cores, so now it wasn’t how fast you can compile but how fast you can write to disk. Solid state drives can help close that gap.
But Firefox is a huge project. At some point the cost of hardware to compile it in 10 seconds is just too significant, even for Mozilla with it’s fancy new Google-deal.
I wondered where the build was spending most of its time. I couldn’t find anything to profile make (at least not through the man page, and the name is not google friendly, and people don’t call it gmake anymore 
). Since what I want to do is pretty simple, I downloaded the source and made a few changes to log the seconds since epoch when we enter/leave a directory. A crappy python script later, and here is the broken results:
Total time for accessible is 77 seconds. Project was entered 3 times
Total time for b2g is 0 seconds. Project was entered 0 times
Total time for browser is 13 seconds. Project was entered 3 times
Total time for build is 0 seconds. Project was entered 3 times
Total time for caps is 6 seconds. Project was entered 3 times
Total time for chrome is 6 seconds. Project was entered 3 times
Total time for config is 0 seconds. Project was entered 3 times
Total time for content is 572 seconds. Project was entered 3 times
Total time for db/sqlite3/src is 7 seconds. Project was entered 3 times
Total time for dbm is 0 seconds. Project was entered 0 times
Total time for docshell is 16 seconds. Project was entered 3 times
Total time for dom is 223 seconds. Project was entered 3 times
Total time for editor is 56 seconds. Project was entered 3 times
Total time for embedding is 23 seconds. Project was entered 3 times
Total time for gfx is 106 seconds. Project was entered 3 times
Total time for hal is 6 seconds. Project was entered 3 times
Total time for image is 18 seconds. Project was entered 3 times
Total time for intl is 31 seconds. Project was entered 3 times
Total time for ipc is 186 seconds. Project was entered 3 times
Total time for js/ductwork/debugger is 1 seconds. Project was entered 3 times
Total time for js/examples is 0 seconds. Project was entered 0 times
Total time for js/ipc is 3 seconds. Project was entered 3 times
Total time for js/jsd is 3 seconds. Project was entered 3 times
Total time for js/public is 0 seconds. Project was entered 0 times
Total time for js/src is 17 seconds. Project was entered 4 times
Total time for js/xpconnect is 61 seconds. Project was entered 3 times
Total time for layout is 378 seconds. Project was entered 3 times
Total time for media/libjpeg is 3 seconds. Project was entered 3 times
Total time for media/libnestegg is 0 seconds. Project was entered 3 times
Total time for media/libogg is 1 seconds. Project was entered 3 times
Total time for media/libpng is 1 seconds. Project was entered 3 times
Total time for media/libsydneyaudio is 0 seconds. Project was entered 3 times
Total time for media/libtheora is 2 seconds. Project was entered 3 times
Total time for media/libtremor is 0 seconds. Project was entered 0 times
Total time for media/libvorbis is 2 seconds. Project was entered 3 times
Total time for media/libvpx is 3 seconds. Project was entered 3 times
Total time for memory/jemalloc is 0 seconds. Project was entered 3 times
Total time for memory/mozalloc is 0 seconds. Project was entered 3 times
Total time for mfbt is 0 seconds. Project was entered 3 times
Total time for mobile is 0 seconds. Project was entered 0 times
Total time for modules/freetype2 is 0 seconds. Project was entered 0 times
Total time for modules/libbz2 is 1 seconds. Project was entered 3 times
Total time for modules/libjar is 6 seconds. Project was entered 3 times
Total time for modules/libmar is 1 seconds. Project was entered 3 times
Total time for modules/libpref is 7 seconds. Project was entered 3 times
Total time for modules/zlib is 1 seconds. Project was entered 3 times
Total time for mozglue is 0 seconds. Project was entered 3 times
Total time for netwerk is 126 seconds. Project was entered 3 times
Total time for nsprpub is 2 seconds. Project was entered 2 times
Total time for other-licenses is 0 seconds. Project was entered 0 times
Total time for parser is 29 seconds. Project was entered 3 times
Total time for probes is 0 seconds. Project was entered 3 times
Total time for profile is 3 seconds. Project was entered 3 times
Total time for rdf is 14 seconds. Project was entered 3 times
Total time for security/coreconf is 0 seconds. Project was entered 0 times
Total time for security/dbm is 0 seconds. Project was entered 0 times
Total time for security/manager is 109 seconds. Project was entered 3 times
Total time for security/nss/lib is 0 seconds. Project was entered 0 times
Total time for services is 2 seconds. Project was entered 3 times
Total time for startupcache is 3 seconds. Project was entered 3 times
Total time for storage is 28 seconds. Project was entered 3 times
Total time for testing is 0 seconds. Project was entered 0 times
Total time for toolkit is 88 seconds. Project was entered 3 times
Total time for tools is 0 seconds. Project was entered 0 times
Total time for uriloader is 19 seconds. Project was entered 3 times
Total time for view is 2 seconds. Project was entered 3 times
Total time for widget is 40 seconds. Project was entered 3 times
Total time for xpcom is 85 seconds. Project was entered 3 times
Total time for xpfe/appshell is 8 seconds. Project was entered 3 times
Total time for xpfe/components is 3 seconds. Project was entered 3 times
Total time for xpfe/test is 0 seconds. Project was entered 0 times
Total time for xulrunner is 0 seconds. Project was entered 0 times
The total time calculated is 2398
The total time is off. Firstly, you’ll see NSS is not being tracked. What usually happens is that the Makefile is generated in the object directory. In the case of NSS, it comes with a Makefile already. This is breaking my script, but it takes approx. 72 seconds.
Secondly, The totally time since I ran “make -f client.mk build” is approx 2662 seconds. Even factoring in the NSS time, that still leaves 3 minutes unaccounted for. So I still might be missing some projects. Part of that time was spent traversing the tree and running configure.
In the past, when we only had single-core computers, we tried fixing the problem by using tools like distcc to distribute the load to slave machines. With multi-core machines, this is less and less ideal since the overhead is quite substantial (a preprocessed file would go through and object files come back). However, I wonder if we can achieve better build times by distributing projects across machines. That would present some challenges (like dependencies), but it might give better results.