For a long time the only really thing you can do is pass -jN. -j merely parallelized the process. It was a good idea to have -j2 even on a single core just so that the CPU was busy doing something. That was often the weak point, you just couldn’t compile fast enough.

Of course, now we have dual-tri-quad cores, so now it wasn’t how fast you can compile but how fast you can write to disk. Solid state drives can help close that gap.

But Firefox is a huge project. At some point the cost of hardware to compile it in 10 seconds is just too significant, even for Mozilla with it’s fancy new Google-deal.

I wondered where the build was spending most of its time. I couldn’t find anything to profile make (at least not through the man page, and the name is not google friendly, and people don’t call it gmake anymore ). Since what I want to do is pretty simple, I downloaded the source and made a few changes to log the seconds since epoch when we enter/leave a directory. A crappy python script later, and here is the broken results:

Total time for accessible is 77 seconds. Project was entered 3 times
Total time for b2g is 0 seconds. Project was entered 0 times
Total time for browser is 13 seconds. Project was entered 3 times
Total time for build is 0 seconds. Project was entered 3 times
Total time for caps is 6 seconds. Project was entered 3 times
Total time for chrome is 6 seconds. Project was entered 3 times
Total time for config is 0 seconds. Project was entered 3 times
Total time for content is 572 seconds. Project was entered 3 times
Total time for db/sqlite3/src is 7 seconds. Project was entered 3 times
Total time for dbm is 0 seconds. Project was entered 0 times
Total time for docshell is 16 seconds. Project was entered 3 times
Total time for dom is 223 seconds. Project was entered 3 times
Total time for editor is 56 seconds. Project was entered 3 times
Total time for embedding is 23 seconds. Project was entered 3 times
Total time for gfx is 106 seconds. Project was entered 3 times
Total time for hal is 6 seconds. Project was entered 3 times
Total time for image is 18 seconds. Project was entered 3 times
Total time for intl is 31 seconds. Project was entered 3 times
Total time for ipc is 186 seconds. Project was entered 3 times
Total time for js/ductwork/debugger is 1 seconds. Project was entered 3 times
Total time for js/examples is 0 seconds. Project was entered 0 times
Total time for js/ipc is 3 seconds. Project was entered 3 times
Total time for js/jsd is 3 seconds. Project was entered 3 times
Total time for js/public is 0 seconds. Project was entered 0 times
Total time for js/src is 17 seconds. Project was entered 4 times
Total time for js/xpconnect is 61 seconds. Project was entered 3 times
Total time for layout is 378 seconds. Project was entered 3 times
Total time for media/libjpeg is 3 seconds. Project was entered 3 times
Total time for media/libnestegg is 0 seconds. Project was entered 3 times
Total time for media/libogg is 1 seconds. Project was entered 3 times
Total time for media/libpng is 1 seconds. Project was entered 3 times
Total time for media/libsydneyaudio is 0 seconds. Project was entered 3 times
Total time for media/libtheora is 2 seconds. Project was entered 3 times
Total time for media/libtremor is 0 seconds. Project was entered 0 times
Total time for media/libvorbis is 2 seconds. Project was entered 3 times
Total time for media/libvpx is 3 seconds. Project was entered 3 times
Total time for memory/jemalloc is 0 seconds. Project was entered 3 times
Total time for memory/mozalloc is 0 seconds. Project was entered 3 times
Total time for mfbt is 0 seconds. Project was entered 3 times
Total time for mobile is 0 seconds. Project was entered 0 times
Total time for modules/freetype2 is 0 seconds. Project was entered 0 times
Total time for modules/libbz2 is 1 seconds. Project was entered 3 times
Total time for modules/libjar is 6 seconds. Project was entered 3 times
Total time for modules/libmar is 1 seconds. Project was entered 3 times
Total time for modules/libpref is 7 seconds. Project was entered 3 times
Total time for modules/zlib is 1 seconds. Project was entered 3 times
Total time for mozglue is 0 seconds. Project was entered 3 times
Total time for netwerk is 126 seconds. Project was entered 3 times
Total time for nsprpub is 2 seconds. Project was entered 2 times
Total time for other-licenses is 0 seconds. Project was entered 0 times
Total time for parser is 29 seconds. Project was entered 3 times
Total time for probes is 0 seconds. Project was entered 3 times
Total time for profile is 3 seconds. Project was entered 3 times
Total time for rdf is 14 seconds. Project was entered 3 times
Total time for security/coreconf is 0 seconds. Project was entered 0 times
Total time for security/dbm is 0 seconds. Project was entered 0 times
Total time for security/manager is 109 seconds. Project was entered 3 times
Total time for security/nss/lib is 0 seconds. Project was entered 0 times
Total time for services is 2 seconds. Project was entered 3 times
Total time for startupcache is 3 seconds. Project was entered 3 times
Total time for storage is 28 seconds. Project was entered 3 times
Total time for testing is 0 seconds. Project was entered 0 times
Total time for toolkit is 88 seconds. Project was entered 3 times
Total time for tools is 0 seconds. Project was entered 0 times
Total time for uriloader is 19 seconds. Project was entered 3 times
Total time for view is 2 seconds. Project was entered 3 times
Total time for widget is 40 seconds. Project was entered 3 times
Total time for xpcom is 85 seconds. Project was entered 3 times
Total time for xpfe/appshell is 8 seconds. Project was entered 3 times
Total time for xpfe/components is 3 seconds. Project was entered 3 times
Total time for xpfe/test is 0 seconds. Project was entered 0 times
Total time for xulrunner is 0 seconds. Project was entered 0 times
The total time calculated is 2398

The total time is off. Firstly, you’ll see NSS is not being tracked. What usually happens is that the Makefile is generated in the object directory. In the case of NSS, it comes with a Makefile already. This is breaking my script, but it takes approx. 72 seconds.

Secondly, The totally time since I ran “make -f client.mk build” is approx 2662 seconds. Even factoring in the NSS time, that still leaves 3 minutes unaccounted for. So I still might be missing some projects. Part of that time was spent traversing the tree and running configure.

In the past, when we only had single-core computers, we tried fixing the problem by using tools like distcc to distribute the load to slave machines. With multi-core machines, this is less and less ideal since the overhead is quite substantial (a preprocessed file would go through and object files come back). However, I wonder if we can achieve better build times by distributing projects across machines. That would present some challenges (like dependencies), but it might give better results.

So now I have it wake the computer up at 7:50 so that in 5 minutes it can play the nightingales before playing the Good Morning Dub. One issue though is that Windows will go back to sleep in 2 minutes. You can modify that using a registry key so that it will have enough time to play the music.

Now while the alarm clock website is accomplishing the task of waking me up at the right time, it doesn’t do a very good job of keeping me awake. Maybe I need to make an interactive component to shut off the alarm

1. Cache the images so they don’t keep taking up my bandwidth
2. Prefetch the images so that a noticeable lag isn’t taking place while the image loads.

Caching is a bit tricky. While I want the names to stay consistant, I also want to be able to change the images at anytime. What I want is the browser to cache kitten.jpg until next year, or until I decide that I want a different kitten image and it should pick the new kitten.jpg (almost) immediately.

I have seen this problem solved in Ruby on Rails. What they did is append some query string identifier as part of the image. So instead of retrieving kitten.jpg, you would retrieve kitten.jpg?mittens. When the image changed, so did the identifier to something like kitten.jpg?bucket. This lets you cache aggressively, while still being able to change the image fairly regularly. (I am aware of other headers such as Last-Modified that might be a better solution. But from my understanding that only works with documents and not images. If an image is cached, the browser will use it until it expires as opposed to doing a HEAD request to see whether the Last-Modified date changed. I could be wrong about this).

I decided to do something similar, in that the md5 of the file will be appended.

The other thing I was missing was prefetching. I understand I could easily do this by hiding a bunch of images in the HTML page, or dynamically creating an Image object and setting the source. But this is 2011 dammit. We have standards that were promised, and behold link prefetching! It turns out only Mozilla and Webkit support this, which is lame. But that’s fine considering this website is only for my use.

I put in a pretch link using this method and used Firebug’s net panel to see whether it was working. And apparently it didn’t. I found out much later on that Firebug doesn’t capture it. You have to look at the server’s access log so see whether the image was retrieved.

Next, I checked out Mozilla link prefetching FAQ, which informed me that URLS with query strings are not prefetched. I found out much later that this is a lie (and removed it). This was actually fixed back in 2003 by someone who also thought it was stupid.

So here I am, also at 01:00. Reheated a plate of pasta and now enjoying some lukewarm soup. I promised myself yesterday I would be in bed by 23:00 at the latest. Tomorrow will be unpleasant, but at least this feels accomplishing.

I’ve decided to just zip up the thing and ftp it. But I found it interesting that Django projects are actually not trivial to share. For example, what files would you include?

It actually seems like you are not supposed to share settings.py (nor manage.py, or urls.py really. These are auto-generated for you). At least, the secret_key should be secret. But you also have a bunch of settings such as database information that should not be public either. If you change the file, any version control system will declare it as a changed file. If you .(git|hg)ignore it, what happens if you need to add a new setting to settings.py?

So I thought maybe you share django apps instead of the project? That’s stupid. My project has templates, static files. Are those supposed to be a separate project?

So it looks like a fucking mess and I’m zipping this shit up and calling it a day.

After seeing what Zamboni does, it appears they modified the manage.py to load a settings_local.py file instead of settings. So you would copy settings.py to settings.local.py and use it. That’s actually a pretty good idea.

Down with the system, long live the system!

One of the great joys of Firefox is that if you don’t like something, you can change it (for the most part). This was mainly done through extensions. Don’t like the default developer tools? Get something better! Concerned about privacy? You’re in luck! You did not need someone’s blessing or permission, you could just write it. Sometimes an extension is not ideal. I don’t believe you can override C++ components in Firefox, and you might have limitations to the functionality you can override depending on the browser.

For many browsers, you could write a NPAPI plugin. The benefit of this is that Flash was able to do so many things that has really only been possible in the last 3 years or so. Canvas, video, audio, offline storage was never possible without Flash when I started college. And yet it has so much influence because we knew what was possible and what was important.

We are slowly replacing Flash sure. Even the webcam might one day exist on the desktop. But you can’t have a API and a spec for everything. You just cannot satisfy everyone’s requirements. You can advocate, but the eventual decision will lie in the hands of browser vendors. Having the web compete with itself is not ideal, and we see that when you have to encode a video twice for everyone to play it.

Which is why I having a plugin service is important, and removing it is a step back. It’s not pleasant or perfect, but if it makes a web competitive outside of browser vendors, I think it’s worth it.

But the challenge really is – will anyone use it? And how do I convince people this is a good idea? Maybe good ideas do not need convincing, they just need creating.

Genres in particular are very fluid. Even though we have hundreds of genres and sub-genres, it is not uncommon for there to be ambiguities and mixes of several types of genres. It took me a while to discover the difference between trance and house music, and even today it puzzles me. A lot of the blame is due to mislabelling or generalizing of artists and albums but that is another topic.

Musical preferences is also very different between people who like even the same genre of music, because the genre itself has changed and evolved. If you listen to early dubstep (Girl from Codeine City by L-Wiz) and more recent dubstep (Game Time by Zomboy) tracks, they are two different sounds. Though, I’m fairly sure each classifies as their own sub-genre by now.

When you try to mix these two algorithmically, the results do not hold great promise. At least, in my case. If you ever listen to tsubaki, his sound is pretty unique. Beatbot will return some reggae (getting warmer), hip hop (colder), and dubstep (colder). Nothing of which sounds comparable.

Recently though, I found tsubaki on soundcloud. More importantly, soundcloud lets you see what other artists that artist is following. And I now found another artist that makes similar music – dubsalon. So here we have the artist recommending (or at least, listening) to another artist.

This is something that I think we should take advantage of, if not already (which I think we do not). This information is a bit more difficult to harvest, but might be more useful.

• Ruby blocks are *fantastic*. If you ever had to work with generating XML, I highly suggest Nokogiri to generate and parse XML documents. XPATH? Yes please! (Ok, XML sucks but when you have to use it, this library makes it painless)
• Rails has a ton of shortcuts that make development easier. They have a ton of HTML helpers to auto-generate HTML. The framework takes some getting used, but it works quite well.
• Rails expects you to develop a certain way. IMO, this a very negative attribute until you get used to it. Your controllers names MUST be plural (cashiers, merchants, etc). Your models MUST be singular (cashier, merchant, etc). Breaking the rules leads to frustration and headache.
• Rails expects you to build a certain way. In my application, I did not need to use a database (everything is done using a REST service). But Rails makes this difficult to do.
• The Rails installation procedure is easy, but a bit too easy I managed to screw up a few times (shame on me) and just did gem install rails. That will install the latest Ruby on Rails on your machine. The problem is when I had to move my code to the server. Between development, rails went up a few versions (from 3.0.x to 3.1.x) and a lot of stuff broke. I wish at the time I knew I can install a specific version. Which leads me to…
• Rails is difficult to update. Well, so are most frameworks. I think we need to get better at this developers…

It is difficult to say whether I enjoy Rails or Django. I am definitely more comfortable with Django, but it is great getting to know both. I am still a newbie at Rails, so hopefully things get easier as I do more work on this.

I went to the WIND store and complained. They offered a solution where all data that was not covered under the plan would be disabled. That was an ideal solution, so I accepted it. I had to call customer service if I wanted to dispute the data charges.

A week later and I was still being charged data. I called them up to dispute the charges. After talking to a representative (two actually. The first one I lost connection with), it turns out that I was accessing data on a WIND away zone. The horseshit term they use when you are outside their network coverage.

I live/work/play in downtown Toronto…

The last data charge was around 8pm. That was about the time I was exiting the YMCA… in downtown Toronto.

If you check their coverage map, downtown Toronto saturated as a WIND Home zone. I was apparently hitting a WIND Away zone.

This isn’t unheard of, nor unique to WIND. A friend complained that Fido once charged him an obscene amount because he was hitting a US tower while in Victoria BC. Oh yes, that will be quite a bit of roaming.

I was refunded $2 as a one time good gesture. Because I switched numbers when I went to the WIND store, they could not refund me the amount from my previous number. It was a net loss of $14 on WIND’s crappy network coverage. The only solution they offer is to call WIND and have them remove any WIND away coverage. This is not ideal in the situation that I am in one of WIND’s away zone and I will need to use my phone in an (non-911) emergency. So I would have to remember to call and re-enable it. Very inconvenient. In fact, the correct solution is for WIND to deliver the coverage they promise. Maybe they should learn a thing or two from Verizon:

http://youtu.be/OPwPo-IAQ-E

I hope that network coverage improves. The offer really good rates, but at a major cost of reliability and coverage. I will be with them for at least the next few months until I can get my phone unlocked. Let’s see if I stick with them in the long run.

Overall I do like their new website. It is fast, clean, hackable. I thought they were using the audio tag and doing some neat stuff to get the waveform but it turned out that was an image and they are still using flash. Which is probably the wisest thing to do but not very HTML5ish

Having a HTML version of the website was also something I thought would be an interesting project (back when they only had a flash-only site). A few weeks ago, I decided to do one anyways, and launched beatport5 – a HTML5 version of the website (I may end up renaming it). It has a few caveats:

• It’s pretty much not optimized for screens with a resolution under 1920×1080 oops
• It works best in Gecko browsers, needs improvement for webkit, and pretty terrible under trident browsers. Don’t even ask about anything under IE8
• The playlist is not working yet
• The search functionality is very simple. You can search for tracks, releases, artists or a global search. But you are not able to search for x track by y artist.
• The landing page needs improvements
• I am not working on this full-time

Some interesting stuff:

• We both use DOM local storage. They seems to use it for their playlist queue. Neat!

Some wishlist that I need to put on their mailing list:

1. I would like to avoid hitting my server, so it would be great if their website enabled HTTP access controls for their API and mp3 files
2. Maybe a way to add stuff to their shopping cart?
3. There’s more but I’m drawing a blank…

I haven’t really worked in what value I would add that isn’t already in the beatport site. I would like to incorporate the beatport flac converter somehow. Likely anything meaningful I want to do would have to stored on my server. One experiment I would like to attempt is a way to discover music, much like the beatbot, but maybe less like it, and more awesome.

Why release this now? Release early, release often! It’s not complete, but I hope to spend time on it and be useful to me and others as well. My main concern right now is making this work cross-browser. Next is screen resolution fix, and an appropriate landing page.