For a long time the only really thing you can do is pass -jN. -j merely parallelized the process. It was a good idea to have -j2 even on a single core just so that the CPU was busy doing something. That was often the weak point, you just couldn’t compile fast enough.

Of course, now we have dual-tri-quad cores, so now it wasn’t how fast you can compile but how fast you can write to disk. Solid state drives can help close that gap.

But Firefox is a huge project. At some point the cost of hardware to compile it in 10 seconds is just too significant, even for Mozilla with it’s fancy new Google-deal.

I wondered where the build was spending most of its time. I couldn’t find anything to profile make (at least not through the man page, and the name is not google friendly, and people don’t call it gmake anymore ). Since what I want to do is pretty simple, I downloaded the source and made a few changes to log the seconds since epoch when we enter/leave a directory. A crappy python script later, and here is the broken results:

Total time for accessible is 77 seconds. Project was entered 3 times
Total time for b2g is 0 seconds. Project was entered 0 times
Total time for browser is 13 seconds. Project was entered 3 times
Total time for build is 0 seconds. Project was entered 3 times
Total time for caps is 6 seconds. Project was entered 3 times
Total time for chrome is 6 seconds. Project was entered 3 times
Total time for config is 0 seconds. Project was entered 3 times
Total time for content is 572 seconds. Project was entered 3 times
Total time for db/sqlite3/src is 7 seconds. Project was entered 3 times
Total time for dbm is 0 seconds. Project was entered 0 times
Total time for docshell is 16 seconds. Project was entered 3 times
Total time for dom is 223 seconds. Project was entered 3 times
Total time for editor is 56 seconds. Project was entered 3 times
Total time for embedding is 23 seconds. Project was entered 3 times
Total time for gfx is 106 seconds. Project was entered 3 times
Total time for hal is 6 seconds. Project was entered 3 times
Total time for image is 18 seconds. Project was entered 3 times
Total time for intl is 31 seconds. Project was entered 3 times
Total time for ipc is 186 seconds. Project was entered 3 times
Total time for js/ductwork/debugger is 1 seconds. Project was entered 3 times
Total time for js/examples is 0 seconds. Project was entered 0 times
Total time for js/ipc is 3 seconds. Project was entered 3 times
Total time for js/jsd is 3 seconds. Project was entered 3 times
Total time for js/public is 0 seconds. Project was entered 0 times
Total time for js/src is 17 seconds. Project was entered 4 times
Total time for js/xpconnect is 61 seconds. Project was entered 3 times
Total time for layout is 378 seconds. Project was entered 3 times
Total time for media/libjpeg is 3 seconds. Project was entered 3 times
Total time for media/libnestegg is 0 seconds. Project was entered 3 times
Total time for media/libogg is 1 seconds. Project was entered 3 times
Total time for media/libpng is 1 seconds. Project was entered 3 times
Total time for media/libsydneyaudio is 0 seconds. Project was entered 3 times
Total time for media/libtheora is 2 seconds. Project was entered 3 times
Total time for media/libtremor is 0 seconds. Project was entered 0 times
Total time for media/libvorbis is 2 seconds. Project was entered 3 times
Total time for media/libvpx is 3 seconds. Project was entered 3 times
Total time for memory/jemalloc is 0 seconds. Project was entered 3 times
Total time for memory/mozalloc is 0 seconds. Project was entered 3 times
Total time for mfbt is 0 seconds. Project was entered 3 times
Total time for mobile is 0 seconds. Project was entered 0 times
Total time for modules/freetype2 is 0 seconds. Project was entered 0 times
Total time for modules/libbz2 is 1 seconds. Project was entered 3 times
Total time for modules/libjar is 6 seconds. Project was entered 3 times
Total time for modules/libmar is 1 seconds. Project was entered 3 times
Total time for modules/libpref is 7 seconds. Project was entered 3 times
Total time for modules/zlib is 1 seconds. Project was entered 3 times
Total time for mozglue is 0 seconds. Project was entered 3 times
Total time for netwerk is 126 seconds. Project was entered 3 times
Total time for nsprpub is 2 seconds. Project was entered 2 times
Total time for other-licenses is 0 seconds. Project was entered 0 times
Total time for parser is 29 seconds. Project was entered 3 times
Total time for probes is 0 seconds. Project was entered 3 times
Total time for profile is 3 seconds. Project was entered 3 times
Total time for rdf is 14 seconds. Project was entered 3 times
Total time for security/coreconf is 0 seconds. Project was entered 0 times
Total time for security/dbm is 0 seconds. Project was entered 0 times
Total time for security/manager is 109 seconds. Project was entered 3 times
Total time for security/nss/lib is 0 seconds. Project was entered 0 times
Total time for services is 2 seconds. Project was entered 3 times
Total time for startupcache is 3 seconds. Project was entered 3 times
Total time for storage is 28 seconds. Project was entered 3 times
Total time for testing is 0 seconds. Project was entered 0 times
Total time for toolkit is 88 seconds. Project was entered 3 times
Total time for tools is 0 seconds. Project was entered 0 times
Total time for uriloader is 19 seconds. Project was entered 3 times
Total time for view is 2 seconds. Project was entered 3 times
Total time for widget is 40 seconds. Project was entered 3 times
Total time for xpcom is 85 seconds. Project was entered 3 times
Total time for xpfe/appshell is 8 seconds. Project was entered 3 times
Total time for xpfe/components is 3 seconds. Project was entered 3 times
Total time for xpfe/test is 0 seconds. Project was entered 0 times
Total time for xulrunner is 0 seconds. Project was entered 0 times
The total time calculated is 2398

The total time is off. Firstly, you’ll see NSS is not being tracked. What usually happens is that the Makefile is generated in the object directory. In the case of NSS, it comes with a Makefile already. This is breaking my script, but it takes approx. 72 seconds.

Secondly, The totally time since I ran “make -f client.mk build” is approx 2662 seconds. Even factoring in the NSS time, that still leaves 3 minutes unaccounted for. So I still might be missing some projects. Part of that time was spent traversing the tree and running configure.

In the past, when we only had single-core computers, we tried fixing the problem by using tools like distcc to distribute the load to slave machines. With multi-core machines, this is less and less ideal since the overhead is quite substantial (a preprocessed file would go through and object files come back). However, I wonder if we can achieve better build times by distributing projects across machines. That would present some challenges (like dependencies), but it might give better results.

1. Cache the images so they don’t keep taking up my bandwidth
2. Prefetch the images so that a noticeable lag isn’t taking place while the image loads.

Caching is a bit tricky. While I want the names to stay consistant, I also want to be able to change the images at anytime. What I want is the browser to cache kitten.jpg until next year, or until I decide that I want a different kitten image and it should pick the new kitten.jpg (almost) immediately.

I have seen this problem solved in Ruby on Rails. What they did is append some query string identifier as part of the image. So instead of retrieving kitten.jpg, you would retrieve kitten.jpg?mittens. When the image changed, so did the identifier to something like kitten.jpg?bucket. This lets you cache aggressively, while still being able to change the image fairly regularly. (I am aware of other headers such as Last-Modified that might be a better solution. But from my understanding that only works with documents and not images. If an image is cached, the browser will use it until it expires as opposed to doing a HEAD request to see whether the Last-Modified date changed. I could be wrong about this).

I decided to do something similar, in that the md5 of the file will be appended.

The other thing I was missing was prefetching. I understand I could easily do this by hiding a bunch of images in the HTML page, or dynamically creating an Image object and setting the source. But this is 2011 dammit. We have standards that were promised, and behold link prefetching! It turns out only Mozilla and Webkit support this, which is lame. But that’s fine considering this website is only for my use.

I put in a pretch link using this method and used Firebug’s net panel to see whether it was working. And apparently it didn’t. I found out much later on that Firebug doesn’t capture it. You have to look at the server’s access log so see whether the image was retrieved.

Next, I checked out Mozilla link prefetching FAQ, which informed me that URLS with query strings are not prefetched. I found out much later that this is a lie (and removed it). This was actually fixed back in 2003 by someone who also thought it was stupid.

So here I am, also at 01:00. Reheated a plate of pasta and now enjoying some lukewarm soup. I promised myself yesterday I would be in bed by 23:00 at the latest. Tomorrow will be unpleasant, but at least this feels accomplishing.

I’ve decided to just zip up the thing and ftp it. But I found it interesting that Django projects are actually not trivial to share. For example, what files would you include?

It actually seems like you are not supposed to share settings.py (nor manage.py, or urls.py really. These are auto-generated for you). At least, the secret_key should be secret. But you also have a bunch of settings such as database information that should not be public either. If you change the file, any version control system will declare it as a changed file. If you .(git|hg)ignore it, what happens if you need to add a new setting to settings.py?

So I thought maybe you share django apps instead of the project? That’s stupid. My project has templates, static files. Are those supposed to be a separate project?

So it looks like a fucking mess and I’m zipping this shit up and calling it a day.

After seeing what Zamboni does, it appears they modified the manage.py to load a settings_local.py file instead of settings. So you would copy settings.py to settings.local.py and use it. That’s actually a pretty good idea.

• Ruby blocks are *fantastic*. If you ever had to work with generating XML, I highly suggest Nokogiri to generate and parse XML documents. XPATH? Yes please! (Ok, XML sucks but when you have to use it, this library makes it painless)
• Rails has a ton of shortcuts that make development easier. They have a ton of HTML helpers to auto-generate HTML. The framework takes some getting used, but it works quite well.
• Rails expects you to develop a certain way. IMO, this a very negative attribute until you get used to it. Your controllers names MUST be plural (cashiers, merchants, etc). Your models MUST be singular (cashier, merchant, etc). Breaking the rules leads to frustration and headache.
• Rails expects you to build a certain way. In my application, I did not need to use a database (everything is done using a REST service). But Rails makes this difficult to do.
• The Rails installation procedure is easy, but a bit too easy I managed to screw up a few times (shame on me) and just did gem install rails. That will install the latest Ruby on Rails on your machine. The problem is when I had to move my code to the server. Between development, rails went up a few versions (from 3.0.x to 3.1.x) and a lot of stuff broke. I wish at the time I knew I can install a specific version. Which leads me to…
• Rails is difficult to update. Well, so are most frameworks. I think we need to get better at this developers…

It is difficult to say whether I enjoy Rails or Django. I am definitely more comfortable with Django, but it is great getting to know both. I am still a newbie at Rails, so hopefully things get easier as I do more work on this.

Overall I do like their new website. It is fast, clean, hackable. I thought they were using the audio tag and doing some neat stuff to get the waveform but it turned out that was an image and they are still using flash. Which is probably the wisest thing to do but not very HTML5ish

Having a HTML version of the website was also something I thought would be an interesting project (back when they only had a flash-only site). A few weeks ago, I decided to do one anyways, and launched beatport5 – a HTML5 version of the website (I may end up renaming it). It has a few caveats:

• It’s pretty much not optimized for screens with a resolution under 1920×1080 oops
• It works best in Gecko browsers, needs improvement for webkit, and pretty terrible under trident browsers. Don’t even ask about anything under IE8
• The playlist is not working yet
• The search functionality is very simple. You can search for tracks, releases, artists or a global search. But you are not able to search for x track by y artist.
• The landing page needs improvements
• I am not working on this full-time

Some interesting stuff:

• We both use DOM local storage. They seems to use it for their playlist queue. Neat!

Some wishlist that I need to put on their mailing list:

1. I would like to avoid hitting my server, so it would be great if their website enabled HTTP access controls for their API and mp3 files
2. Maybe a way to add stuff to their shopping cart?
3. There’s more but I’m drawing a blank…

I haven’t really worked in what value I would add that isn’t already in the beatport site. I would like to incorporate the beatport flac converter somehow. Likely anything meaningful I want to do would have to stored on my server. One experiment I would like to attempt is a way to discover music, much like the beatbot, but maybe less like it, and more awesome.

Why release this now? Release early, release often! It’s not complete, but I hope to spend time on it and be useful to me and others as well. My main concern right now is making this work cross-browser. Next is screen resolution fix, and an appropriate landing page.

Despite Firefox extensions growing more complex, there are two areas of extension development that are still a pain to deal with: debugging and logging.

What pains me is that extensions have a pretty crappy choice of methods to use for logging purposes. There is dump(), Components.utils.reportError(), FUEL’s Application.console.log(), and maybe some others I am missing (ChromeBug is another possibility, but I haven’t used it). But these methods of logging don’t scale very well. If you have more than 1 extension using the same logging service, you introduce noise that can be difficult to sort through.

It striked me that there was nothing simple to capture and filter your logging information. Since this problem was annoying me, I decided to create an extension to deal with this.

Introducing Debug Log

The extension is uncreatively called Debug Log (unrelated to Jeremy Gillick’s DebugLogger). It shows a Windows-style event viewer with basic filtering. To use it is really easy. First I will show how it is used, and next how to use it when the Debug Log extension isn’t installed:

var log = {};
(function() {
        var modules = {};
        Components.utils.import("chrome://debuglog/content/DebugLog.jsm",
        modules);
        log = new modules.DebugLog("my_extension_slug_name");
        log.info("Hello")
        log.warn("Trimming to 8 characters. String : " + s);
        log.error("MyTerribleFunction", exception);
        log.assert(foo != null, "foo should never be null");
})();

Which results in the following:

Assert is a bit different from typical assertion in other languages. It will not quit the application, nor would it throw an exception (unless you passed a wrong parameter to assert).

To use it properly, you must also account for times when DebugLog is not installed.

var logging = {};
(function() {
        try
        {
                var modules = {};
                Components.utils.import("chrome://debuglog/content/DebugLog.jsm",
                modules);
                logging = new modules.DebugLog("JavascriptPlusPlus");
        }
        catch (e)
        {
                logging.assert = logging.warn = logging.info = logging.error = function() {}
        }
})()
logging.info("Starting...");

The extension still has a lot of work to be done, but it’s useable now. So now is the time to release and it is available at https://addons.mozilla.org/en-US/firefox/addon/246799/. Enjoy!

I have been busy making my own implementation of SHA-1. To better learn about why so many people depend on it for basically everything from SSL to tamper detection mechanism. I have a bigger project idea, but that is not important right now. What is important is that SHA-1 does everything in big endian, and I am on x86-64 which is a little endian machine.

Remember that a big endian machine has the most significant byte first, and little endian has the most significant byte last.

For example, let’s say I want a 64-bit integer to hold the number 1. This is how it’ll be stored:
Big endian:
1 = 0000 0000 0000 0000 0000 0000 0000 0001
Little endian:
1 = 0001 0000 0000 0000 0000 0000 0000 0000

SHA-1 stores the size of the message as a 64-bit integer in the last block during padding (each block is 512 bits). Since I have a little-endian machine, I wrote a function that correctly switches endian and now, the 1 appears as the as it should.

However, SHA-1 loops through each block in 32-bit integers.

*((unsigned int*)0000 0000 0000 0000) = 0
*((unsigned int*)0000 0000 0000 0001) = 16 million and change on little endian machine instead of 1 as I expect

so the second time, I have to do another endian change, this time a 32-bit endian change, so that it appears as :
0001 0000 0000 0000

so I get back 1.

This is a PITA, and a frustrating one. Mainly because I couldn’t figure it out for a few days. But feel so accomplished for figuring it out. Accomplished and embarrassed.

Here, we start initializing the AES object using CBC mode:

>>> from Crypto.Cipher import AES;
>>> aes = AES.new('some key here', AES.MODE_CBC, 'INIT_VECTOR')
Traceback (most recent call last):
File "<console>", line 1, in <module>
ValueError: IV must be 16 bytes long

oops. You’ll have to make you’re initialization vector 16 bytes long. Also, your key has to be 16, 24, or 32 bytes long as well. Let’s do something better :

>>> aes = AES.new('J2-+sfd%932mIt:{', AES.MODE_CBC, 'wir&/>H54mgd9a";')

ah! much better. Even if it was me smashing my hand against the keyboard. Now let’s encrypt/decrypt something important.

>>> aes.encrypt('the answer to life the universe and everything is 42')
Traceback (most recent call last):
File "<console>", line 1, in <module>
ValueError: Input strings must be a multiple of 16 in length

You’ll have to do the dirty work remember:

>>>> ciphertext = aes.encrypt('the answer to life the universe and everything is 42195479204957')
>>> ciphertext
'f0\xa9\xf9f&X)\x0e\x08=\x06\x97\xcbF\xddK\x1a\xa6i\x1d\x02"}\xd9\\\xaa\xb6\xd9J\xe3Q\x07\xaev\x012\xbf\rPN\xd2\xf9\xf7$\x93\xe0/\xcb\xae9\x91K\xd01\xab\xb7\xdb\reR\xff\xef\x1c'

Much better. Now lets decrypt it:

>>> aes.decrypt(ciphertext)
'\xc8\xaf.\x97\x05\x80\n\xe9\xe6\xc4Ju\x04\xbe\xa1Nfe the universe and everything is 42195479204957'

Woah! That isn’t the whole message! So what’s going on?

Remember that initialization vector you set in the beginning? That sets the stage for the first block. But each block becomes the initialization vector for the second block, and so on. So when you decrypt, it is using the initialization vector from the block before. That’s why the first 16 bytes are screwed up. This is a feature of CBC, but not ECB:

>>> aes = AES.new('J2-+sfd%932mIt:{', AES.MODE_ECB, 'wir&/>H54mgd9a";')
>>> ciphertext = aes.encrypt('the answer to life the universe and everything is 42195479204957')
>>> aes.decrypt(ciphertext)'the answer to life the universe and everything is 42195479204957'

And yes, this is a feature. Read the block cipher modes wikipedia article for a better explination. So what’s the answer? Simply, to call aes.new() again before calling decrypt!

The reason maps doesn’t support Minefield is because of *drumrolls* … browser sniffing. Developers… no wait… GOOGLE web developers, I thought we moved on?

The actual bit of code is here unminimized and tidied up ;

function isBrowserGeolocationSupported(){
    if (window.navigator &&
        navigator.userAgent.search("Firefox") != -1 &&
        navigator.geolocation)
        return true;
    if (window.navigator &&
        navigator.userAgent.search("Chrome") != -1)
        return Number(String(/Chrome\/[0-9]+/.exec(navigator.userAgent)).substr(7))>=2;
    var gearsFactory=null;

The hell? Ok, so I understand they do a bit of browser sniffing because it looks like Chrome had a old/broken implementation of geolocation. But I wish there was a more graceful way of doing this (maybe something like navigator.geolocation.version < 1). One that didn't break every application that may implement geolocation that isn't named Firefox. Because, those exist too.

But as anyone can tell you, it’s not an easy task (going towards damn near impossible). Firstly, you can’t really use a lexical parser. Well, you can, but it won’t be dependable. Let’s take an example out of the Reviewer’s guide :

document["crea" + "teElement"]("s" + "c" + "r" + ["i", "p", "t"].join(""));

Which is sneaky way of creating a script element, though I question the competence of someone who will use this as their main line of attack (it’s practically spelled out for you). But taking this as a use case, and ignoring the fact that they can use document[cheese] instead, I wondering if parsing the javascript would make figuring this stuff out any better.

Happily, I have spidermonkey and a js shell to do any parsing I wish. But I found out some cool things that you can do in the shell, such as looking at the bytecode for an entire function using the dis() command.

This was interesting. Certainly, there are some optimizations you can do for :
document["crea" + "teElement"]("s" + "c" + "r" + ["i", "p", "t"].join(""));
I would be shocked if it didn’t end up spelling out :
document["createElement"]("script");

I had a few hurdles to overcome. Firstly, document is not defined in the javascript shell. Thinking it was defined in the xpcshell (owww. I was misled by some apparently unused tests and my general ignorance that xpcshell tests went into unit/ and not js/ directory) I went through the added trouble of coping dis() and related functions from js.cpp to xpcshell.cpp. Once I realized that document wasn’t defined, I made a document mock object just to see what the blasted bytecode would look like.

I was a little disappointed. This source:

var document = {
createElement : function(s) {
print("damn");
}
};

function foo() {
document["crea" + "teElement"]("s" + "c" + "r" + ["i", "p", "t"].join(""));
}

dis(foo);

Ended up being this bytecode :

00000:  name "document"
00003:  string "createElement"
00006:  callelem
00007:  string "s"
00010:  string "c"
00013:  add
00014:  string "r"
00017:  add
00018:  newinit 3
00020:  zero
00021:  string "i"
00024:  initelem
00025:  one
00026:  string "p"
00029:  initelem
00030:  int8 2
00032:  string "t"
00035:  initelem
00036:  endinit
00037:  callprop "join"
00040:  string ""
00043:  call 1
00046:  add
00047:  call 1
00050:  pop
00051:  stop

Source notes:
  0:     0 [   0] newline
  1:     6 [   6] pcbase   offset 6
  3:    37 [  31] xdelta
  4:    37 [   0] pcbase   offset 19
  6:    43 [   6] pcbase   offset 25
  8:    47 [   4] pcbase   offset 47

So, almost. The document["createElement"] part was correct, but the .join() wasn’t optimized. Although I wasn’t overly estatic, I knew that this was just one (somewhat lame) use case in the countless of possible others.

This is making me rethink whether lexical tools are the way to go. While they don’t give you any definitive proof that there is a possible security hole, they can still be useful. For example, if you want to use XMLHttpRequest, then you have to call it at least once in your program (even if you say var Widget = XMLHttpRequest). And at least that can bring up warning flags, or at least give editors a place to look.

I don’t think any tool can completely replace a human being. But hopefully, tools will help make the review process easier because you can start looking at high-risk areas first rather than starting from a arbitrary point and not coming across something until 10 minutes later.