Taming the beast from within

The next 5 paragraphs are me whining. To get to the real import stuff, start at paragraph 6

So I have been pouring two weeks into WildOn, which is finding out how many addons exist out there in the wild. But before I start unleashing web crawlers on the web causing havoc and chaos, it will be helpful if we could compare what’s out there with what we know. What we know is everything from AMO, so we start there. The point of this extra work is to have some results, so that when we release a web crawler on AMO and tell it to find all the extensions, we’ll have something to compare it’s results to.

Actually, even this was a bit confusing. AMO provides an API to view its addons (well actually, two versions of the API, with the older being slightly more useful). But that information was eventually scrapped for several reasons. The main one being is that there is a lot of information on AMO that isn’t on the extension itself (such as, What operating systems are supported, and is the addon a theme or an extension. While the former has been supported since Firefox 2, I have rarely seen it used, the latter is completely optional). This makes any sort of conclusion inconclusive because you don’t have enough information.

Then there was the problem of having too much information in the database. To the point where ~4000 addons took up ~1.8gigs of information. To an sqlite datbase, this can get slow. When you try some queries, such as the number of extensions that support the ‘jp-JP’ locale, this can get to be even more intensive process as you build a table that comprises of tens of thousands of rows (one row for each guid/locale combination). The reason for this is because older versions where being included in the same table as the newest version of the addon. Some addons had something like 50+ different versions. The solution seemed to be to move old extensions to a different tables. SQL queries seem to go much faster.

Another issue that makes me loathe RDF is install.rdf. I strongly disagree with the use of rdf for anything It becomes difficult to parse with a regular xml parser (there are a few python rdf libraries out there. But rdflib, the most promising, seems to like not working and not having good examples. Only sheppy can save them now, but he’s working on mdc). Especially with rdf:resource, which I am completely ignoring right now. So it seems that AMO editors like to get creative with install.rdf, which has caused problems for me (eg. I can not rely on targetPlatform. Some extensions actually have their targetPlatoform in the Description tag. I know this because one of the extensions had Firefox’s GUID ). Also, some other quirks like having the id as an attribute of Description instead of a new tag. All things that are probably perfectly valid, but make my life significantly more difficult.

YAP was that many early extensions did not use chrome.manifest. And some newer ones don’t. So to look up locale information, they were either in install.rdf or contents.rdf. This makes me (and by extension, kittens and baby Jesus) sad. I don’t have a fix for this yet.

But enough about problems, what about SUCCESS!?

Ok. So I managed to get a local copy of every extension that is on AMO. Since parsing an analyzing and writing to persistent storage takes a long time, I decided to save myself some trouble and just do the first 2500 extensions (out of the ~7K folders that I have).

Of the 2500 ‘extensions numbers’, 1630 where successfully analyzed. This is mainly because extension numbers don’t increment perfectly (eg. there is no addon #1. The first one starts at #4. Only about 100 addons failed to parse, giving me a success rate of 94%. Some extensions had quirks in them (eg. bad RDF) that were either invalid or I couldn’t figure them out.

Out of the 1630 extensions, this is what xulrunner-like applications they supported :

And Here are the approximate numbers :

This looks ok so far. One expects a few non-Firefox extensions. The Thunderbird numbers seem a little low. Reminder that this is only ~33% of the total addons.

Locales seem to be a bigger mess, as there are many early extensions that don’t use chrome.manifest, so I decided to skip it, but now realize I have to fix it. Out of 1630 addons, only 464 addons had chrome.manifest files that I was able to read. But here is the breakdown anyways :

Number of locales : 173 (en, en-US, en-GB are all considered different locales). There are some invalid locales. For example, Xultris has an invalid locale called xultrisLocale. This can be fixed with a regex expression, but anyways.

Some locales will have 0 supported extensions. This is because We are only counting the most up-to-date extension, and not counting previous versions which may have supported that locale. While doing a graph for each locale would be unwise, a much wiser choice would be to break it down into language.

So which languages are best supported?

And here is the obligatory graph for those numerically challenged by high school mathematics teachers.

So what does this lead to? First I need to fix locales. We need to get the vast majority of them. Next, I want to profile all the extensions and not just the first 2500. And then, I want to start looking at web crawlers and learning how to crawl a simple website before unleashing a monster on AMO.