Back-up and recovery with duplicity

A number of months ago I made the decision to start making back-ups of my data. There was no event that caused this to happen, but rather a conversation with a friend that convinced me it was a good idea. So I purchased a USB drive and used TrueCrypt to encrypt the drive (this actually wasn’t a super-great idea since I now have to manually mount the drive before I can do a back-up).

For software, I used deja-dup which comes standard with Ubuntu. At the time I used Ubuntu on my desktop and laptop. It was a very easy to use, so kudos to the deja-dup team. At some point I updated my laptop to use Xubuntu and decided against using deja-dup because it had some gnome dependencies (though looking at it now it doesn’t seem to have many dependencies now, but there are still some gnome ones). I know deja-dup was basically a front-end for duplicity for managing back-ups, so I just learned the command-line interface.

It actually turned out to be pretty simple. I will paste the command and what each flag means:

duplicity incremental
--full-if-older-than 1M
--no-encryption
--log-file /tmp/dup.log
--verbosity info
--ssh-askpass
--exclude-globbing-filelist="$HOME/.duplicity-file-list"
$HOME
ssh://me@someserver//backup/location > /tmp/dup2.log
argument purpose
incremental do an incremental back-up. This saved a ton of time because only files that changed will get backed-up
–full-if-older-than 1M This is set to 1 month. This creates a full backup every month. I do not recall why I chose 1 month.
–no-encryption The drive is encrypted so I don’t need to encrypt the files
–verbosity info This was used mainly for debugging and making sure the files I wanted backed-up was happening
–ssh-askpass prompt for the ssh password. Typically I leave the drive attached to my desktop, so I have to ssh to the desktop to perform the back-up
–exclude-globbing-filelist=”$HOME/.duplicity-file-list” in my home directory I have a hidden file that contains the contents:

+ /home/me/.ssh
- /home/me/Downloads
...

This tells duplicity to back-up the .ssh directory, but do not back-up the Downloads directory

$HOME back-up my home directory
ssh://me@someserver//backup/location > /tmp/dup2.log the location of the back-up directory. I also redirect the log file to the temp directory.

This has been going on for several months and thankfully I didn’t need it until a few days ago when a botched Ubuntu 13 upgrade (PS. Never fall asleep during an upgrade) gave me a nice “General error mounting filesystem” boot-up message. While annoying, it did at least give me an opportunity to try Linux Mint.

But Linux Mint had no option to save my files. It was going to remove all my documents and saved files as part of the installation process. I hesitated for a bit. While I did have back-ups, I think I only done recoveries on a small scale and a long time ago. I decided to go for it anyways. The installation was quick and flawless, but it was time to do a recovery. I checked to see whether Mint had deja-dup and they did not. The had their own tool for backup. So I opted to do it over the command-line. It actually wasn’t that bad. The command for that was:

duplicity
--file-to-restore home/me/Documents
file:///backup/location
/home/me/Documents
argument purpose
–file-to-restore home/me/Documents The file/folder you want to restore. Note that this does not have the leading /
file:///backup/location Location of my back-ups. I used file:// URI because this was on my desktop so I did not ssh to anywhere
/home/me/Documents The location I want the back-up to

This actually went pretty smoothly. The only issue I hit was a few files that had long filenames and duplicity failed to recover those. I have not found a solution to this problem yet.

The duplicity man page is quite nice. Lot’s of information, but you can find what you’re looking for quite easily.

Aside

I wonder if the number of car accidents increase during heat waves. Today I saw:

  1. Traveling southbound on Spadina, I saw a cyclist run a red light going eastbound on Adelaide – while it was still green for southbound traffic. The car next to me had to break, and so did a few cars going northbound. God forbid that cyclist hits anything other than a car, he could seriously hurt someone
  2. Going back home, a taxi was slowly coming up to an intersection on a green light and I was coming up behind him. “Is going to make a right? He’s not signaling for it. *speeds up*…”. Hard breaking follows as he was indeed making a right
  3. Again going back home I saw a cyclist berate some woman in a car. I didn’t see what occurred, but he was yelling that she almost killed him and that he had the right-of-way. He then proceeded to run a red light

Aside

I saw this on my twitter stream: http://www.ismytwitterpasswordsecure.com/

After a good laugh, it came back to me that passwords are pretty crummy. Then an idea struck – wouldn’t it be great if there was no password?

Humour me. Let’s say for example you want to sign up for Twitter. You click “sign up” and you pick your twitter handle, give them some personal information, follow a few people (this wasn’t required when I started, but apparently now it is). And then you are done. No password to enter, no password to remember.

When you visit the twitter page, you are signed in. When you log out, your browser asks you whether you want to sign in.

Not talking about an external service. Imagine a webpage doing something like :

if (navigator.hasCredentials)
{
        issueAPOSTRequest(navigator.username,
                          navigator.password);
}

or

var password = navigator.createPassword();
setHiddenPasswordInput(password);

There are some caveats. Accessing multiple accounts might be trickey. Also sharing your browser might not be a great idea. API obviously not final, nor well thought out, nor given any security considerations. This idea is in draft only and may never be implemented. There is no warranty of any kind, neither expressed nor implied…

Show more!

One of the issues I hit on beatsearch is that on the Artist page, some artists have long biographies. Really long.

Screenshot of an artist's bio that rambles on

This actually exposes two bugs. The first being the lack of line breaks. The second being, I don’t want to show that much text. This post is about the latter.

There are many different solutions to this rather common problem. Most implementations will stop after a certain number of characters and provide a link for more. I wanted to do something a bit different and instead stop after a certain number of lines. 3 lines to be exact. Oh, and this has to work regardless of whether you are on a mobile device or on a big monitor.

Fun! Especially when it has to work across multiple browsers!

This was actually pretty easy to do. The line-height CSS property was used to figure out the height:

// #biography is the HTML paragraph tag that contains
// the artist's biography
$('#biography').css('line-height');
// 22.4px

Great! Now I can set the height of the paragraph to 3* the line height

$('#biography').css('height', parseFloat("22.4px") * 3);

And cause text that goes beyond that height to be hidden:

$('#biography').css('overflow', 'hidden');

Which gives you this result:

The first 3 lines of an artist's biography shown

Lastly we have to put in a link in the bottom-right corner so that the user can view the full biography if they wish. This is pretty simple as well thanks to absolute positioning. We can make the biography element relative, which makes it a containing block, and inside the biography element put in an absolute positioned tag. You can read more about containing block and positioning from this helpful w3c article.

$('#biography').css('position', 'relative');
// Insert a span tag
var span = $("<span></span>")
           .text('show more')
           .css('position', 'absolute')
           .css('bottom', "0px")
           .css('right', '0px')
           .css('background', 'white');
$('#biography').append(span);

The code should be fairly self-explanatory. We position the text 0 pixels from the right and bottom. We need to set the background colour to white so that we don’t show overlapping text to the user. This is probably the one thing I don’t like about this method is setting the background color.

The big benefit over cutting off after x number of characters is that it works regardless of the width of the biography. On a desktop maybe 70 words can fit in 3 lines but on a mobile only 30 words.

Things are a little complicated in Internet Explorer because line-height may not return the height in pixels but rather a unitless number. This is completely valid, just not very helpful. You can get an idea of how to solve it by viewing the fix.

Last thing to do is to set a click handler to that span that removes the span and removes the styles we added to the biography element. But I’ll leave that as an exercise to the user.

Beatsearch

A few weeks ago I needed an escape project to get my mind off some things, and then I remembered something I could do. A while back I worked on an HTML5 version of Beatport.com. While I didn’t finish a bunch of things I had planned for it, one area that it does excel over the official site is that it loaded much faster so it was more pleasant to search.

Since then, Beatport’s public API has CORS support, which means I can completely remove the server side component and implement it entire in JavaScript and web technologies!

I got to work with a bunch of neat technologies in the process. Most of which I already knew :

  • Foundation CSS framework. The site looks pretty snazzy in mobile as well, but it’s not mobile friendly. More on that later
  • Mustache.js
  • underscore.js and jQuery. Though there is a lot of overlap, and most of the utilities I used in underscore are available in jQuery as well.
  • require.js
  • q.js

I learned a bunch of small things. Such as:

  • Foundation was pretty painless. A weird feature of its grid system (they wrote CSS for this, so it’s likely not a bug) is that two rows in succession causes the inner row to go outside the bounds of the parent row. For example:
    <div class=row>
        <div class=row>
            This row's left and right width will be 15px larger than it's parent.
        </div>
    </div>
    
  • I found myself fighting with mustache.js. It’s logic-less (mostly) but that presents a number of limitations that you have to hack around. Tom is smarter than me
  • Micro-templating! Putting large amount of HTML in your JavaScript felt wrong. Another idea I thought of was to move the HTML to a separate file and load it using AJAX, but micro-templating was just a better idea
  • Foundation is awesome. Just a reminder. I don’t think it’s that much different than bootstrap, but these CSS frameworks are the best thing since sliced bread
  • underscore was ok, but found most of the functionality present in jQuery. One annoying aspect of underscore is that they don’t support require.js AMD support. It was an easy fix however
  • Having a separate development/production version of javascript/css files I found to be a mess that I don’t think anyone has solved yet – at least for client-side only code. I took a stab at it by having a bash script make symbolic links, but it has shortcomings that doesn’t really work very well.
  • While the foundation framework makes it a breeze to create a fluid responsive design, beatsearch is a fairly image heavy site so there are lots and lots of requests. It’s not optimized for high latency connections. Maybe if it’s worth the effort, having a placeholder image could be done instead and the phone will just hit the cache.
  • I didn’t plan for this but it was too tempting not to, but I ended up using jplayer to play beatport’s sample audio files. It was really easy to use, and now I can play tracks! Not able to set-up playlists yet, but that’s a future option.

Hold crap you made it this far. Might as well check beatsearch out.

Ending support for Safe 1.1.2

My safe extension hasn’t changed much for several months. It’s been happy doing it’s thing and AMO has been happy bumping compatibility updates.

Why would I want to interfere with such a perfect system?

One thing that has impressed me about Firebug is that it is a restartless add-on. It is not anything new. But the ease of installing/uninstalling is just wonderful. If Firebug can do it, Safe should also be able to do it.

After a few days worth of work, I finally have Safe 2.0.0 working as a restartless add-on. Great. The next question is – what Firefox version is it compatible with?

The obvious answer was, of course, whatever version I was using. In theory, I could test with all versions of Firefox until Firefox 4 when restartless add-ons was available. But I don’t have that kind of time.

Then I realized during this rewrite (it was a fair amount of work to make an extension restartless) that I used a bunch of relatively new features that was released in Firefox 10. I could do the appropriate workarounds but do I really have to?

I went to my add-ons statistics dashboard and found that about 78% of my users were using the latest release of Firefox. If I go back 2 additional release versions, that increases to ~85%. There is a really long tail here…

My extension is also available for SeaMonkey, for which I only have between 0-3 users :( I haven’t tested my code with SeaMonkey, but if it’s cheap to support than I rather support it because SeaMonkey is great browser.

So it looks like my time is best spent checking that my extension works for the latest release of Firefox on Windows/Linux and Mac if I had it. I am really glad I took a step back and evaluated the best decision rather than falling into my usual trap of mismanaging time/effort.

JavaScript funk

There are two reasons why I hate reading other people’s code:

  1. Their code is so much prettier and better organized than mine
  2. It sometimes leaves my head scratching

While reading about restart-less add-ons so I can make my safe add-on be restartless as well, I came across some example code by Edward Lee that looked like this:

const {classes: Cc, interfaces: Ci, manager: Cm, utils: Cu} = Components;

When writing Firefox extensions, by convention Cc and Ci are shortcuts to Components.classes and Components.interfaces. I was able to piece together what it was tring to do (eventually), but I could not understand why this was working.

In my feeble attempt I looked up ECMA-262 which I think provides the answer but I am not so sure. The SpiderMonkey JavaScript engine seems to be the only one that supports this syntax as well as some examples below. Running these examples produced a syntax error in Chakra on IE and V8 on Chrome using the developer tools.

Here is some other fun things I found:

let [ a, b, c ] = [ 1, 2, 3 ];
c; // 3

It is similar to how you can do multiple assignments in python. :

a,b,c = [ 1, 2, 3 ];
c; // 3

This is a neat shortcut to get variables pointing to some function

let { abs } = Math;
abs(-5); // 5

And going back to Edward’s code, we can assign a different variable name to the function.

let { abs : a } = Math;
a(-14.2); // 14.2

Still wondering whether this a SpiderMonkey (and therefore only Firefox/Seamonkey) extension or whether other JavaScript implementations will support this someday as well.

Q

A number of months ago someone told me about Q which is an implementation of the promises framework. I did not understand the problems it was trying to solve at the time, so I never bothered going to deep into it.

My good friend Tom mentioned promises and when.js during our normal chats which I misheard as WinJS (another promises javascript library, but only available for Windows Store apps). when.js seems like a node thing and not really for a web page (by that I mean it’s multiple files, which is a bit odd for a JavaScript library). If you don’t know what promises are, I suggest visiting other literature that will do a much better job than I to explain it.

So I went back to Q but I still had the problem I had before. The documentation is very weird to me as it starts going into what promises can do but doesn’t explain how to actually make a promise until half-way into the documentation. It’s amazing how well one simple when.js example goes into explaining how promises work. Q eventually does put the pieces together, but the non-sequential structure doesn’t work well for me and I got frustrated.

So here is an example I would use to explain promises. I got a bit all over the place, but that’s because I was trying to get the basics down to one sample

Debug Console:

This message appears first
Max Power blah blah
true
Hello World!