Saturday, December 13, 2014

Don't ever leave Try::Tiny behind

Hit a bug at work a few days ago; it looked as if Try::Tiny was not doing its job of catching an exception, letting it pass right through instead.  The exception was thrown several layers below the try, which itself was also several layers deep, with lots of evals in-between.  It took me quite a while to prune away all the unrelated bits, until the problematic part of the code was pared down to its simplest expression:
try { die } catch { warn };
Huh?  This piece of code should catch the die and emit a warning instead, but here it was, dying on me.  What was going on?

Actually, there's nothing wrong with that code, and it will do exactly what it is meant to do.  Unless you forget the use Try::Tiny.  Then all hell breaks loose.

In most cases, failing to use a module will typically result in a "Undefined subroutine" or "Can't locate object method" error message that will point to your mistake in an obvious manner.  But in this case, the manifestation can be quite puzzling.

At first, I figured that maybe perl was parsing the first block (i.e. the first argument of the try subroutine) as an anonymous hash; since the try prototype hasn't been declared yet, there's no way to know that this argument is actually a block.  The die will then be executed during the evaluation of the try arguments.

Close, but no cigar.  Turns out that perl is indeed parsing it as a block -- just not in the context we expect:
$ perl -MO=Deparse -e 'try { die } catch { warn }'
do {
}->try(do {
Here's the indirect object syntax rearing it ugly head; Perl assumes that try is actually a method to be called on whatever class/object the following block will return.  Yuck.  (Notice that the same also applies to catch.)

(Maybe this will finally push me to break the habit of using that syntax for class methods.  Yeah, I know it's unhealthy, but I can't resist...)

Saturday, November 29, 2014

PC Plus : Des heures de plaisir à s'inscrire

Eu beaucoup de plaisir l'autre jour à m'inscrire au programme PC Plus du Choix du Président (maudit que ça se dit mal).  Le programme lui-même mériterait un autre billet, mais restons-en à l'inscription.
Non.  Sérieux.  On parle d'une entreprise qui doit brasser des millions au Québec.  Pas pouvoir mettre d'accents, c'est comme pas pouvoir donner un numéro de téléphone dans le 418.  Non -- juste, non.

(soupir)  Bon, poursuivons quand même...
Euh, je crois que c'est l'adresse que l'on confirme ici, et non pas un courriel en soi.
Tiens, un autre mot de passe avec une limite arbitraire.  Pourquoi 8?  Bah, pourquoi pas.  (J'admet toutefois que c'est bien assez pour protéger une simple liste d'épicerie.)
Oh, wow!  Comme un magazine en papier -- ça prend un certain temps pour imprimer les étiquettes, j'imagine.
Uh-huh.  (À noter que leur captcha n'est que du texte noir sur un fond rouge.  Probablement plus difficile à lire pour un humain que pour un robot.)

Euh, "accéder" à mes données?  Elles sont directement dans le formulaire que j'essaie de vous soumettre, mes amis.

(C'est comme ça, leur site jamme de temps à autre.  Un rien comparé à leur application mobile.)

Bon, peut-être que l'inscription a passé quand même, et que c'est simplement le login automatique qui a planté.  Vérifions en demandant notre mot de passe par courriel.
Je vous assure qu'il s'agit d'une adresse courriel parfaitement valide; elle ne figure simplement pas dans vos dossiers.  (Et pas sûr que ça donnerait grand chose de réessayer.)

Heureusement, leur système a fini par prendre du mieux, et j'ai enfin pu m'inscrire et loader ma carte.
Tiens, une offre pour le beurre.  N'importe quel beurre?
Ça a le mérite d'être clair.

Sunday, November 2, 2014

Those goddamn copy-protected password fields

Fuck you, PayPal, for deliberately preventing me from pasting in the password field.  These stupid "security" measures actually entice me to choose a shorter, simpler password, because who wants to painstakingly type 20 gibberish characters one by one -- twice?

I take solace in the fact that my frustration is shared by many other people.  (Here's a much more eloquent rant on this matter.)

Friday, October 31, 2014

CrashPlan: Our watch is stuck on 2003

CrashPlan has this cool feature where it watches for new and changed files (and directories) in real-time; it immediately notices when there's something new, and schedules it for the next backup.

On Linux, this is done with inotify.  This usually requires a little bit of fiddling with sysctl(8), since the default maximum number of watches allowed per user is typically much lower than the number of entries in the backup file selection.

However, you may find that this feature still cannot be enabled, no matter how high you raise that maximum number.  Turns out that CrashPlan first checks the kernel version for inotify support (which was introduced in 2.6.13), and cannot parse a version number with only two components.

Guys, seriously?  Seriously?  It's been more than three years since 3.0 was released.  This should have been fixed a long time ago, and I most certainly shouldn't have to write a goddamn uname library wrapper to make it work.


That being said -- once you've finally got it working, it's really cool, and well worth the effort.

Thursday, October 30, 2014

CrashPlan: Don't forget the umlaut

Good thing that I tested a full restore from my new CrashPlan backup, as I found that something was missing: all filenames containing non-ASCII characters were omitted from the backup!

It turns out that Java is to blame -- at least in part.  Filenames are, after all, strings, and Java treats them as such; any filename returned by a system call (as an array of bytes) is decoded into a String object (as an array of code points) based on the character encoding of the current locale.  The same goes in reverse: any String filename passed as argument to a system call is encoded back.  If all goes well, both operations should be exact opposites, and cancel each other; the string we give to open(2) should be byte-for-byte identical to the one we got from readdir(3).

If, however, the filename is not properly encoded accordingly with the current locale, it may contain sequences of bytes which are invalid, and cannot be converted into a code point.  (This is typically the case with ISO-8859-1 filenames under a UTF-8 locale.)  In that case, the Unicode replacement character (U+FFFD) is used instead -- that's what it's for, after all.  Consequently, the re-encoded filename will not be identical to the original, and will refer to a (most probably) inexistant file with a weird name.  (The effects can be perplexing at first, such as listed files not really existing.)

If the C locale is in effect (typically because $LANG -- or $LC_ALL or $LC_CTYPE -- was explicitly set to "C", or left undefined, either of which can often be the case for init scripts, or when using sudo), then only ASCII characters are allowed; any filename with non-ASCII characters (be it encoded with UTF-8 or ISO-8859-1) will definitely not work.

CrashPlan actually accounts for all of this, and makes sure to set $LANG to "en_US.UTF-8" if it was previously undefined.  (It also enforces UTF-8 as the current codeset.  If your filesystem is still using a legacy encoding, welcome to the 21st century.)  This ensures that UTF-8 filenames will be properly handled.  Assuming, of course, that en_US.UTF-8 is a valid locale.

That's the catch: on a Debian system, locales are not installed as-is, but rather generated on demand (to save space).  And it's quite possible for en_US.UTF-8 to not have been generated, if another UTF-8 locale is being used in its stead.  In that case, failure to set $LANG will result in an invalid locale, falling back to the C locale, under which non-ASCII filenames cannot be handled properly.

CrashPlan's fault in all this is quite simple: it does not appear to output any error or warning message in this situation.  Seems like a serious oversight to me.

Setting $LANG to the proper locale in bin/run.conf would do the trick, but according to Code42, this file will be overwritten when upgrading to a new version.  (And unlike that other bug which prevents the client from launching, this one could easily go unnoticed if reintroduced.)  It's probably best to play it safe, and just generate the damn US locale.

Problem solved.

Saturday, October 25, 2014

CrashPlan: Kicking the tires

After a lot of research and reading, I'm pretty much sold on CrashPlan.

I'm currently in the process of uploading my /home partition (only 1.9 days to go!) as part of their free trial.  (Kudos to them for not putting any cap or limit -- you can try it out as much as you want.)  I had heard reports of issues with their upload/download speed, but it's all going as smooth as butter over here.  Once I've run a successful restore dry-run, I'll be another happy customer.

My only non-encryption-related issue so far is that there doesn't seem to be an easy way to remove a single file (or folder) that has already been deleted.  (Being deleted, it no longer shows up in the File Selection list, and therefore cannot be deselected.)  Of course, with the lack of any quota, this is not that much of an issue, but it's still bugging me a bit.

(My thanks to Nelson Minar for his tips on increasing the inotify limit and turning off inbound backups.)

Monday, October 13, 2014

Another day, another segfault

$ CrashPlanDesktop
$ tail -n 18 /opt/crashplan/log/ui_output.log
# A fatal error has been detected by the Java Runtime Environment:
#  SIGSEGV (0xb) at pc=0xc981c81d, pid=17363, tid=4136905536

I'm starting to believe there's a curse on me...

UPDATE: Well, I'll be damned; adding -Dorg.eclipse.swt.browser.DefaultType=mozilla does work.  Apparently, this is an eclipse bug -- and here I thought that eclipse was merely an IDE.

UPDATE 2: This issue is actually documented on CrashPlan's website.  I guess I didn't look for it hard enough.

Sunday, October 12, 2014

Reinventing the OfflineIMAP wheel

I just realized that by attempting to hack IDLE support around mbsync, I was basically reinventing OfflineIMAP.  Hurray me.

(I had actually considered OfflineIMAP when I was initially looking for an IMAP sync-er, but most of the comments out there painted it as a clunkier, buggier, unmaintained alternative.  After taking a second look, this doesn't seem to be the case, at least not in the current version.  I guess I'll have to give it a try and see for myself.)

No fate but what we make

If I'd only taken five minutes to look at the damn code, instead of spending the whole evening poking at it with gdb, I would've easily found the missing comma that was causing the segfault from my previous post.  (sigh)

Wednesday, October 8, 2014

SELECT ... FOR UPDATE on absent rows

Finally tracked down today at work the source of a year-old bug that was causing (rare) intermittent MySQL deadlocks:
(0 rows returned)
Huh?  How can this deadlock -- didn't I just get a write lock before?  If not on the record itself (which didn't exist), then at least on the gap where it would be inserted, right?

Turns out that MySQL/InnoDB doesn't acquire an exclusive lock in this case.  It will get a shared lock (on something), though, preventing any concurrent INSERT for that row, but making it possible to deadlock when INSERT requests the proper exclusive lock it requires.  Hilarity ensues.

(Despite comments to the contrary in the bug report, I can reproduce this for any value, large or small.)

(Update: This apparently varies from one DBMS to another.)

Tuesday, October 7, 2014

I just can't escape fate

I've spent way too much time futzing aroung with programming and debugging these past few days.  I really need to settle down for a while, and take care of all the things that I've put aside and are now piling up.  Like, say, my monthly accounting.
$ gnucash
Segmentation fault

(Update:  This turned out to be even more complicated than I thought, involving GCC; filed Debian bug #764510.)

Monday, October 6, 2014

Sicker Happier

Somehow, "I've been feeling under the weather" has turned into "let's copy all my mail under IMAP, switching from my SpamAssassin setup to my provider's, replacing most of procmail with Sieve scripts, fetchmail with mbsync, and converting what little remains to maildrop, skipping Postfix entirely".

And then, "not sleeping enough and feeling much worse" morphed into "instead of running mbsync every 30 seconds, let's write a multi-threaded Python script that IDLEs on each mailbox".

This is fucked up.

(The SpamAssassin switch was worth it, though.)

Tuesday, September 30, 2014

Every time you lie... Git kills a kitten

Just wasted over two hours of my life over the fact that Git no longer supports $GIT_CONFIG.  Except when it FUCKING PRETENDS TO.

(I'll skip filing a bug report for the moment -- otherwise it would probably only be a stream of swear words.)

(Followup:  I filed Debian bug#763712, and was amazed at how quickly the dev team reacted.  Kudos!)

Monday, September 29, 2014

An interface by any other name

If you're looking to rename a network interface, you've probably come across this udev rule:
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:00:00:00:00:00", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="foo"
And if you're like me, you won't be content to blindly copy this rule without understanding what it all means.  What are all those constants for?  Why can't we go for the simpler example shown in Writing udev rules:
KERNEL=="eth*", ATTR{address}=="00:00:00:00:00:00", NAME="foo"
What's wrong with that?

Well, I don't know if it could be said that there's anything wrong with that rule (other than the fact that it is uselessly invoked for any other action), but it does lack precision; if there ever is, say, an "ethical" device that happens to have the same address attribute (maybe it is attached to the Ethernet port and inherited the attribute), it would also match that rule.  Probably unlikely, but you never know.

Here's the first rule again, deconstructed:
  • ACTION=="add"
    This shouldn't need any explanation. :)
  • DRIVERS=="?*"
    Ensures that there is a driver for this device (i.e. it is a physical device, not a virtual one).
  • SUBSYSTEM=="net"
    Provides a context for the following attributes (since there could be other subsystems with the same attribute names).
  • ATTR{type}=="1"
    Ethernet hardware type (defined in if_arp.h)
  • ATTR{address}=="00:00:00:00:00:00"
    MAC address, obviously :)
  • ATTR{dev_id}=="0x0"
    In case there are multiple devices with the same MAC address, this matches the first one.
(You can get more information about the various attributes for a net device.)

I think that with all of the above, the KERNEL match is superfluous; although I guess it could be used to skip any interface that has already been renamed to something other than "eth*".

And now I can rest, contented.  :)

Not all USB flash drives are removable

I had assumed that all USB flash drives had a removable sysfs attribute of 1 (they are, after all, "removable"), but this is not always true.  (Apparently, removable was meant for floppy drives and such, where the media is removable.)

Curiosity got the best of me, and I tried to figure out where that value was coming from.  Google suggests that this is a USB property, but lsusb shows nothing to that effect.  Searching the kernel code didn't help much; the end result comes from the GENHD_FL_REMOVABLE gendisk flag, set by sd.c from the struct scsi_device removable flag, set by, erm, someone.

To make things more confusing, there is another removable sysfs attribute for USB devices, supposedly "inferred from a combination of hub descriptor bits and platform-specific data such as ACPI", whose description sounds like what I wanted in the first place, but which always returns unknown for me.

That was an hour well wasted.

Sunday, September 28, 2014

$? is a very fragile thing


foo() { return 42; }

while ! foo; do
    if [ $? -eq 42 ]; then
        echo "foo() has indeed returned 42"
        # This should exit with a status of 42, right?
        exit $?
It should've dawned on me that there's a glaring bug in there: the return status of the echo command will overwrite the value of $?.

It took me some more time to realize that there are two bugs in there: ! is also a command, and its return status will also overwrite $?.

It took me much longer to realize that there are three bugs in there: [ is also a command, and yadda yadda yadda.

God I get tired of this shit sometimes...

Saturday, September 27, 2014

The Five Ws (and a few more letters)

More information than I'd ever thought possible about the multiple ways of knowing if/where a command is available from a shell.  My head is still spinning.

As thick as a phonebook

$ cat .
cat: .: Is a directory

$ perl -pe '' .

Turns out you can open(2) a directory just fine, so Perl doesn't bat an eye over that.  It's only the subsequent read(2) that will fail, indicated by readline (or read, or sysread) returning undef, which you're supposed to check for.  Yeah, like anyone actually checks for these things.

The worst part is that autodie won't save your ass in this situation:
use autodie;
use warnings;

open FH, "<", ".";
1 while <FH>;
close FH;

print "Uncaught: $!\n";
Apparently, not much can be done about this.  :(

Letter to cgmanager: You have to learn to let go

This is the second time that I'm unable to unmount an otherwise non-busy block device, only to find out (after killing nearly every process, forcing me to reboot anyway) that the culprit is cgmanager.  This is quite annoying.

(Nobody else on the interwebs seems to have this problem.  Oh well, at least I'll know who I should yell at next time.)


Older, wiser programmers will have figured out that the bug in my previous post was due to an uncaught SIGPIPE.  I'd completely forgotten about those.  :)

The shifting nature of the bug (which is what really had me confused) was due to a race condition between the parent process writing to the pipe, and the child closing it (by exiting).  Here's an overly simplified version of what happens in both processes after the clone():
/* parent - writer */

/* child - reader */
execve("/bin/false", ...)
If the parent attempts to write to the pipe after it has been closed on the other side by the child's exit(), then a SIGPIPE obviously occurs.  However, since our simple string is small enough to fit in a pipe's buffer, the parent may very well get the chance to close the pipe before the child.  At this point, the child's close() will simply discard any data in the pipe's buffer and exit; its exit status will then be returned by the parent's waitpid(), stored in $?, and cause Perl's close() to return a false value.

The script example given in the previous post is simple enough (without autodie) for the parent to get there first.  Adding autodie then introduces just enough complexity for the parent to take a little bit more time, giving the child a chance to exit first.  (Even using strace is enough to influence the result, making this a true heisenbug.)

Note that in the parent-closes-first case, the failure has nothing to do with the pipe itself (hence why $! is left empty), but is simply due to the child returning a non-zero exit status.  (Thus, replacing false with true would make the script fail some times, but not always.  Now there's a real head-scratcher.)

Thursday, September 25, 2014


This one had me stumped for a while:

use 5.014;
use autodie;

open FALSE, "|-", "false";
print FALSE "Hello world!";
say "Closing filehandle:";
close FALSE or die "close() doesn't die...";
say "...but doesn't succeed either";
What made it even confusing was that commenting out autodie allowed close to at least (silently) fail and return a false value (without even an error message in $! as would be expected).  I used autodie to catch any unchecked errors, dammit, not to hide them even further!

I guess that's what I get for living a life sheltered away from the raw, bare-metal, non-child-proof world of C programming.  :)

I Can Haz Triplequotes?

Writing a udev rule:
[...] PROGRAM="/bin/sh -c '... something-that-must-be-quoted ...'"
Drat.  :(

Wednesday, September 24, 2014

Google+ and X-Sender-ID

I just realized today that although Google+ emails come from a non-descript "noreply" address, they include an X-Sender-ID header with the info I needed.  With that and a procmail recipe, I'm all set!  (I've had so very few moments recently where things just work, so that was appreciated.)

Tuesday, September 23, 2014

Snatching the root device from grub-install

# grub-install --debug --boot-directory /media/usb/boot /dev/sdh
grub-install: info: guessed root_dev `hostdisk//dev/sdh' from dir `/media/usb0/boot/grub/i386-pc'.
grub-install: info: setting the root device to `hostdisk//dev/sdh,msdos1'.
Goddamit, GRUB!  This is not what you've just set as root device -- core.img cannot possibly know what /dev/sdh means.

I can't believe I have to resort to strace to get the right answer:
write(2, "grub-mkimage --directory '/usr/lib/grub/i386-pc' --prefix '(,msdos1)/boot/grub' --output '/media/usb0/boot/grub/i386-pc/core.img' --format 'i386-pc' --compression 'auto'  'ext2' 'part_msdos' 'biosdisk' \n", 203) = 203
There: "(,msdos1)" is the actual root device.  Was that too much to ask?

A nicer alternative to init=/bin/sh

My thanks to the Gentoo guys for showing succintly how to solve the whole "/bin/sh: can't access tty; job control turned off" problem when booting directly into a shell.  With that and a call to /etc/init.d/rc, I got myself a simple script to automatically login as root into runlevel S and stay there permanently:
/etc/init.d/rc S

while true; do
    setsid sh -c 'exec login -f root </dev/tty1 >/dev/tty1 2>&1'

Friday, August 22, 2014

Unpacking the last ext3 box

Finally converted my last "home" partition to ext4.  The time it takes to tar it all up  (350 GiB) has dropped from 3.5 hours to 70 minutes; a full fsck, from 2.5 minutes to 10 seconds.  Quite happy.

(I was a bit surprised, as these are mostly big files; I expected ext4 to have less of an impact here than with a partition full of little files, scattered all around.  Shows how little I know about this stuff.)

Thank you LVM

I still can't get over how cool it is to be able to move an entire mounted partition from one drive to another, live, without missing a beat.  (Okay, so it stutters a little bit, obviously.)

Thursday, August 14, 2014

T is for Title

(Finally noticed the quite obvious "Post title" input field, right there above.  After a mere several days.  I expect to have fully mastered this whole blogging thing by 2038.)

Linux capabilities are here

My thanks to Craig Small for making me aware of how simple Linux capabilities are.  I remember hearing about them from time to time, but they usually got mingled with SELinux in my mind, and I assumed they were just as complicated and troublesome.  Little did I know...
 $ ls -l /bin/ping
-rwxr-xr-x 1 root root 44104 Jun 18 17:37 /bin/ping
Look Ma, no SUID!  I've been using them all this time.  :)

One thing escaped me, though: are capabilities set up at boot time (like sysctl), or are they stored on the filesystem, like the old SUID?  Turns out they are indeed stored, as attributes:
$ lsattr /bin/ping
-------------e-- /bin/ping
Erm, I mean, extended attributes:
 $ getfattr /bin/ping
Oops, getfattr(1) only displays user attributes by default.  My bad:
$ getfattr -m '-' /bin/ping
# file: bin/ping
Oops, getfattr only displays the list of attribute names by default.  My bad:
$ getfattr -d -m '-' /bin/ping
# file: bin/ping
Huh.  This kinda looks like base64, doesn't it?  Ah, the manpage doesn't really say which encoding is chosen by default.  Let's give this another try:
$ getfattr -d -e hex -m '-' /bin/ping
# file: bin/ping
There we go.

Wednesday, August 13, 2014

It's 1975 all over again

This one had me confused for a moment:
$ date
Wed Aug 13 14:34:53 EDT 2014
$ # The join() is only for cosmetic reasons
$ join " " => localtime
6 36 14 13 7 114 3 224 1
$ # Let's grab only the HH:MM part (items 2 and 1)
$ my ($h, $m) = localtime[2,1]
$VAR1 = 56;
$VAR2 = 44;

Oh, I see: I forgot my parentheses.
$ ($h, $m) = (localtime)[2,1]
$VAR1 = 14;
$VAR2 = 41;
Much better.  But where the hell did the first set of values come from?
$ join " " => localtime[2,1]
56 31 8 17 10 75 1 320 0
$  scalar localtime[2,1]
Mon Mar 10 02:35:04 1975
Dang, I forgot that localtime() can take an argument, a Unix time number (just like localtime(3) takes a time_t as argument).  This is all starting to make sense.
$ [2,1] . ''
$ [2,1] + 0
$ scalar localtime 164751992
Sat Mar 22 15:26:32 1975
(I'm still a bit surprised that Perl didn't say a word about this.)

Tuesday, August 12, 2014

Dear Time::Piece,

Dear Time::Piece,
$ use Time::Piece

$ my $now = Time::Piece->new->strftime("%F %T")
2014-08-12 17:36:04
$ Time::Piece->strptime($now, "%F %T")
Runtime error: Error parsing time at /usr/lib/perl/5.18/Time/ line 469.
I hate you.

(more info)

Dear Postfix,

Dear Postfix,

If I enable debugging with these debug options:
Am I crazy to assume that these debug messages will be sent to syslog with the debug priority?

Dear syslog,

Dear syslog,

If I SIGHUP you, telling you to reload your configuration, and you reply this:
Aug 12 17:18:15 hostname rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="30944" x-info=""] rsyslogd was HUPed, type 'lightweight'.
Am I crazy to assume that you actually, for realz, reloaded your configuration?

Sunday, August 10, 2014

Lies I've repeatedly told myself these past few weeks:
  • "This should be easy."
  • "There's probably a feature that will do the trick."
  • "It will only take a few minutes to implement."
  • "This feature works exactly as documented."
  • "I should be just about done in a few more minutes."
  • "Surely, it can't be that difficult."
  • "This feature is documented."
  • "Only five more minutes and I'm off to bed."
 If only I could say that this was the last time...

Unable to locate package policy

  $ apt-cache show policy
  N: Unable to locate package policy
  E: No packages found

What do you mean, APT, you can't locate my package policy?  Did I screw up /etc/apt/preferences somehow?

How strange, I can't even find another occurrence of this error message on the interwebs...

(scratches head)

Oh, you can't find a package named "policy".  Erm, yeah, I don't know where I got the idea that this was a valid command either.  Sorry, that one's on me.

Saturday, August 9, 2014

Heard many good things about this new newfangled "blogging" thing that's all the rage among hipster kids these days.  I thought I'd try it out and see what the fuss is all about, before the media latch into this and it goes mainstream.