Home Download Learn more Tech Help

Tiger Envelopes: A lesson learned about testing

There's an old myth in software development that if you're good enough, you can write code that works right the first time. That may occasionally be true for small pieces of code under ideal circumstances, but not in the real world. During beta testing we got a hard and embarrassing lesson about this.

In version 0.7.7 we thought that when the user's mail client originally used an SSL connection to a remote mail server, then the Tiger Envelopes proxy also used an SSL connection to that server. This turned out to be true for IMAP, but not SMTP or POP servers.

We had automated testing using POP and SMTP so we didn't test them as much by hand. DixieMail, our automated testing portable mail server, didn't support IMAP, so we relied on hand testing for IMAP. But Dixie didn't support SSL either, so the automated testing didn't test SSL. We knew that IMAP supported SSL because we tested it, by hand. We assumed that the other protocols did too. They didn't.

Yes, this was just during beta testing, we got out a fix fast, the final release was a long way off, and most importantly Tiger correctly failed to connect. No one's mail was exposed by this bug. But excuses don't protect users.

"If you fail to test, it will fail to work."