About ten years ago, my company started to send phishing simulation emails to all employees multiple times a year. A few days ago I fell for their trick by clicking a link in such an email that is very well written and claims to provide employee’s year end information. This is embarrassing for a long-time IT professional with skills including web programming, web server admin, mail transfer agent configuration, among others. While watching the video about the danger of phishing, mandatory (I think) for anybody clicking such a link, I decided to summarize all my thoughts on this and related matters.
To err is human. There are two kinds of errors, one due to lack of skills, the other due to accident, carelessness, sloppiness. Let’s talk about the second kind only. These errors are mistakes. And there are two ways to reduce the chance of making them, one by improving the system, the other by enhancing the operator or the person’s alertness. Corporate training on reducing mistakes almost exclusively focuses on the latter, for example, by repeatedly stressing the danger and cost of such mistakes. In 2006 I worked at a big e-commerce company. At such a training session for all employees, the speaker told us that one second of downtime would cause $3000 worth of loss in company profit. Not sure if it’s true, but that’s the only thing I remember, as the rest was the same content everyone would expect in every company. There was no follow-up analysis to show that such trainings did reduce mistakes, although I guess it did, a little bit.
I’ve been an Oracle DBA for over 20 years. To me and probably other DBAs, the biggest horror stories we heard many years ago are about accidentally deleting datafiles, control files, or redo logfiles. Unless you’re on Windows where you can’t delete a file being used, it’s just too easy to rm (remove) a critical file on the file system. Then, by about 2010, most companies had adopted Oracle’s ASM technology, which takes over database file management and eliminates access to these files by OS shell level commands such as rm, cp (copy), mv (move). I don’t know if I’m the only one noticing that in recent years people on various Oracle discussion forums no longer talked about recovering database files accidentally deleted or overwritten. ASM is designed to manage raw files that would otherwise be difficult to manipulate and to improve I/O performance. The benefit of avoiding damaging or deleting database files by taking away the ability to type the short-string commands like rm is surely a pleasant surprise. (ASM does have its own command line interface for special use, where commands such as rm work, but the commands will fail if the files are actively used.) This is a textbook example of reducing mistakes from the system side.
But that’s just Oracle. One former system admin here was said to have deleted a huge chunk of data by mistake, presumably by the infamous ‘rm -rf’, and was let go. In our DBA group, we also manage MySQL and other databases, where the data files are still directly placed on the file system, with no intermediate layer like Oracle ASM. How do we avoid accidentally deleting such files? Harping on the criticality of data to the DBAs can only go so far. Some companies create an alias to rm so that every time you run the rm command it pauses for confirmation before it removes the file. But as you may guess, typing such commands many times a day will numb your sense and you enter ‘y’ to confirm it almost unthinkingly. In view of this ineffectiveness, I proposed adding a file attribute to the file system code so that critical files such as database files can be marked with this attribute and cannot be deleted but can be in-place updated and resized (required by the running database), unlike the immutable attribute, which prevents any kind of change. I posted a message about this idea on the Stack Exchange forum and it was mocked by others and later deleted by an omnipotent admin. Well, fine. Then I suggested a non-technical idea: name a critical file with a special string in the name e.g. ‘NoRm’ and alias rm to a simple shell script that checks for ‘NoRm’ before actual rm. The idea is that if you get a confirmation prompt only occasionally, you as a human do get alerted.
Back to the case of clicking the link in a phishing simulation email. Since the email is sent to us by an external company, probably contracted to do this drill on us, and we’re repeatedly told we need to be very careful when receiving external emails, I went ahead and created an Outlook rule that automatically moves all external emails into a special folder. Hopefully having to make an extra effort of going to that special folder to read new messages (from outsiders) is a good enough reminder that I need to be extra careful.
All the above are tackling the problem from the system perspective. From the personal perspective, I don’t need more verbal or visual reminders of the danger of phishing. But I gave more thought to why I fell for the fault. I remember one day back in 2010, I made a mistake in deleting data in a production database. It was quickly fixed but the incident left a long-lasting impression on me. I still remember I took a short nap before I worked on that ticket. And I recently clicked the mock phishing link also after a short nap, a routine after-lunch nap. Now I think maybe I tend to make mistakes right after taking a nap. With this realization, I decided to add a special name to the cell phone alarm that wakes me up after my lunch break nap. I may do something more, maybe like putting an unusual piece of paper on the laptop before going to nap. Anyway, there’s not too much creative to do from the personal end without running the risk of mental numbness by repeating the same reminder too many times.
It’s not useless to keep reminding users how dangerous phishing is. But its usefulness is quite limited due to human nature, including insensitivity to repeated warnings. On the other hand, there is a lot to be done to reduce this risk from the system perspective. Some require creative thinking and an open mind.
Contact me
To my Miscellaneous Page