Closer to the Metal – How Low-level Work Can Make You a Better Data Scientist

Last week I had to do a major operation for my computer. Namely, I had to replace the hard drive, as it was failing (regular warnings from the computer’s SMART diagnostics reminded me of the fact). Also, the fact that there wasn’t a single computer shop around that was a) open for business and b) willing to undertake such a task, didn’t help things. So, after waiting for about a month for a new hard disk to arrive by post, I took out my toolbox and started the operation of changing the SSD that my computer had. Naturally, I had backed up all my data beforehand and gotten a USB disk ready with an OS image installed so that I use after the new hard disk was up and running.

I won’t go into detail regarding the unbelievable challenges this process entailed (from stuck screws to a failing USB disk, to archive files that were apparently corrupt and unable to restore their content to the new hard disk). Instead, I’d like to focus on the gist of this whole experience, something that’s by far more relateable than the specifics of my situation. In essence, this whole situation was a “close to the metal” kind of experience, one that was both grounding and educational in a hands-on sense. Planning things is fairly easy but executing the plan and improvising alternative routes due to unforeseen (and possibly unforeseeable) circumstances is something we can all learn about. For example, at one point I had to find a different way to get the system running (an alternative USB disk), do a video call with a friend of mine (thanks Matt!) to troubleshoot the issue, and even come up with a contingency for backing up data in the future, so that it’s less prone to issues.

How does all this relate to data science? Well, in data science / AI projects we often have to deal with challenging situation that require us to get out of our comfort zone. We may even need to go to “closer to the metal” territory, e.g. the OS shell, for ETL tasks and such. Also, we may have to re-examine the architecture of the model used (e.g. the number of nodes for each layer, in the case of an ANN), the data used for the training of the model (do we really need all of the variables / data points?), and other factors that we often don’t think about.

Being closer to the metal is not something that concerns programmers only or computer technicians. It’s a state of mind that can come in very handy, even in high-level professions such as ours. Just like a good leader in a company has good relations with every echelon of his organization, even people he doesn’t interact with on a regular basis, a good data scientist ought to do the same. Detachment is useful in problem-solving but let’s not make it our default way of being. Sometimes we need to roll up our sleeves and handle tools we don’t usually use (e.g. the aforementioned screwdriver). With the right attitude, this can be a growing experience. Cheers!