Recently, I took on a role as Systems Administrator with focus on Research Support at Oberlin College and Conservatory. It is both a broader and deeper user base than in my days as a GIS manager, but also more focused on just the systems administration side of things. The kind of queries I used to get from IT people “but what does this system do?” I now find myself uttering to others. Usually I understand the answers, but occasionally I wonder how much I really know about quantum calculations for lattice structures in chemistry, or how much I truly understand stem cell differentiation in plants in the absence of true plant structures, but it’s like drinking the ocean: every day I am a little more thirsty and there’s a whole ocean out there.

If one is to get into, say furniture building, there are lots of little lists of the best tools for how to get started in the field, including what tools you need. A few years ago, I think when it was edited by Chris Schultz, Popular Woodworking had a list of such tools: the (X) number minimum number of tools needed for a total of <$5000 for getting started in the furniture making trade. Similar lists exist for guitar making, etc., and some companies even sell toolboxes pre-packed with that minimum. A few years ago, one of my favorite toolmakers, EC Emmerich, had such a kit with all the hand tools needed for a life of furniture making and finish carpentry. I was super tempted to order it, but it was outside the budget at the time.

Sometimes I think about what such a list of tools would look like for various technical disciplines. So, for a digital cartographer, what is a minimum toolkit? For a GIS analyst, what is that minimum toolkit, etc.. With software, it can be a little deceptive of course: what qualifies as a single tool, vs. what is itself a toolkit, e.g. GRASS, ArcGIS whatever, or QGIS are whole ecosystems.

It is an interesting exercise even if it doesn’t have a perfect physical equivalent. Perhaps QGIS or GRASS is just a CNC machine equivalent and the analogy doesn’t fail, nevertheless, we might think about what such a toolkit might be for a Systems Administrator. And just like furniture making, finish carpentry, or lutherie, the list might vary based on specialization.

Ansible in the toolkit

In my previous job, I maintained roughly a dozen servers. The number varied through time and according to purpose. Nevertheless, it was probably never more than two dozen, and rarely less than a half dozen, so the number is a good median to start with. In my current job, I have at least 10 times that many machines, between a couple of computer labs, a number of physical hosts, a few virtualization environments, and a number virtualized hosts. Suddenly the tools of my trade needed more power.

Ansible, Puppet, Chef, and similar orchestration tools have emerged in the last few years to make systems administration more predictable, reliable, and elegant. Very early on, I identified Ansible as the next tool in my toolkit for Systems Administration.

Ansible for Apt

I will show a rudimentary example for today: that of using Ansible to manage apt updates and upgrades on Debian or Ubuntu machines. Plenty of good tutorials on Ansible exist. These are just some limited examples from my current workflow.

Ansible has a bajillion modules for doing a range of different tasks. It is declarative, agentless (it relies on ssh for Linux hosts), and works on a philosophy of idempotency, i.e. the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.

Running updates

In our first case, we have a lab of n-computers. We want to manage them to roughly operate the same, want to push updates, add applications, etc.. We can use the apt module to update the cache, do any safe upgrades and perform an autoremove at the end. The YAML file for doing so is pretty simple:

Pushing new repositories

I had a recent request to upgrade https://racket-lang.org/ on the lab machines. As they are running an older but supported version of Ubuntu, their existing repo for the programming language Racket is a bit out of date. But, we can change this with ansible.builtin.apt_repository and point to a current ppa:

Custom deployments outside APT

Sometimes, there isn’t a repo for the install needed, or for some esoteric reason, we need to do a custom deploy of an application, like a simple jar that just needs a shortcut, or a pre-built binary or similar. In this case, we can use Ansible’s copy to place our files where they need to be, and set the appropriate ownership and permissions for use.

This particular use case required I learn a little about .desktop files and their configuration and placement. It took a little contemplative work and play to get it right, but it was pretty straightforward in the end. And there was truly something magical about deploying this to all the lab machines once it was tested and functional on one.

New hammer: looking for old nails

I really like Ansible as a tool for orchestration. It gives me some very nice, thoughtful, repeatable, and powerful utilities. I like my new hammer: time to interpret the world of systems administration as a bunch of nails for a while… .