The Unbearable Lightness of Cookie Dough

There is a future I can see where every data engineering article has some form of baking-centric metaphor at it's heart, and they all coalesce into an impenetrable boulangerie of insight into something, be it electrons or yeast. A Marvel Cinematic Universe of flour-y metaphors if you will. I am of course referring to Jordan's excellent article Data Engineers Should Be Held To The Same Standards As Bakers, and that today I wish to use cookies as a metaphor to help with making technical choices.

As a point of normalized global vernacular, I believe that all cookies are biscuits but not all biscuits are cookies. Here I am using the American form because the idea started as me making an oblique reference to a line from a classic tv show in a meeting with two fellow Hermits who are probably too young to get it (sorry lol). If you get the reference, type F in the chat.

Making any technical choice involves trade-offs - things you actually care about, and things you should let go of. A highly useful phrase any engineer should have imprinted on their soul is "What are we optimizing for?", which I think is corpo-veneer for "Who do I want to be at the end of this?". Am I the person that wants things done fast? Are my systems stable enough they will outlive me? Whatever it is, there is a cookie metaphor that fits your shape.

There are many cookie shapes, this one is mine

A baker of cookies must care about many different things. An eater of cookies likely does not care about any of them. If you really need a cookie right now and that is your singular concern, go buy one. This is the First Normal Form of cookie enjoyer and will now be dismissed, but I wanted to call it out because this fits the Rent-Buy-Build approach of working with software at organizations. Just buy the damn cookie if you want it that bad, it'll be cheaper in the long run. (We are talking about purchasing a SaaS.)

Now that is said, and we fall deeper into the abyss, buying is off the table and as a cookie enjoyer you find yourself also needing to become a baker of cookies. You now have two hats and some interesting engineering concerns to trade off against.

I am declaring that our platonic ideal of a cookie is a chocolate-chip one. I am not taking questions at this time.

Thus, if we are to manufacture a chocolate-chip cookie then we have a few options open to us which I will structure into some rough categories:

• We buy one of those rolls of prefab cookie dough
• We make the cookie from ingredients
• We become artisans
• Ia! Ia! Cthulhu fhtagn!

Let's jump into it.

Prefab cookie dough

It comes in a foil roll, you slice it into rounds and you bake it at 180 celsius for 20 minutes, and you have what I guess we can call a cookie. Job done.

This is a metaphor for spinning up a docker container that you grabbed from a vendor website. This is quite often the optimal solution. You don't need to know how to make the dough, you want the most efficient path of least-resistance that gets you to your goal of approximately delicious cookie eating.

This makes you the efficient, economical cookie baker. You learn nothing about the craft of dough making, but you ship a cookie real fast and it's good enough. If this is who you want to be in this moment, you have made a very good choice. A lot of engineers will stall at this point because it doesn't feel very engineer-y but it's often the best choice for the problem in front of you, and knowing when you want to be an artisan craftsperson vs when you want to be the person that has cookies in 20 minutes is absolutely vital to learn early, and ponder often.

We move on.

Cookies from ingredients

If you've attempted making cookies from scratch and never thought too deeply about it, this is probably where you sit.

We acquire flour, butter, eggs, chocolate chips, and we follow the recipe. There's not too much thinking involved, but a lot more moving parts than our prefab scenario so we need to stay on our toes. The point is we have additional agency here and can customise the dough to our liking, and we have additional responsibility to get it right because the training wheels are now off.

This metaphor can apply to self-hosting a SaaS by constructing custom infrastructure and having some nice spin-up/tear-down automation. Maybe it is creating a cool dimensional model with some clever underlying abstractions to allow for extra customization. I'm not your mom, you get where I'm going.

This makes you the adaptable problem solver cookie baker. Someone who can see shortcuts and understand that putting in a bit of extra effort now will yield a more desirable result. This is a position of maximal flexibility because you are able to fit the cookie to your specific tastes, and you aren't expending extraneous effort engineering the ultimate cookie. You'll still be waiting 20 minutes, plus your prep time, but at the end of the day you're eating cookies and they're good.

Artisan cookies

As you take your first bite there is satisfying resistance that immediately yields to soft, oozing decadance. You are transported to a scene from your childhood and for one sublime moment you forget that you are a ghost haunting a bone mech clad in the flesh armour, for you have entered an oasis of existential reprieve and you are now one with the cookie.

If this sounds like the cookie for you, buckle up my son (gender neutral), because we are going to need all of the stuff we said above except this time we're being very selective about our flour quality. We've done this enough to know that we weigh all our eggs and bring them to room temperature before doing anything else. Our mise en place is locked in, and nobody is eating cookies tonight because we are not savages and understand that before this dough hits the oven we're going to be chilling it for 12 hours to develop the flavours and texture. Did I mention we're making our chocolate chips from scratch?

The metaphor is that we're building something bespoke, it's probably in Rust, and nothing off-the-shelf is going to come anywhere close to what we need. We are probably comfortable sacrificing speedy turnaround here because we want something high quality that is a perfect fit for our unicorn problem.

This makes you the master craftsperson cookie baker. You can see so many ways to get lost and, with katana drawn, you carve a path through impenetrable chaos so that others may walk it safely. This is a position of full flexibility and vast responsibility. The potential for disaster is so close, and you need to constantly check all your assumptions and practises because there is a huge time and effort expenditure on the line, and maybe you have cooked too hard - but maybe we really need to be in the cookie pleasure dimension and this is how you get there.

Cosmic horror cookies

As you take your first bite there is a pain at the base of your skull where it meets your spine, and you are transported to the shadowy shores of Carcosa and strange is the night where black stars rise and here flaps the tatters of The King.

You have cooked too hard and only madness awaits you.

The metaphor here is that six months ago somebody asked you to spin up Postgres so they could bang some numbers into it instead of Excel, and when they checked back in on you they found that you were almost finished on your custom DSL and certain on reaping thusfar unimagined productivity gains if we just keep hacking for a year. Two at most.

This makes you the clown cookie baker. It is the genuine desire to be the master craftsperson but likely without the thousands of hours experience and burnt fingertips that birth such a person. It is also a highly seductive thing to be, and many of us that hit keys for a living have probably been this person at least once. I know I have.

Don't be this person.

Cleaning up

We're done!

I hinted at it in that last one, but to state it explicitly - these are not progressions, they are roles that we assume when we begin any piece of work or face any technical decision. Nobody is immune from clown mode no matter how many songs of their bravery ring throughout taverns across the land. Often the most masterfully crafted decision is to just grab the prefab cookie dough and not think too hard about sulfides or whatever.

The value lies in the intentionality of this choice: Which baker do I want to be in this moment?

The answer I believe, as given to me by a master craftsperson many years ago, is to be "the least worst" baker you can be. Give it a shot.

If you liked this essay, subscribe to our blog to get notified when new essays come out. We're focused on writing helpful, entertaining, and human writing, and we simply trust that the marketing will handle itself. We won't flood your inbox with trash. We will never, ever share your email with a third party.