I'm With Stupid

What do you do when you hear a really bad idea? It's that moment, sitting in a meeting, when you hear someone recommend something that doesn't sound like the right thing to do. On a good day, this can be managed by relying on the average intelligence and critical thinking of the room. There's probably a collective agreement, among those who didn't raise the bad idea, that it's something that shouldn't be done. That mutual understanding usually acts as a stop so the bad idea can be crowbarred out of everyone's way. Unfortunately, it's rarely a good day when a bunch of overpaid white-collared workers are shoved in a conference room / Teams call for 90 minutes. So what do you do when someone pipes up in support of this bad idea? And then someone else gives the whole thing a nudge in the same direction? It starts to gain momentum and before you know it you're careening downhill with no way to stop this stupid thing from happening.

This is what it sometim- ok, often feels like sitting in corporate meetings. Someone suggests they can write some "simple" code to unzip archived data, push the files into the modern day workflow, and delete the old files when they're done. This might be ok for a hobby project, but when the archived data is 80GB of sensitive information and losing even a single byte to corrupted files is not an option for the organisation, this isn't a "simple" fix. Almost always, the guy who suggested the bad idea is some dude who only learned enough programming so he could feel like he's better than his work buddies. You know you're in for a ride when someone casually drops the "oh yeah, I do a bit of Python". And listen, I'm all for people upskilling and moving into new fields - I was a Certified Jock^TM for 15 years before formally entering the tech industry. But like everything worth doing, data and software engineering require a lot of study and practice so you don't chase a ball down the wrong hill.

On Positioning...

It's extremely hard to navigate in the moment when a bad idea gets a footing. Groupthink and politeness take over as one-by-one people start to voice support for a crackpot idea that someone made up on the spot. Soon, most of the room is gaggling like drunk geese. It's especially hard to raise concerns with the way meetings are held these days. You're listening to a screen of blank tiles, making non-verbal communication utterly impossible. You sound like the asshole when you suggest that maybe a process that's as critical and risky as moving massive amounts of senstive data might need more thought and planning.

Positioning is everything. It doesn't matter what actual knowledge anyone has in a debate. It's all about how they're percieved as the a) trusted b) expert in the specific situation - these two things are hierarchical, which means that if someone is trusted, they don't need to be an expert. This is where grifters find their sweet spot. They know just enough to be dangerous (and I will stress here: everyone in tech needs to strive to know enough to be safe) and can spin a yarn as easily as they breathe. The promises made by sales managers from a big consultancy sound like expert advice, and these firms certainly appear to be trusted[^1], which makes it hard for an actual expert to pipe up and say that maybe the company doesn't need to go to the cloud and it would benefit more from fixing up its existing on-premise database.

Sales is a critical skill for data and software engineers. It's popular to talk about the "soft skills" that everyone needs, but that's a stupid concept because it's vague and full of hand-waving (apparently, the term "soft skills" was coined in the 1960s by the US Military - why would anyone anchor themselves to a social construct from that era of modern history when social ideologies were created by a bunch of dudes with the emotional maturity of a 14-year old?!).

Let's be concrete: data and software engineers need to be able to sell their good idea so that organisations don't choose to follow the bad idea. As an expert, if you want to get your point across, you need to be trusted. To be trusted, you need proof that your solution will produce the best outcome for the organisation. Now, a trusted non-expert will try to generate this proof by (1) making something up on the spot and talking in circles until they land on a startegy that sounds convincing, (2) recommending a bad idea they've used in the past simply because it just so happened to work this one time or (3) outright lying.

The weapon that engineers need to use against these false proofs is to debunk them. But the straightest path...

...isn't the right one. Instead, we need to enter the sales arena and join the battle with a thoughtful and well-planned strategy. April Dunford talks about providing market insight to a potential customer in her book Sales Pitch. When a bad idea is proposed, embrace it and start a discussion about it:

• What benefits does the idea provide to the org?
• How quickly can the problem be solved by following through with this idea?
• What are the risks and problems that might come up?
• What are the real consequences if one of these problems happens?

Once everyone is talking through the proposed (bad) idea, the room's critical thinking I mentioned earlier will hopefully stop the momentum. As the data expert, I often have to lead every part of this conversation, but just one other person in the room providing answers to these questions will help make it obvious that the idea is a bad one. This then opens the opportunity for alternatives. Hopefully you have one ready (e.g., a Sales Pitch à la April Dunford), but this isn't necessary (or easy) when the goal is to put an end to a bad idea that was sprung on you during a 90 minute Teams meeting.

We Still Need a Solution

The next step for the Expert Engineer is to propose an alternative and convince the gaggle that it's the sensible direction to head in. Again, the urge to speak truths...

...needs to be pushed deep, deep down, just like every other emotion except anger[^2].

It's incredibly important to communicate with empathy when it becomes apparent that a large amount of money, time, and stress has already been wasted on a project. The benefit of the doubt has to be given to the people who - fingers crossed - inadvertently made the wrong choice to buy that enterprise software, migrate a broken database to the cloud, trust a big consultancy[^3]. Sales people are gonna sell, and unfortunately blatant lies are not off limits to anyone, anywhere. There may very well have been a consultation period where internal teams were canvased for the utility of having a cloud-based system, but they may not have quite understood the benefits of centralising workflows at the time. And so a half-baked solution was developed by under-achieving engineers at a consultancy after their over-achieving salesperson made ludicrous promises. The not-technical-enough manager from your org who was responsible for signing off on the solution delivery didn't want to look like they were in over their head (and let's be honest, no one wants to feel this way). So they nodded, raised their eyebrows, and enthusiastically hmmmm'd their way through the handover presentation delivered by our overzealous sales scumba- I mean salesperson.

Whatever the case, we're here now and the organisation needs a way out of the hell hole it finds itself in. I'm a strong advocate for designing a best-fit solution by following best practices. And this doesn't mean tearing everything down and starting from scratch. What it does mean is that corners can't be cut for the components that need to be rebuilt. No one used version control for the custom application? Start your initial Git commit with the existing codebase and iterate from there (but first make sure some dummy didn't store sensitive information or passwords in the codebase...). The stack wasn't created from IaC (Infrastructure-as-Code)? Break out the documentation for Terraform and create dev, test, and prod environments. You find a database password hard-coded in a script? IMMEDIATELY LEARN HOW AWS SECRETS MANAGER WORKS AND FIX IT!!!

The best-fit solution may involve a chunk of the existing dumpster fire. But there is always the option of swallowing a loss, ditching the sunk cost fallacy, and starting over with a clean slate. Again, communicating with empathy is so important for reasons I've already mentioned. Whatever path gets chosen, it has to be walked with high integrity - both in morals and design patterns.

A Not-So-Funny Story

A while back when I was contracting with a large organisation, I worked alongside a group of consultants from an external firm (they weren't from one of the Big Consultancies, but they were cut from the same cloth...). I was the data engineering / data science guy and they were SMEs (subject matter experts) there to provide some extra fire power for pushing out data from bespoke software developed by the organisation. These consultants were experts in preparing inputs for the software and their contribution was incredibly valuable, but they weren't programmers by any means - that's what I was there for. One day, one of the consultants pinged me on Teams to ask if I could restart a desktop that ran the software since I happen to be in the office. He was having trouble remoting into the machine and wanted me to do a hard reboot (i.e., push the button on the computer until it turned off and then push the button again so it turned back on). I voiced my concerns that this action could turn the computer's boot drive into a brick if there was an update currently running - the org was in the process of updating the version of Windows across all its machines. I suggested he contact the IT department instead and he agreed that was a good idea.

About 10 days later, we were in a weekly huddle with the rest of the team working on the project (oh, how I despise these wastes of time...). Buddy from the consultancy openly asked if any of the 8 people in the meeting could kindly do a hard reboot of the desktop that he still couldn't remote into. After more than one experience with the above-mentioned bad ideas that spread like the plague during these types of meetings, I piped up first to raise my concerns - for the second time - about fucking up the computer. I warned that one or more of the hard drives on the machine could be completely lost if they turned the computer off during a critical update. I then asked if there was anything on the desktop that was needed for the project. And that's when the fury hit me. Our good friend the consultant - who was being paid around $AUD240 per hour - shared that there was around 3 YEARS OF HISTORICAL DATA THAT WASN'T BACKED UP ANYWHERE ON THE GODDAMN DESKTOP HE'S BEEN USING!!! Without even checking with the IT department, this guy was willing to risk millions of dollars[^4] of their client's money in lost data. Not only that, but the data was necessary for litigation purposes for the organisation, so they would've had no foot to stand on during an audit.

What To Do, What To Do

I try not to be the angry guy on the internet. We already have one of those at our company. But it's progressively harder and harder not to be angry when you see these horrific ideas get put into action on a regular basis. I wrote about the consequences of low expectations and how the biggest negative impact of these bad ideas is on the souls of people. It's horribly draining to have to constantly fight for reason in a room full of unreasonable ideas.

I propose that we commit to doing better. For instance, data engineers can read the docs for a couple of hours before they start configuring dbt for a data migration. And execs can have a listen to a podcast or watch a YouTube video that explains the concept of a Data Warehouse before they buy Snowflake. Change doesn't need to be drastic to stop bad ideas before they start.
[^1] Though it seems like that trust is starting to waver, at least in Australia

[^2] I kid, I kid... Not about surpressing the urge to yell at someone in the workplace, though.

[^3] Full stop. No other caveate can be provided when you trust a big consultancy with your tech stack.

[^4] It took anywhere between 8 and 16 hours to process data through the organisation's software for less than 10% of the output for a given project. There were no checkpoints built-in, so a human had to monitor the workflow every time to know if an error occurred. I'm proud to say that I found a few things to automate so that someone didn't have to stare at a computer and wait for the thing to finish (or fail).

If you liked this essay, subscribe to our blog to get notified when new essays come out. We're focused on writing helpful, entertaining, and human writing, and we simply trust that the marketing will handle itself. We won't flood your inbox with trash. We will never, ever share your email with a third party.