Good Intentions

The road to software hell is indeed paved with good intentions.

Nowhere in humanity has it ever been as easy to wish to do something good, and instead make a terrible horror of a project.

Working with elite mentors, it has taken me decades to become decent at using good judgment before I begin a large software project.
Like a chess game, it is extremely challenging to see what outcome a ‘good software plan’ will result in.

Software is also a frustrating and time-consuming activity. Engineers are often trying to improve the craft, but are generally not in a position to do so.

In my blogs I will detail three totally flawed, $ 100-million dollar software efforts, each of which was totally unnecessary.

Each of the three cases shares at least one key quality—trying to improve a software system in a novel and unproven way.

2nd, all three share the quality of having never been compared to alternatives properly and in a detailed manner—only a hand wavy comparison stating ‘this will be better’ - without any real data.

Apple: Value scrutiny of ideas

Apple has a core discipline where they look at every detail in a design, and look for the hardest parts of the problem.

This approach is necessary for good software design (if seemingly unpleasant.)

Tiny details can destroy any hopes of success for a software effort. We will show a few here and how this is a pattern in industry.

Failing to scrutinize a plan / design is an easy way to fail the analysis phase.
This often occurs due to a natural aversion to avoid disagreeing in corporate software.

Disagreeing is not fun, makes one unpopular, and is the highest value activity a software engineer and manager can do.

Be like Gilfoyle

You need more Gilfoyle from Silicon Valley, but it is an unpopular and unfriendly way to be (comically so.)

GIlfoyles unfriendliness and brutality is what makes him the most valuable team member. And no one likes Gilfoyle (but Gilfoyle is ok with it.) This is the viewed (and often is) the antithesis of ‘EQ’ from the great book Emotional Intelligence; however, this critical thinking and critiquing is the most valuable virtue in software. The ‘Gilfoyle’ quality of an engineer, needs to be the most valued and triumphed, but it is actively discouraged “he is not nice.”

As a parent, friend (and lover!), I value kindness and sensitivity. But not in engineering conversations.

HBR on scrutiny at Apple

This Harvard Business Review document about Apple Culture is a must-read. The key takeaway is that at Apple, they look at the hardest part of a problem first, and scrutinize ideas before they begin. Sadly, I have never worked at Apple, but it is admirable if it is actually held up in practice, which is reported to be by HBR.

(side note: HBR is an incredible resource for software managers, because software managers are in fact running large sub-departments, and should understand economics, business, but I am only beginning and very few of us do.)

No benefit to Disagree (Corporate Game Theory)

From a game theory perspective, disagreeing with another Engineer is always a loss of value.

Why is this?

The rewards from the disagreement for any employee never outweigh the economic cost of the disagreement.

This kind of scrutiny benefits owners of the company; primarily, and the CEO.

Inside the company, only the senior executives and stockholders benefit from actual performance of the software system.
Not the engineers. The customers obviously benefit from disagreement and correct decisions and software. This use of the customer is convenient as a proxy argument for saying ‘do what is good for business, not yourself.’

It is true if the employee software engineer holds stock, then the stock price has value, and the actions have an impact on the stock price in some small way. But this small stock value impact (on some tiny percentage of the ‘float’), for disagreeing in a design meeting never comes close to the cost to the engineer raising the disagreement. The biggest potential benefit to any Engineer is always a promotion, and the best way to achieve the benefit is to be well liked. The easiest way to be well liked, is to agree. This is the simple math of software design and seniority. Every senior Enginner is aware of this simple equation, and avoiding disagreements. Somehow I tend to disagree more than most, because I think life carries some deeper meaning, and my time is too valuable on this earth to work on bad software. Some would say I disagree often, but in fact I am even trying hard to be agreeable; I simply have seen more software failures and am burdened with knowing how it ends.

Write down and review the plan: look for gaps!

The key is writing down (a design) both exactly what you want to do, and why, with as much detail as possible.

As Leslie Lamport says, a great video: Code is not Programming

The programming is mostly trivial.
The design dictates the outcome.
The design, and 2nd most, what projects you choose to invest your time and effort in.
Coding is relatively easy for any senior Engineer, tedious but straightforward.
Particularly in business systems.
Let’s say 98+% of it is straightforward message passing.

Amazon generally does an amazing job of having things written down. There are now cracks in the system where some someone senior enough can say they do not like a plan, with the best ideas are not used, some otherplan is used instead which favors politics.
Politics over code. Amazon is large enough where everyone is after their own success, and no one cares about the firm. It still works surprisingly well but is incredibly dysfunctional.

Off-topic, Amazon is Darwinism instead of cooperation—inside a company where people are supposed to help each other succeed.

Stay tuned for the three biggest project failures, each more than 100 million dollars in time and money, each with trivial alternatives which were missed or discarded…