On Leadership

[ROUGH DRAFT] Notes on leadership and Principal Engineering

Leadership is the art of being the bigger person. A few examples and quotes capture this most beautifully. My son Patrick said to me “the Ying and the Yang of life is one person being nice and one being mean. This should be expected, and not an aberration.” Wise words from my 17-year-old son.

Barak Obama famously embraced respectful disagreement:

2017 - Obama’s farewell speech

As a principal engineer, much of your job is to disagree respectfully, and empathetically.

Second, is to try to head off public disagreements , which is a form of empathy.

Finally is to manage disagreements to resolutions; and when possible, provide a decision framework for similar decisions (e.g. see my blog here “Cost of a Microservice.”)

Every conversation is “crucial”

The most important thing to note about a Principal Engineer role, (or a management role) is from the book “Crucial Conversations”. The book explains how to conduct safe and productive conversations when important or sensitive decisions are being made.

As a Principal Engineer, every conversation you have is a crucial conversation. You will have limited interactions with hundreds of people as a Principal Engineer.
Few people may understand you or your point of view, until they have known you for a year or more.
These interactions will be important conversations for the people interacting with you, either confirming or denying their beliefs on a decision. Along with whatever decision you are contributing to, you are also confirming or denying the other persons ability to frame and communicate a difficult technical idea. They may be profoundly impacted by your feedback, and it could discourage them from their hard work.

To do this well, you need to be complimentary and encourage people to try their hardest. The easiest way to do this, is point out the good things you find in any work. This won’t be easy, but you get used to it. Take off your critical thinking, problem-solving hat. Start by noting everything positive you find, and thanking everyone for their hard work on the effort.

Your goal is to encourage what is well done first of all. This will make for an environment of support for the team you are on, in which people feel safe. Safe in the sense of the Crucial Conversations book. Without this safety, people won’t ask questions and won’t disagree. There will be no sharing of information or ideas on modern software development, or how to complete the project. Without this open sharing and creativity, you have lost the best output of a team of people, which is to reflect on each others views and ideas, and to improve the ideas before the painstaking act of authoring the software, and launching it.

Before the software is written, when decisions are being made on direction and investments are when people have the ability to make major contributions, and also major mistakes. If mistakes are made, you need to gently explain why the idea is mistaken, and emphatically see the good value in the idea. You also need to acknowledge you may be wrong, or no one may agree with you, or understand your value judgement. In the end, the software can be designed and built thousands of ways, like playing a chess game. There are only value judgements, and different values being expressed.

Always acknowledge this, and the value in any idea, if you must go against it. Do it in private, with an audience of one if you can, and if you can’t, do it with a tiny audience. Without this, you are creating an unsafe environment, and people won’t want to discuss things with you or work with you. They may be forced to because management wants your take on things, but they may not want to get your input.

When you give feedback, the easiest way to suggest something is to say “I like to do.” statements, about how you like to do things. Then you are not saying anything negative. You are merely saying, this is how I like to do it. Not that the current way is wrong, and also you are not saying that your way is better, just that you like to do it this way.

If you want to be slightly more forceful, you can then ask “could we try it once my way, and compare.” This second push can be saved for times when you are certain something needs to be re-done. Again you are not saying the current way is bad, only that your approach has certain merits. You can mention the merits, maybe briefly.

Other times you may mention how you prefer things, but then leave it to the engineer if they want to try it your way.
You leave it as an opportunity for them.
This is the only way to give feedback as a principal.

As a manager, you can be a bit stronger and say that you prefer things be done a certain way; but again you are not saying it is necessarily better, but it is what you know - you have this preference.

Every conversation you have, and every bit of feedback you give is this exact act happening repeatedly. The most difficult thing about Principal Engineering is that every day, every conversation you have is a “Crucial Conversation.” It is like you are a congress person, an elected official, elected by the management team to give opinions on much of the work (where management co-owns the decisions, and then they have their ownership chain, and are making the same recommendations upwards.)

In contrast to pre-principal engineering work, you are not going to lunch with your crew of four programmers every day, and can reflect back tomorrow on something which happened recently, or months ago and laugh, retract what you said. As a principal, you may not speak to the engineer you talk with today for months, (and you may not remember what you said.)

You need to use extreme care for these conversations, for this reason. You are in this way like a Priest or a public figure. People have very little time to get to know you. Despite the focus of your life being technical, you are now partly a priest and partly a politician - good luck! :)

Peer-review your work with other Principals and Senior Engineers

(Staff / Senior Staff)

There is clearly competition to lead in any business. The major job of the leaders is to make decisions. There will be though hundreds of these two be made, and leaders will not always be right, including you. This is a part of why Amazon has the leadership principle “Right A Lot.”

Being “right a lot” means having the ability to change your view before investments in software are made. The difficulty is once you have published ideas in a large public forum, you’ve made a public recommendation. It is better to avoid making a public recommendation, when you can keep it private and circulate it with leadership. If you can do this with peers and stakeholders, you are better off than once you’ve published a recommendation to management. In this sense you are keeping your cards to yourself until you have collected diverse opinions. This has the impact of delivering messages as a united front of principal engineers, instead of you alone, perhaps against another principal engineer.

As I write this, I think cross-system engineering decisions should be reviewed by all principal engineers whose systems will be touched by any given decision, in private. This gives the other principal engineers a chance to take a look before you distribute an idea to a winder audience, most importantly up the chain where leadership are very busy. If you create noise and reverse decisions or opinions, you are creating more churn in their leadership process. This also can reduce trust in your recommendations, which is not good for you yourself, or the org later, after you have lost trust.

Look to confirm any decisions with as many principal engineers as possible around you, before publishing them; even if in a draft phase, and a quick 15 or 30-minute meeting. As principals, they likely will understand the initiative quickly, and can give you feedback, also raising it with them. They may think about it off-line and get back to you. It keeps them in the loop, this way they are not surprised about decisions which impact their area.

By confirming first with other Principal Engineers, when you approach the management team, you are showing them a well-formed, and completed analysis. It may feel like you are cheating by asking other principal engineers, and then taking their feedback and running with it if it is better. But you are not; as there is a decision you are making to incorporate their inputs.

If their approach is not your favorite, you should get back to them offline, and see if you can work it out. If you do agree with them, then you can simply thank them in the document for the idea, and also let them know you used it. Either way, they understand where you are coming from, as their systems and interfaces are being impacted by your decision. And better you use their approach, than to contradict them; they will be happy you consulted them first, and used their approach!

As a principal, almost everything you touch may be across principals, and it is better to involve more of them. They may not be able to attend the meeting, but they may do a quick 10 minute grok of the document you attach to the meeting invite. This gives more visibility and helps you not only be viewed as, but also actually be a team player.

Long term relationships over software

The best way to be effective in the long term, is to have a group of senior engineers who listen to each other. When you listen to each other, and maybe even like working with each other, you are willing to take feedback on ideas without any emotion. When you do this, you find better ideas, and ideas are improved upon before they are minted as code (for all time usually.) You never really have time to revisit anything; “nothing so permanent as a temporary fix”, also “you can never unscramble an egg.” You need to make the best decisions possible, and this will be a combination of ideas, and improvements on ideas, and then pros and cons. Usually through doing this, you can drastically improve a software system. Usually there can be multiple rounds of this, to get ideas flowing, or to introduce alternatives as a first step before you move forward. Often times it takes discussions back to getting requirements, and how the requirements impact the software. All of these activities are ideas being raised and then discarded.

Similarly, during production outages, ideas need to be raised and discarded. If you need to refocus on a different approach during an outage, it is very similar to the design decision conversations, but happening in real time instead of slowly with time to reflect and return to the discussion after sleeping on it.

The only way to be effective with a team of senior people is to trust and like each other. If you don’t, you will not be open during these discussions, or you won’t be yourself. You may not listen to the team, and the team may not listen to you. When this occurs, you are now moving much slower towards good ideas for the software (or the outage), being pursued, and in a valuable order of operations. Then the battle is lost.

Before you can have productive conversations, you need to be willing to walk away with no connection to decisions you do not agree with. This is incredibly difficult for rational people. Programmers are logicians, making hundreds of small value judgements every day when they write code. We continue to flex our logic muscle until it is overdeveloped. As a principal engineer, you need to turn off this logic and reason muscle, and instead overvalue emotions and relationships. In a sense you need to turn everything you know and your soul of an engineer, doing everything as well as possible always and overvalue the relationship over the software. Eventually, after a year or years of doing this, you will be able to get what you want with the software. It is usually about six months to a year with any group. But you must do this, and it is extremely unnatural. Start doing this immediately. This applies to all engineers in general, but more-so to managers and principal engineers.

Respect and manners

The only expression we humans have is what we say, what we write. Be well-mannered in all your interactions, is an easy start. Without manners, we are acting like animals. Keep your manners at all times, act as you would in church. Because the business you are working in, is a church to the person who founded it, and the core team who have worked there forever. Do not disrespect the Church at any time, for any reason. It is not your church, it is theirs, and you are a part of it.

This seems easy, but you will have passionate discussions about software, some of them, how to handle major outages, during outages. These are easy times to lose your manners, and the most important time to keep your manners, at all costs.

Well mannered in heated disagreements

As a principal engineer, most of the discussions you are having are about ambiguity and may involve dissenting opionions, which amount to disagreements.

Also, as everyone wants to be a principal, everyone will be hurt that they are not you. This is painful, and I remember it. I remember being passed over for promotions I thought I deserved. In hindsight, I did not deserve them, but not for technical reasons, for emotional ones. Technically it is possible or probable I did deserve them, and I thought “how important is everything else, where are here to write software!” Leadership is mostly about emotional control under duress, and leadership is mostly emotional duress. While you are being attacked, you must turn the other cheek, and not be upset by attacks of your ability, or your understanding. These will be constant. As a principal, you are now the measuring stick by which all others will measure themselves (and this is a natrual thing for them to do.) You need to ignore it, and walk away. This is probably the most difficult thing to do. It can be or feel like parenting. It can be emotionally exhausting, but you can adjust to doing it, and ignoring it is a muscle you can flex. Flex it a few times, and realize how good it feels. It also makes you look very good, but more importantly it is your job to do this. It is probably the most difficult part of being a principal engineer or any leadership role, to have these kind of attacks, and to expect them; and not hold them against the other person. If they don’t attack or challenge you, then they have no way to improve themselves. This is exactly what you want as a principal engineer, for the benefit of your programming team and the firm.

As the hardest part of the job, I view it like a game, and I try to keep score with myself. I give myself 100 points every time I drop a challenge to myself technically or otherwise , as a reward. You want people using you as a measuring stick. It means they are trying to get better, and shows they hold you in high regard. If they are NOT measuring against you, then you have a problem - the measuring against you is the ultimate sign of respsect, from them.

You must treat it as such, and not a challenge to you yourself. This is the most counter-intuitive thing I have found about leadership.

It is not about “giving zero effs”, nor about “don’t sweat the small stuff.” Instead, it is about wanting to be challenged, welcoming it as part of your job, and rewarding yourself for not making it emotional.

Ruthless respect for everyone

Ruthless respect for Senior Principals

Unless absolutely necessary, do not challenge the status-quo. If you do need to challenge it, do it in the following manner.

1. Privately
2. Documented value, documented approach
3. Tested completely
4. Fully scoped cost wise
5. Peer reviewed privately
6. Reviewed with every senior principal privately
   1. (and ready to discard your work, when you realize why things were done a certain way.)

When you challenge the status quo, you may be adding tremendous value. It may be in some way you know about from your experience, and people may not see it. You may also be wrong, either in that your approach does not work, or better, is that it is not a priority, or cost-effective use of programmer resources. Finally, it may have risks to change, or risks to churn or burnout for the team to handle it. There is very little reason to challenge the status-quo, if things are working. Usually, there are opportunities to make money which are more important than any refactoring or change. This is real business growth, which is the most important thing you can do.

When or if you do want to challenge the status quo, you should do it very carefully and privately, and document it fully, on your own spare time, and then review it with a peer you know well, and then quietly with your manager to see if they are interested in pursuing it as a project. By this phase, most status-quo challenges will come out as not valuable, for cost-benefit or opportunity cost reasons primarily, but could also be risk, or things you don’t know about. Beyond any value, it will be disruptive to the person who owns the status quo, and the senior principal or principal who decided on the status quo. When you do present the change, you must present the exact path forward, fully, and completely, with a concise executive summary, and exactly why you are doing it. If you need to do this or are asked specifcially to evaluate a prior decision, outline that this is extremely difficult, and make sure to manage the expectations of your manager on this, as to what the cost of your time and energy will be, and what the outcome will be. One of the outcomes will be hurt feelings of the team, the engineers, management team, and senior engineers, but also the senior mangement team. It will be very painful for them to learn about a strategic mistake they have made. Be sure to let your manager know you are being asked to do this, and the risks involved in doing this kind of anlaysis. Do not volunteer for these, unless they are critical to the very survival of the business unit you are in, or the livelihood of everyone around you.

As Marvin Gaye said, “We are all sensitive people.”.
Working in senior positions in software is stressful. By challenging the status-quo, you are causing more stress than the existing workload, and the generally stressors of life, on the senior engineering team (Senior Principals + Principals), the executive team, and the management team. No one will understand or appreciate your challenge to the status-quo. People are challenging it constantly, as this is natural for problem solvers to do - programmers.

It is part of your muscles you use to move code every day, to want to improve the house you live in. But you must avoid house improvements and renovations at all costs here. It causes undue stress and can create rifts in the software team. If you must do it, you must invest very heavily using your own time and research - and your manager must commit to this fully before you begin.

If you don’t invest privately, it will create major rifts with you and the management team, and the entire engineering staff - making you wildly unpopular. This will lead to you not being satisfied, as you are now a wildly unpopular politician. You won’t go far, given you are a politician, if you are not popular. This means, challenging the status-quo can only be something which is absolutely necessary or revolutionary for the system you are working on - it can not be in any way trivial, and you must completely prove it. Only do so if you are asked to do this, and then, ask for a major proper commitment to do this, with the understanding you will need to move to another project or area after it is done, with this transition plan made well in advance of your critique.

Due to the risks of challenging prior decisions, this is work typically done by expensive, outside management consultants (McKinsey etc.), because at the end of the engagement, they can no longer work with the existing team, as they may have had to recommend their removal, or an entirely different software strategy than was employed, making them look bad.
Leave this awful task to a management consulting team if can, or make sure you will be recognized for the extreme career risk you are taking on.

By avoiding unnecessary challenges to the status-quo, you are also respecting the Senior Principals around you, and they are constantly being bombarded with questions for improvements they could or should be making to the software.

Amazon Principal Engineers have a value called “Respect what came before.” In this blog, I’ve presented a more elaborate explanation of this, in terms of what it feels like to be challenged, after you have written a software system (usually under some duress.) Avoid these challnges unless they are well-thought-out, prepared, and absolutely necessary (and also - more valuable than anything else you can be doing for the business!)

Ruthless respect for other Principals

Always compliment any other principal if asked about them. All of you are in challenging, stressful, high stakes software positions. These are very difficult jobs. You may not like the person, but always have the respect and say they know what they are doing, and expect they will do the same for you. They will not be perfect, and neither will you. Nor are the people asking you about them. On their best day, they are very good, otherwise they would not have been able to reach this milestone. Even if today is not their best day, they can have another one, and you will too.

Ruthless support of management

TODO

Ruthless support of new-hires

TODO

Examples of crucial conversations

TODO

Microservice vs. “Monolith” (service of libraries)

How to make an Ads system multi-currency

####

Appendix

Best leadership books

1. Crucial conversations
   1. Note - every conversation in business is crucial - particularly senior positions
2. Leadership and Self-Deception
   1. A book about caring for co-workers, and their well-being.
   2. Trying too hard at your job, or adding noise - is a form of not caring enough, and a difficult one.
3. The Innovators Solution
   1. An important book about how to focus on ideas, and filter out ideas for value proposition.
4. Emotional Intellgence
   1. This should be a business bible. Outside of business (or software business), emotions don’t run as high
   2. No where is there as much artistry and ambiguity, combined, in the software field
      1. No where is the intersection more profound of these skills than politics, and senior software roles

Software is necessarily messy - don’t fix it

Software is a mess, everywhere, necessarily.

There is a view that any code once written, instantly becomes technical debt. For most business systems, this is likely true. You are rarely writing a product, but instead writing a distributed messaging system. By way of being distributed, there are usually multiple artistic views of how to write software being expressed in different tiers of the sytsem, and different authors. Without someone like Linus or a core group, or even more senior engineers reviewing every decision and commit (to move at the speed of business), you end up with different approaches to software being expressed. The goal of software is to accomplish a goal - it is text in one or more files of different types and kinds. Soon much of it may be written by AI. The important thing in business, and also in life, is to build useful tools which have a value - not beautiful software. This overriding value objective will always be (correctly) expressed in the majority of software development. As I like to say, “The software is never finished.” I also have yet to work on a perfect software system. We are still in the early days of “C” as operating systems, and JavaScript as the front end. These systems are essentially the first ever widely adopted, iterations of expression of software as code. There are more coming, like Rust and WASM, and there will likely be more. Your job as principal is to see what is good, and to steer the group towards completing projects, while keeping productivity high. Making perfect software would require unlimited resources. You have a finite resource and certain value goals. You need to make choices which support the current set of value goals, and which will also support the future goals with an overall strategy which can be built upon, software hygiene and otherwise.

Your job is ambiguous and difficult, but in the sense of leadership, it is not perfect software.