A Software Velocity Quest
Don’t overly focus on your software architecture, until you have first maximized your software process itself.
Without the process in place, there is no point trying to reorganize the software.
At a fundamental level, reorganizing a software system has no effect other than shuffling around where code is committed, and what buttons are pushed and by whom, to release a software system.
There is a much greater virtue that must be held above all, in the quest for the mighty “software velocity.”
The virtue is software process hygiene.
Hygiene is multi-faceted, and poorly practiced in industry. Before you try to do anything, practice software hygiene. Once you are completely healthy in this regard, and not before, then you can think about the next set of changes.
A problem in the SaaS software industry today; think first about how to alter the software, with terrible hygiene. This is the software equivalent of telling your teenager to go forth and make babies, when your teenager has barely learned how to take care of itself. You would not do this as a parent; don’t do it as a software parent.
Your software process and hygiene needs to grow up. This maturation process is mechanical and straightforward, yet, it is skipped. The reason it is skipped is
- lack of awareness (ignorance)
- incorrect value judgement
- lack of creativity
Without knowing what it is, people can’t seek for an unknown goal. Without realizing the value, the focus of software is diverted elsewhere.
Generally, everyone thinks their software process is hygenic enough, and are wrong.
If you ask the developers who work on one project, “What can we do to improve velocity?”.
The answer never is, “first, we must improve the systems software development process, our system, because the software development process we ourselves practice is unhygienic.” Why is this, why don’t they say it?
The developers are close to the system, working on it every day. Because they can be productive after perhaps authoring it or working on it for years, they themselves think it is a productive environment. But they are not the test of productivity. The test of productivity is a new developer signing up to work on the project, a new full stack developer trying to deliver a small feature. How would a new highly productive coding joiner do this; what steps would they have to take? Are those steps documented precisely, and are they repeatable?
Fordo
Truly, one of the best in the software industry is a friend of mine named Jason Ford.
Thanks Jason if you read this for being an incredible mentor, the greatest really ever, and a dear friend.
I reported to Jason, and we worked shoulder-to-shoulder to expand the Jefferies investment bank, Equity Derivatives technology, to support the growing market making operatoins of Jefferies. Jason started at 6am every day to verify the US sytems were working, giving proper time to correct any data or other issues, before trading began around 7:30am. We met many days at Jefferies at 6:00 AM, when he was training me. I remember getting the 5:20am Metro North train, many cold mornings to meet Jason. Those were some of the best days of my career coding, meeting Jason at 6:00 AM for coffee and two hours together.
Jason had developed himself, a large services environment; you could even perhaps call it microservices, but I would simply call it services. No fewer than 16 services were necessary to run the core US Equity Options market making systems.
One of the most shocking and eye opening things I have ever seen, and I will never forget this. Jason had on his PC, 16 windows shortcuts. You can have a windows shortcut open on specific coordinates on your monitor. Jason had set the coordinates of each of the 16 shortcuts, so that by clicking on the 16 icons, a grid of 16 console windows would appear on his screen. And voila, the entire system was running on his Desktop PC, ready to develop. Next he would start the C# user interface, and begin to test any feature he wished, with a complete working copy running on his own PC. Mind you this list of systems starting locally included:
- Real time derivative pricing
- Porfolio aggregation in real time + risk
- In-memory security master
- A complete EMS/OMS for trading,
- ETF creation/redemption
- ETF basket trading
..to name a few key parts of the system which were started.
This represented starting one of the most complex
investment banking divisions, on his PC.
Jason did this non-chalantly one morning, without giving
it any thought. My head nearly exploded; you have
organized and done the work to allow any developer
to start the entire investment banking division on your PC?
Yes he said, of course. How else would I develop the system?
Jason took for granted, this was the only way to remain organized and be productive. To say that Jason is an elite craftsman and practitioner does not begin, and we would not be surprised to learn Jason was and is also an elite C++ engineer. As an elite C++ engineer, Jason merely needed this situation, to make himself able to go fast.
Jason was aware that without these investments, he would not be able to debug the system, in the event of an emergency. When you have built and then are also managing an investment banking, market making business, you need to be able to solve any issue in real time. If you fail to solve a production issue, you can count the tens of thousands of dollars by the minute. And we didn’t have hundreds of engineers. We didn’t have tens of engineers. We had five engineers. This was a growing startup, inside Jefferies.
Merely to cope with the enormous burden of the task at hand, Jason had developed techniques for practicing development, beyond any I had ever seen. I think to this day, it was the cleanest and most hygienic - and efficient development environment - I have ever worked in.
On my first day, I had cloned, built and ran this same exact environment, myself. The longest effort was setting the location of 16 console windows, which took about 15 minutes. I noted to Jason, that it took me 15 minutes. He said “yes, you could check those console windows in as well, but it is only one time and takes 15 minutes. Some people like to position the windows differently.”
Indeed Jason, indeed. Within 15 minutes I was running a large portion of a large investment bank, on my PC, and was productive. As shockingly good as this was; it was only the beginning of my education with Mr. Ford.
Jason was manging the team of five, which grew to ten, and he was actively coding himself, for half the day. I generally recall Jason coding from 6-8am, then going to the gym, then coming back and coding until lunch. After lunch was time for meetings, and helping me and other developers. This experience was a master class in development velocity, productivity, time management, and most of what you may read about in the Harvard Business Review journal. Jason invented these practices himself; code in the morning. Without coding on the effort, Jason I think knew intuitively he would lack the pulse of the project, and the specific challenges we were facing. It didn’t matter that Jason was an SVP at Jeffieries - he was coding.
Also, Jason would sing our names every morning when we arrrived, I arrived every day to Jason smiling and signing:
“Rodeo, is in the house, let’s hear it for Chris.”
This was my morning greeting, smilling and singing my praises on “the trading floor”, to the other developers. And he did this morning greeting for all of his team members. Jason was having fun, and doing what he loves, and leading us to victory. And we were victorious, along with being the tightest knit software team I have been a part of. We worked 8-5, no one burned out, and everyone grew and improved as an engineer and leader under Jason. To say Jason is an elite leader and technologist is an understatement. Jason is an absolute gem of a person, and a real testament to the quality and passion of the New York software financial practice. Jason - I value your friendship more than you know, and those two years were the best apprenticeship I ever had.
“It is too challenging to make a good Dev like Prod”
When I hear, and I have heard this multiple times in the past two years, from very senior software people “It is too difficult,” when asked if we have this exact ability Jason invented, I think, you are mistaken and are not being inventive enough.
Jason needed this to survive and thrive when running
Jefferies, and he invented solutions.
At Jefferies your PC for example,
would not run the full global markets; it would run a slice
of the US equity option market. And you could configure
which symbols it was pricing locally, with a config file.
In this way, Jason invented techniques to make a smaller,
identical copy, with one good example
of the production system
which could be run. Why is this easy? You don’t need
every single unit to run production; you need a small piece
of it which represents production. Can you not define
a small piece? Can you not invent one? Really?
A test customer, who has a small subset? (sometimes known as
a Golden Copy.)
It is hard for me to believe when I hear “it is hard to run prod locally.”, after having worked for Jason, and then repeated this same exercise in other shops, myself since then.
This hygiene is the most valuable investment you can make for your software development velocity, but it is never made first. First, teams look at splitting up the software, using microservices. As a secondary concern only do engineers and teams review making a hygenic software environment.
This is the wrong order of operations entirely, and can become a tragic flaw of major SaaS software operations; which I have seen first hand, and some big dot coms who you thought were elite. To say they are not elite in this regard, and in many others, is an understatement.
If necessity is the father of invention, which it was for Jason, the lack of necessity is the death of invention for big software vendors. Rather than ponder what it would take to make one developer 2x as productive, Software Directors these days like to think, how can I hire 2x the software developers. These are the wrong thoughts to be having when running a software operation; but it is good for self-interested Directors. Having 2x the staff, makes them more important, maybe bigger bonuses. Making the staff 2x productive is hard to measure, and not as sexy as having 20 or 50 developers, when instead, you could have 10 highly productive and senior developers. I think most corporate managers have only worked in big corporations, and maybe never considered how to make each developer as productive as possible, and instead focus on how they can add more people.
This approach to software productivity (velocity) is totally wrong and backwards. It is happening everywhere in the world today, with people learning about microservices and thinking they work for Amazon.
Microservices do not work for Amazon, and Amazon has the worst per unit engineer output I have ever witnessed.
Amazon is good at producing projects which produce enormous scale of revenue quickly; they are good at Business. Once they have achieved the scale of returns, then they throw staff at the problem. Amazon are not good at per unit software engineer returns - Amazon are a “scale” shop, working on massive opportunities.
You are not like Amazon, and don’t have the same opportunities. You need to be efficient. Amazon will start a new business line to make a billion dollars tomorrow if the right VP has a defensible idea. The engineering is not their primary concern, it is the scalable output of the business. You don’t have the same capital as Amazon, or the same ability to hire 100 engineers, train them, and push them to execute. And the Amazon output per developer is extraordinary low; they work with the largest development staff of any shop.
So don’t try to be like Amazon, and split up your system into Microservices. You need to first, focus on the per unit, per engineer productivity. To do this, the developers need the ability to run prod and make a clean working prod quickly.
Working for Jason, I could make an Equity Options investment bank on my PC in 15 minutes, and Jason was one developer, who made this possible—himself alone.
I find it hard to believe when large software shops say it is challenging to this, when I’ve done it alone, and worked for mentors who did it alone - and made it look easy.
(and while singing to me every morning when I arrived.)
Make it easy to run prod - and have perfect hygiene, before you take on any re-architecture of your code. You need the hygiene anyway. And once you re-architect, you need this to continue.
Without having this first, you are actually making it more difficult to be productive, and hurting the productivity per engineer. You can hire more engineers, but they can’t work across the system. It is a recipe for incredible inefficiency which I have seen first hand, truly - hundreds of engineers working on projects which are done much better by five engineers, working in a hygienic software environment.
I’ll devote another post into how you make a hygienic software environment.
I enjoyed talking about the benefits I’ve seen first hand, and my friend and mentor Jason Ford, a worthy post. Besides being the best leader and software architect ever, Jason Ford is a pleasure to report into, and also a lights out C++ coder. Jason, you are really one of the best in the business (I’ve been around!), and you deserve this post and more. Certainly in no small part responsible for making me the technologist I am today. At Jefferies, you redefined what is possible with focus, intensity, and ingenuity.