Solve The Right Problem, Win A Prize
2006-12-01
In order to solve a problem, you have to solve the right problem. Obvious really, but not so easy to do in practice, because the root cause is not always waving a red flag.
There has been some interesting discussion online about the design of the Windows shutdown dialog -- or lack of it -- and that had led to more discussion about why Microsoft can't seem to fix things like this and get other things like the Zune anywhere near right. Much blame has been laid on the processes and systems Microsoft has in place to control the gargantuan thing that is Windows, and this comment came to its defense:
The people who designed the source control system for Windows were *not* idiots. They were trying to solve the following problem:
- thousands of developers,
- promiscuous dependency taking between parts of Windows without much analysis of the consequences
--> with a single codebase, if each developer broke the build once every two years there would never be a Longhorn build (or some such statistic - I forget the actual number)
And there you have it. Microsoft is trying to solve a problem that it should never have had in the first place: thousands of developers (ten thousand actually, costing $2B a year). Why? Promiscuous dependency? Why?
Let's look at what this does in real life. From this Slashdot discussion: Why Vista Took So Long
I'd also like to sketch out how actual coding -- what there is of it -- works on the Windows team.
In small programming projects, there's a central repository of code. Builds are produced, generally daily, from this central repository. Programmers add their changes to this central repository as they go, so the daily build is a pretty good snapshot of the current state of the product.
In Windows, this model breaks down simply because there are far too many developers to access one central repository -- among other problems, the infrastructure just won't support it. So Windows has a tree of repositories: developers check in to the nodes, and periodically the changes in the nodes are integrated up one level in the hierarchy. At a different periodicity, changes are integrated down the tree from the root to the nodes. In Windows, the node I was working on was 4 levels removed from the root. The periodicity of integration decayed exponentially and unpredictably as you approached the root so it ended up that it took between 1 and 3 months for my code to get to the root node, and some multiple of that for it to reach the other nodes. It should be noted too that the only common ancestor that my team, the shell team, and the kernel team shared was the root.
So in addition to the above problems with decision-making, each team had no idea what the other team was actually doing until it had been done for weeks.
The end result of all this is what finally shipped: the lowest common denominator, the simplest and least controversial option.
Now contrast this to what happens Apple:
I used to work at Apple, in the OS and frameworks groups.
There is a master "train" for a release; projects that don't change are "forwarded" to that train, meaning no source changes are required. When a project needs to be submitted for a change for the new release, a new "view" is created for its specific changes. Every few days, a build is produced, sometimes using previously compiled bits from the old "train", sometimes its a full world build (which can take several days) but otherwise building all the latest submissions.
Then there's a fairly labor intensive "integration" phase where the built bits are all put on a box and booted. If a "quicklook" QA process shows that the build is hoarked, the integrator goes and pesters the submitters of the latest project that was submitted and gets them to fix it. (Some percentage of the time, the new code has exposed a bug elsewhere, regardless, the project that is the proximal cause of the failure is rolled back to the previous revision, it anticipation that all the projects that need to rev be submitted at once.)
The whole thing is set up through symlinks via NFS, so if you want to see the latest version of any piece of code in the system (modulus those projects that are "locked down" for security issues) you can just get your release name, append the build number, and you've got the source code, symbol'd binaries and build log *for any release* at your fingertips.
When a new build comes out, you just do a clean install. It takes about two hours on the internal network, so typically you pull the disk image and slam it to a firewire drive, (usually, you can bum a disk with the image already grabbed from a teammate) and do a full install in 15 minutes. I can't imagine having to spend a day (as some other posted mentioned) setting up a machine...
Most projects have 3 or 4 contributors. In many cases, and entire framework is the responsibility of a single person (and he or she may actually own several small frameworks.) Lots of small projects produce cleaner interfaces that lead to fewer dependencies. (Of course there are dependencies, and circular ones, but these are kept to a minimum.) Projects are encouraged to use public API from other projects, rather than SPI or other project internals. If there's something useful enough for some other project to use, its first made into SPI for internal consumption, with the goal that developers will eventually be able to use it through a public API.
Most groups don't have dedicated QA by the way - the engineers are responsible for their code, and everyone is generally just very smart about what they're doing.
As to this start menu problem: the entire UI team is about 5 individuals, plus Steve Jobs and Scott Forstall - and they're likely to say "Thats fucking stupid, just do this" and boom(tm), the decision has been made the product ships, and life goes on.
It looks to me like Apple has solved Microsoft's problem by simply not having the problem: many fewer developers, many fewer dependencies. And of course Apple has its not-so-secret weapon: Objective C. What this means is that Apple has the capability to scale much further than where they are now without getting crippled by complexity and bloat. And because they are actually in control of their products (rather than by large customers or record companies), they can maintain this situation as they scale.
As I write, Apple is a $78.3B company and Microsoft a $288B company. I reckon that Microsoft has topped out, so Apple has to grow to less than four times its current size (3.6 times) to have a market capitalization greater than Microsoft. At 20% per year growth, that's only seven years. And during that time, they will eat a huge piece of the PC business away from the current players.
I'll make a wild prediction here: AAPL will be at 160 by the end of 2007. Ninety dollars a share seemed impossible less than a year ago, and yet the price today is 91.66, so it's not that crazy an idea. Why so high if only 20% growth? Because by the end of 2007 Apple will no longer be viewed as a niche player. Once their full potential is appreciated, there will be a huge rush to get on board the stock. Not a bad prize to win.
There has been some interesting discussion online about the design of the Windows shutdown dialog -- or lack of it -- and that had led to more discussion about why Microsoft can't seem to fix things like this and get other things like the Zune anywhere near right. Much blame has been laid on the processes and systems Microsoft has in place to control the gargantuan thing that is Windows, and this comment came to its defense:
The people who designed the source control system for Windows were *not* idiots. They were trying to solve the following problem:
- thousands of developers,
- promiscuous dependency taking between parts of Windows without much analysis of the consequences
--> with a single codebase, if each developer broke the build once every two years there would never be a Longhorn build (or some such statistic - I forget the actual number)
And there you have it. Microsoft is trying to solve a problem that it should never have had in the first place: thousands of developers (ten thousand actually, costing $2B a year). Why? Promiscuous dependency? Why?
Let's look at what this does in real life. From this Slashdot discussion: Why Vista Took So Long
I'd also like to sketch out how actual coding -- what there is of it -- works on the Windows team.
In small programming projects, there's a central repository of code. Builds are produced, generally daily, from this central repository. Programmers add their changes to this central repository as they go, so the daily build is a pretty good snapshot of the current state of the product.
In Windows, this model breaks down simply because there are far too many developers to access one central repository -- among other problems, the infrastructure just won't support it. So Windows has a tree of repositories: developers check in to the nodes, and periodically the changes in the nodes are integrated up one level in the hierarchy. At a different periodicity, changes are integrated down the tree from the root to the nodes. In Windows, the node I was working on was 4 levels removed from the root. The periodicity of integration decayed exponentially and unpredictably as you approached the root so it ended up that it took between 1 and 3 months for my code to get to the root node, and some multiple of that for it to reach the other nodes. It should be noted too that the only common ancestor that my team, the shell team, and the kernel team shared was the root.
So in addition to the above problems with decision-making, each team had no idea what the other team was actually doing until it had been done for weeks.
The end result of all this is what finally shipped: the lowest common denominator, the simplest and least controversial option.
Now contrast this to what happens Apple:
I used to work at Apple, in the OS and frameworks groups.
There is a master "train" for a release; projects that don't change are "forwarded" to that train, meaning no source changes are required. When a project needs to be submitted for a change for the new release, a new "view" is created for its specific changes. Every few days, a build is produced, sometimes using previously compiled bits from the old "train", sometimes its a full world build (which can take several days) but otherwise building all the latest submissions.
Then there's a fairly labor intensive "integration" phase where the built bits are all put on a box and booted. If a "quicklook" QA process shows that the build is hoarked, the integrator goes and pesters the submitters of the latest project that was submitted and gets them to fix it. (Some percentage of the time, the new code has exposed a bug elsewhere, regardless, the project that is the proximal cause of the failure is rolled back to the previous revision, it anticipation that all the projects that need to rev be submitted at once.)
The whole thing is set up through symlinks via NFS, so if you want to see the latest version of any piece of code in the system (modulus those projects that are "locked down" for security issues) you can just get your release name, append the build number, and you've got the source code, symbol'd binaries and build log *for any release* at your fingertips.
When a new build comes out, you just do a clean install. It takes about two hours on the internal network, so typically you pull the disk image and slam it to a firewire drive, (usually, you can bum a disk with the image already grabbed from a teammate) and do a full install in 15 minutes. I can't imagine having to spend a day (as some other posted mentioned) setting up a machine...
Most projects have 3 or 4 contributors. In many cases, and entire framework is the responsibility of a single person (and he or she may actually own several small frameworks.) Lots of small projects produce cleaner interfaces that lead to fewer dependencies. (Of course there are dependencies, and circular ones, but these are kept to a minimum.) Projects are encouraged to use public API from other projects, rather than SPI or other project internals. If there's something useful enough for some other project to use, its first made into SPI for internal consumption, with the goal that developers will eventually be able to use it through a public API.
Most groups don't have dedicated QA by the way - the engineers are responsible for their code, and everyone is generally just very smart about what they're doing.
As to this start menu problem: the entire UI team is about 5 individuals, plus Steve Jobs and Scott Forstall - and they're likely to say "Thats fucking stupid, just do this" and boom(tm), the decision has been made the product ships, and life goes on.
It looks to me like Apple has solved Microsoft's problem by simply not having the problem: many fewer developers, many fewer dependencies. And of course Apple has its not-so-secret weapon: Objective C. What this means is that Apple has the capability to scale much further than where they are now without getting crippled by complexity and bloat. And because they are actually in control of their products (rather than by large customers or record companies), they can maintain this situation as they scale.
As I write, Apple is a $78.3B company and Microsoft a $288B company. I reckon that Microsoft has topped out, so Apple has to grow to less than four times its current size (3.6 times) to have a market capitalization greater than Microsoft. At 20% per year growth, that's only seven years. And during that time, they will eat a huge piece of the PC business away from the current players.
I'll make a wild prediction here: AAPL will be at 160 by the end of 2007. Ninety dollars a share seemed impossible less than a year ago, and yet the price today is 91.66, so it's not that crazy an idea. Why so high if only 20% growth? Because by the end of 2007 Apple will no longer be viewed as a niche player. Once their full potential is appreciated, there will be a huge rush to get on board the stock. Not a bad prize to win.
The Bagelturf site welcomes Donations of any size