Thursday, 20 February 2014

The cost of complexity in software engineering is like the sound barrier. How can we break it?

In response to:

Q: Do successful programmers need to be introverted?
A: It depends.

One unpleasant consequence of network effects is that the cost of communication has a super-linear relationship with system complexity. Fred Brooks indicates that the communication overhead for large (or growing) teams can become significant enough to stop work in it's tracks. As the team grows beyond a certain point, the cost quickly shoots up to an infeasible level. By analogy with the sound barrier, I call this the communications barrier; because both present a seemingly insurmountable wall blocking our ability to  further improve our performance.

This analysis argues for small team sizes, perhaps as small as n=1. Clearly introversion is an asset under these circumstances.

However, irrespective of their innate efficiency, there are obvious limits to what a small team can produce. Building a system beyond a certain level of complexity requires a large team, and large teams need to break the communications barrier to succeed. Excellent, highly disciplined and highly effective communications skills are obviously very important under these circumstances, which calls for a certain type of (disciplined) extroversion; perhaps in the form of visionary leadership?

My intuition tells me that breaking the communications barrier is a matter of organization and detail-oriented thinking. Unfortunately I have yet to observe it being done both effectively and systematically by any of the organisations that I have yet worked for.

Has anybody else seen or worked with an organisation that has successfully broken the communications barrier? If so, how did they do it?