Intelligence is a defining human characteristic, so it is somewhat doubtful that we would ever be able to pin down what it is precisely enough to make an artificial version of it ... we will always try to define and redefine the term "intelligence" in such a manner that humans remain separated and protected as a separate and distinct class of entities ... our collective ego demands no less.
However, if we put our collective ego to one side, we can surmise that intelligent behaviour consists of some general purpose learning and behaviour-directing mechanisms sitting on top of a whole heap of special-purpose sensorimotor mechanisms.
Engineering an artificial system to emulate some or all of that behaviour is a monumental engineering task. Irrespective of the fundamental breakthroughs in learning, generalisation and planning that may or may not be required, we are still left with an exceedingly large and complex software engineering challenge.
In my mind, this represents the primary obstacle to the development of a true Artificial Intelligence: Not better science, but better software engineering practices and (particularly) better software engineering tools to help manage, visualise, understand and communicate complicated software systems.
Now, this is an interesting problem, because software engineering at a large scale is much less concerned with hard technical challenges and much more concerned with "soft" human challenges of communication and politics. The same can be said for any engineering challenge where the complexity of the subject matter, combined with it's lack of visibility makes communication and documentation a key technical challenge.
Let us look at the communication challenge first: One thing that is becoming more apparent in our increasingly information-rich environment is that communication is constrained much more by the time-management and motivation-management of the reader than by the availability of the information.
In other words, it is not enough just to make information available, you have to manage the relationship between the reader and the information as well; so that information is not just dropped in front of the reader, but the relationship between the reader and the information is managed to support learning and effective decision making also.
How might one achieve this?
Friday, 10 May 2013
Saturday, 4 May 2013
Poverty and drugs
In response to this article:
http://www.ft.com/cms/s/2/f5763610-b2bb-11e2-8540-00144feabdc0.html#axzz2SLQSbvxr
(And in particular, a debate in the comments section about soft drinks being luxuries or necessities).
When you are under financial pressure, your ability to control your own environment is reduced, together with your ability to control the stressors in your environment. You have fewer opportunities and less freedom to "step back from it all" to reduce your personal pressure/stress loading. In addition, your reserves of self-esteem are eroded, and your self-image is loaded with negativity, so you naturally turn to the only real alternative to help you control your mental state: drugs, alcohol, caffeine, sugar. Inner strength to survive the grind is not something that is magically available on demand. You need something to give you that strength. Low cost, nutritious food, laced with amphetamines like adderall and ritalin would be a winning product in a depressed economy, where low-wage workers need to pull long shifts to make ends meet and to pay the economic rents with which they are burdened.
http://www.ft.com/cms/s/2/f5763610-b2bb-11e2-8540-00144feabdc0.html#axzz2SLQSbvxr
(And in particular, a debate in the comments section about soft drinks being luxuries or necessities).
When you are under financial pressure, your ability to control your own environment is reduced, together with your ability to control the stressors in your environment. You have fewer opportunities and less freedom to "step back from it all" to reduce your personal pressure/stress loading. In addition, your reserves of self-esteem are eroded, and your self-image is loaded with negativity, so you naturally turn to the only real alternative to help you control your mental state: drugs, alcohol, caffeine, sugar. Inner strength to survive the grind is not something that is magically available on demand. You need something to give you that strength. Low cost, nutritious food, laced with amphetamines like adderall and ritalin would be a winning product in a depressed economy, where low-wage workers need to pull long shifts to make ends meet and to pay the economic rents with which they are burdened.
Sunday, 14 April 2013
Procyclic
The Carnot cycle provides us with a fun little analogy that we can abuse and overextend in any number of ways:-
Energy is turned into work by harnessing and controlling a cycle of heating and cooling; expansion and contraction; increasing and decreasing pressure.
One side without the other provides only transient and worthless motion.
Brought together in opposition; made to alternate; managed and controlled in a reciprocating framework, these transient forces may be tamed and made to perform useful and beneficial work over an extended period of time.
Creative Cycles
In the same way, our innovative capacity does not produce new ideas in a single unbroken stream of creativity, but rather in fits and starts.
Contemplation and intensity must be brought together in alternating opposition:
Contemplation so that the mind may range wide, gathering inputs from disparate sources as one might pick blackberries from a brambly hedgerow.
Intensity so that the mind may delve deeply into the detail, working through the consequences of each decision; picking apart the idea to find it's essence, and turning the idea over and over to find the right terminology the right language, the right conceptual framework within which it may be best expressed
and exploited.
One phase without the other is unproductive. Bring the two together in alternation, and you get productive work. Manage that alternation, and you can start to exploit our real potential for creativity.
Communication Cycles
There has been some discussion about the utilization and design of shared workspaces to maximise spontaneous communication (prompted by Marissa Meyer's edict banning remote working at Yahoo). In my mind, this focuses on only half of the solution: perhaps unsurprisingly, in light of the theme of this piece, I believe that communication and isolation must alternate.
When working in a totally isolated manner, one cannot determine what is needed, one does not know what is important. When sitting in the middle of a bustling open plan office, one often knows what is important, but one lacks the space and calm isolation needed to actually do anything about it. It is noteworthy that many effective organisations mandate specific periods; specific opportunities for interaction, typically involving food or drink, perhaps we could also mandate specific periods for isolation and calm introspection, a "quiet time" each day free from phone calls, emails and other forms of communication.
This may be unnecessary, however, since the ubiquitous headphone-wearing developers sitting in an open-plan-office provides a very flexible compromise, offering opportunities both for focussed work and for collaboration.
Economic Cycles
As with the small, so with the large. "The era of boom and bust economics is over" was a statement that ran not only counter to common sense, but possibly counter to some sort of law of statistical physics.
To believe that we have any real positive control over the economy is a dangerous conceit. A vague and sloppy influence, perhaps, but not control in any recognisable meaning of the word.
The presence of investment cycles for individual businesses as well as individual people is a well understood fact, as is the web of interactions that exists between those economic actors, and (so I would suppose) the presence of effects that promote the synchronisation of these individual cycles into macroeconomic feedback loops.
Why attempt to eliminate these macro-scale cycles by controlling government spending? Why not, instead, as our analogy suggests, control their amplitude and frequency so that the harm that they do is minimised.
Indeed, as Nasim Taleb points out, a small "bust" can fend off a larger and more damaging one. Targeted cyclical and perhaps aperiodic variations in the tax burden might help stress, (or threaten stress) economic actors in a manner that promotes decoupling and desynchronisation, and inhibits catastrophic modes-of-failure, but, as stated above, it would be dangerously arrogant to assume that we really know what is going on, and what the effect of our actions is, or will be.
So we have to exert control, not in a direct and forceful manner, but in a way that tends to the ecosystem, but does not seek to dominate it.
Anyway, I am straying dangerously far from my area of expertise, and my intuition oft runs astray, so let me leave things here before I go too far off the beaten track.
Energy is turned into work by harnessing and controlling a cycle of heating and cooling; expansion and contraction; increasing and decreasing pressure.
One side without the other provides only transient and worthless motion.
Brought together in opposition; made to alternate; managed and controlled in a reciprocating framework, these transient forces may be tamed and made to perform useful and beneficial work over an extended period of time.
Creative Cycles
In the same way, our innovative capacity does not produce new ideas in a single unbroken stream of creativity, but rather in fits and starts.
Contemplation and intensity must be brought together in alternating opposition:
Contemplation so that the mind may range wide, gathering inputs from disparate sources as one might pick blackberries from a brambly hedgerow.
Intensity so that the mind may delve deeply into the detail, working through the consequences of each decision; picking apart the idea to find it's essence, and turning the idea over and over to find the right terminology the right language, the right conceptual framework within which it may be best expressed
and exploited.
One phase without the other is unproductive. Bring the two together in alternation, and you get productive work. Manage that alternation, and you can start to exploit our real potential for creativity.
Communication Cycles
There has been some discussion about the utilization and design of shared workspaces to maximise spontaneous communication (prompted by Marissa Meyer's edict banning remote working at Yahoo). In my mind, this focuses on only half of the solution: perhaps unsurprisingly, in light of the theme of this piece, I believe that communication and isolation must alternate.
When working in a totally isolated manner, one cannot determine what is needed, one does not know what is important. When sitting in the middle of a bustling open plan office, one often knows what is important, but one lacks the space and calm isolation needed to actually do anything about it. It is noteworthy that many effective organisations mandate specific periods; specific opportunities for interaction, typically involving food or drink, perhaps we could also mandate specific periods for isolation and calm introspection, a "quiet time" each day free from phone calls, emails and other forms of communication.
This may be unnecessary, however, since the ubiquitous headphone-wearing developers sitting in an open-plan-office provides a very flexible compromise, offering opportunities both for focussed work and for collaboration.
Economic Cycles
As with the small, so with the large. "The era of boom and bust economics is over" was a statement that ran not only counter to common sense, but possibly counter to some sort of law of statistical physics.
To believe that we have any real positive control over the economy is a dangerous conceit. A vague and sloppy influence, perhaps, but not control in any recognisable meaning of the word.
The presence of investment cycles for individual businesses as well as individual people is a well understood fact, as is the web of interactions that exists between those economic actors, and (so I would suppose) the presence of effects that promote the synchronisation of these individual cycles into macroeconomic feedback loops.
Why attempt to eliminate these macro-scale cycles by controlling government spending? Why not, instead, as our analogy suggests, control their amplitude and frequency so that the harm that they do is minimised.
Indeed, as Nasim Taleb points out, a small "bust" can fend off a larger and more damaging one. Targeted cyclical and perhaps aperiodic variations in the tax burden might help stress, (or threaten stress) economic actors in a manner that promotes decoupling and desynchronisation, and inhibits catastrophic modes-of-failure, but, as stated above, it would be dangerously arrogant to assume that we really know what is going on, and what the effect of our actions is, or will be.
So we have to exert control, not in a direct and forceful manner, but in a way that tends to the ecosystem, but does not seek to dominate it.
Anyway, I am straying dangerously far from my area of expertise, and my intuition oft runs astray, so let me leave things here before I go too far off the beaten track.
Saturday, 13 April 2013
Contrahumanity
Our human instincts do not necessarily act in our best interests, even in a statistical sense. Our need to do the right thing, and even our need to survive, brings our will into inevitable conflict with our humanity. The challenge is extraordinary: given our limited intellectual capacity, it is far from obvious that the exercise is anything other than futile.
Friday, 5 April 2013
Thursday, 28 March 2013
Wednesday, 27 March 2013
On the importance of attention-to-detail in software tools development.
As the Apple example demonstrates, attention to detail, in aggregate, results in a superior product, which enables you to justify charging a premium price.
This truism is particularly apt when it comes to tools, because the affordances and micro-features that one's day-to-day tools offer have a huge impact on the way that we do our day-to-day job. The extent to which we developers follow the path of least resistance is, in many ways, quite sad, but this most human of characteristics places incredible power in the hands of the tool-makers. Best practice is often defined by the tools that we use. As a crude example, witness how Hungarian notation fell out of favour as soon as popular IDEs started to display "type hints" when the mouse hovered over the variable.
Make it slightly easier to do something, or to arrange text in a particular way, and developers will change their behaviour in response.
For many years (2001-2011) I used TextPad on windows, and came to particularly appreciate it's block select mode when managing vertically aligned characters. The shortcut for this feature (Ctrl+Q, B) has been burned irreversibly into my motor cortex. I used this feature not only to manage vertically-aligned assignment statements, but also to quickly add blocks of end-of-line comments, and to support a declarative programming style that relied heavily on in-source tables of parameters & other data.
This declarative approach to software construction had a particularly pleasing synergy with the MATLAB language and the uncomplicated, linear loop-free aesthetic which is made possible by it's impressive collection of array-oriented library functions.
More recently, (2011-2013) I have been using Sublime Text as my editor-of-choice. (As well as a little vim, when I have no choice). Whilst the multiple selection feature of Sublime text is super-duper-awesome in it's own way, I still miss TextPad's block select feature, particularly it's ability to automatically insert whitespace at the end of lines, allowing one to easily reclaim blocks of space to the right of the logic for descriptive commentary. (80 column limits be damned as an onions-in-the-varnish anachronistic throwback).
Anyway, seemingly small things, but still important: What seem like trivial affordances really do shape the way that one thinks and works. For the past year I switched away from Textpad + MATLAB towards Sublime Text & vim + Python, and the combined strictures of PEP8 & PyLint, together with the subtly different affordances of the Sublime Text vs. TextPad steered my development style away from the declarative parameter-driven approach towards a more "traditional" programming style, to the moderate detriment of the "quality" of the applications that I produced.
Despite Python's numerous, obvious and justifiable claims to superiority, there exists an ineffable and emergent quality that language and tools together supply, and a quality that was not (quite) captured in the tools that I was using for the past year.
Another observation: When moving the caret quickly along a line of text using the arrow keys with the "ctrl" key pressed down, the cursor in the MATLAB editor stops very slightly more frequently than other editors, resulting in a slightly "sluggish" feel to the navigation. Paying attention to matters such as this is the equivalent, to a tool-maker, of the auto-maker paying attention to the "thunk" noise that the car door makes when it closes. A feature that operates almost on a subliminal level to create the impression of quality, and, to the tool-user, the feeling of "floating" through the text file making changes, rather than the feeling of wading through a muddy swamp.
Another observation: Little bits of automation to make life easier. When I highlight a word in Sublime Text and press the open-parentheses "(" key, the word is surrounded in parentheses. When I do the same in the MATLAB editor, the word is deleted and replaced by the open parenthesis character.
I am a craftsman. I want to love my tools. MATLAB has elements of greatness, but a large number of flaws as well. Some of these flaws cannot be addressed without making the tool something other than what it is.
Fair enough.
But some can be addressed with passion, attention-to-detail, design sense, and strong, detailed technical leadership.
This truism is particularly apt when it comes to tools, because the affordances and micro-features that one's day-to-day tools offer have a huge impact on the way that we do our day-to-day job. The extent to which we developers follow the path of least resistance is, in many ways, quite sad, but this most human of characteristics places incredible power in the hands of the tool-makers. Best practice is often defined by the tools that we use. As a crude example, witness how Hungarian notation fell out of favour as soon as popular IDEs started to display "type hints" when the mouse hovered over the variable.
Make it slightly easier to do something, or to arrange text in a particular way, and developers will change their behaviour in response.
For many years (2001-2011) I used TextPad on windows, and came to particularly appreciate it's block select mode when managing vertically aligned characters. The shortcut for this feature (Ctrl+Q, B) has been burned irreversibly into my motor cortex. I used this feature not only to manage vertically-aligned assignment statements, but also to quickly add blocks of end-of-line comments, and to support a declarative programming style that relied heavily on in-source tables of parameters & other data.
This declarative approach to software construction had a particularly pleasing synergy with the MATLAB language and the uncomplicated, linear loop-free aesthetic which is made possible by it's impressive collection of array-oriented library functions.
More recently, (2011-2013) I have been using Sublime Text as my editor-of-choice. (As well as a little vim, when I have no choice). Whilst the multiple selection feature of Sublime text is super-duper-awesome in it's own way, I still miss TextPad's block select feature, particularly it's ability to automatically insert whitespace at the end of lines, allowing one to easily reclaim blocks of space to the right of the logic for descriptive commentary. (80 column limits be damned as an onions-in-the-varnish anachronistic throwback).
Anyway, seemingly small things, but still important: What seem like trivial affordances really do shape the way that one thinks and works. For the past year I switched away from Textpad + MATLAB towards Sublime Text & vim + Python, and the combined strictures of PEP8 & PyLint, together with the subtly different affordances of the Sublime Text vs. TextPad steered my development style away from the declarative parameter-driven approach towards a more "traditional" programming style, to the moderate detriment of the "quality" of the applications that I produced.
Despite Python's numerous, obvious and justifiable claims to superiority, there exists an ineffable and emergent quality that language and tools together supply, and a quality that was not (quite) captured in the tools that I was using for the past year.
Another observation: When moving the caret quickly along a line of text using the arrow keys with the "ctrl" key pressed down, the cursor in the MATLAB editor stops very slightly more frequently than other editors, resulting in a slightly "sluggish" feel to the navigation. Paying attention to matters such as this is the equivalent, to a tool-maker, of the auto-maker paying attention to the "thunk" noise that the car door makes when it closes. A feature that operates almost on a subliminal level to create the impression of quality, and, to the tool-user, the feeling of "floating" through the text file making changes, rather than the feeling of wading through a muddy swamp.
Another observation: Little bits of automation to make life easier. When I highlight a word in Sublime Text and press the open-parentheses "(" key, the word is surrounded in parentheses. When I do the same in the MATLAB editor, the word is deleted and replaced by the open parenthesis character.
I am a craftsman. I want to love my tools. MATLAB has elements of greatness, but a large number of flaws as well. Some of these flaws cannot be addressed without making the tool something other than what it is.
Fair enough.
But some can be addressed with passion, attention-to-detail, design sense, and strong, detailed technical leadership.
Confidence and Competence and Information Overload
The internet is a wonderful thing.
Every problem has a thousand solutions: documented; discussed; dissected.
This sumptuous surfeit of suitable solutions seems satisfactory; superb even, save for one small snag:
My effective competence, my ability to work quickly and solve problems, is driven largely by my confidence in my own abilities, my own evaluation of the completeness of my knowledge versus the task at hand.
When I was younger, the arrogance of youth made this easy. I plowed on ahead, ignorant of my own ignorance, I performed, gained accolades, and life was good.
However, faced with ever rising flood-waters issuing from the fire-hose of information that is the internet today, it is all too easy to start comparing what I know with what I could know, which, expressed as a ratio, is always going to be depressingly close to zero. With self-confidence thus undermined, the inevitable consequence of this is for me to loose sight of the fact that I have enough information at hand to get on with it, without worry or concern, and to get dragged down into the rabbit-hole.
This problem is compounded when one tries to pick up a new tool or technology, an activity that (given the pace of technological advance) one needs to do more-or-less continuously.
Now, I love learning, but I cannot resist the temptation to compare my knowledge and level of expertise in the latest tool to my knowledge and level of expertise in the last, and, in the worst-case-scenario one or two weeks worth of experience is never going to compare favourably with six or seven years.
In situations such as this, I have to keep telling myself to HTFU and demilquetoastify my attitude.
Every problem has a thousand solutions: documented; discussed; dissected.
This sumptuous surfeit of suitable solutions seems satisfactory; superb even, save for one small snag:
My effective competence, my ability to work quickly and solve problems, is driven largely by my confidence in my own abilities, my own evaluation of the completeness of my knowledge versus the task at hand.
When I was younger, the arrogance of youth made this easy. I plowed on ahead, ignorant of my own ignorance, I performed, gained accolades, and life was good.
However, faced with ever rising flood-waters issuing from the fire-hose of information that is the internet today, it is all too easy to start comparing what I know with what I could know, which, expressed as a ratio, is always going to be depressingly close to zero. With self-confidence thus undermined, the inevitable consequence of this is for me to loose sight of the fact that I have enough information at hand to get on with it, without worry or concern, and to get dragged down into the rabbit-hole.
This problem is compounded when one tries to pick up a new tool or technology, an activity that (given the pace of technological advance) one needs to do more-or-less continuously.
Now, I love learning, but I cannot resist the temptation to compare my knowledge and level of expertise in the latest tool to my knowledge and level of expertise in the last, and, in the worst-case-scenario one or two weeks worth of experience is never going to compare favourably with six or seven years.
In situations such as this, I have to keep telling myself to HTFU and demilquetoastify my attitude.
Tuesday, 19 February 2013
Batteries Included
Software developers seem to be significantly more productive today than they were even a few years ago. What has happened? Have techniques improved? Have skill levels increased? Are best-practices being more closely adhered to?
One cannot easily isolate any one factor as bearing sole responsibility for the performance improvements, but I suspect that a great deal of our gratitude should be directed towards improvements in the availability, quality and performance of software libraries, together with growing communities around those libraries, promoting education and adoption in the wider engineering community.
After all, the one thing that makes software development so different from other disciplines are the tremendous cost savings available through reuse. The less logic your organization is responsible for, the less the expenditure on development, documentation and maintenance. The main expense incurred when using tools and libraries is in education and training ... this is perhaps not something that we as an industry have fully adjusted to:- neither in the way that we (as professionals) specialize; nor in the way that we (as organizations) recruit new talent or nurture existing talent.
In fact, the notion of talent as something that is exclusively innate and inimitable is a dangerous one. When one makes an experienced hire, the commodity that one is purchasing is (primarily) knowledge, rather than the commission of work/effort to be expended. The knowledge that is required to not have to do the work, to be more precise. Perhaps we need to introduce new work and employment patterns beyond the traditional notions of contractor and perm employee?
Tuesday, 15 January 2013
Personal Statement
My passion for machine vision, machine learning and statistical pattern recognition is longstanding, having started over 10 years ago and continuing today.
Most of my undergraduate AI degree was oriented towards logic, theorem proving, and computational linguistics, which was fascinating in it's own right, but did not strike me as a particularly realistic or pragmatic way of dealing with the messiness and complexity of the real world. As a result, I latched on to the (at the time) less mainstream "soft" computing approaches with enthusiasm, devouring the content of the machine learning, machine vision and neural networks modules avidly. I saw these approaches as a pragmatic alternative to the hard and inflexible grammar-based approaches to Natural Language Processing espoused by the main body of the department.
This view of machine learning as a pragmatic tool, at odds with ivory-tower academicism has stuck with me ever since, even as the subject has become more mainstream (and more academic and mathematically sophisticated). As a result, I tend to focus on simple techniques that work, rather than techniques which demonstrate mathematical chops and academic sophistication. I am fortunate in this regard, because, paradoxically, the solutions to difficult problems are often conceptually simpler and mathematically less sophisticated than the "optimum" solutions to simple problems. Perhaps a little bit of the Yorkshire/Lancashire culture of engineering pragmatism rubbed off on me during my time in Manchester.
Another thing that was dawning on me as I finished my undergraduate degree was the importance of scale. As I attempted to find datasets for my hobby projects, (Far harder back then than today), I began to develop suspicions that scale, rather than any qualitative leap in understanding, was going to be a key factor in the development of genuinely interesting artificial intelligence techniques. From this came my interest in machine vision, which I saw as a key "gateway" technique for the collection of data -- to help the machine build an understanding of the world around it and to "bootstrap" itself to a more interesting level.
I was lucky with my first employer, Cambridge Research Systems, where I had the opportunity to work with some very talented people, both within the company and across our customer community. From that experience, and the abortive neuroscience PhD that I started, I learned a lot about the neuroscience of biological visual systems, particularly the older, lower-level pathways that go, not to the primary visual cortex, but to the evolutionarily older "reptilian" parts of the brainstem. In contrast with the "general purpose" and "reconfigurable" nature of the cortex, these older pathways consist of a large number of (less flexible) special-purpose circuits handling things like eye movements and attention-directing mechanisms. Crucially, these lower-level circuits enable our visual system to stabilise, normalise and "clean" the data that we present to our higher-level cortical mechanisms. This insight crosses across well to more commercial work, where the importance of solid groundwork (data quality, normalization and sampling) can make or break a machine learning implementation. I was also fortunate enough to pick up some signal processing and FIR filter design fundamentals - as I was writing software to process biological time-series signals (EOG) to identify and isolate events like saccades and blinks.
At around about this time, I was starting to become aware of the second important thread in my intellectual development: The incredible slowness of software development, and the difficulty and cost that we would incur trying to implement the large number of these lower level stabilization mechanisms that would be required.
I left Cambridge Research Systems specifically to broaden my real-world, commercial software development experience, working at a larger scale than was possible at CRS. Again, I was lucky to find a role with Sophos, where I learned a great deal from a large group of very talented C++ developers doing Test Driven Development in the highest-functioning Agile team I have yet encountered. Here, I started to think seriously about the role of communication and human factors in software development, as well as the role that tools play in guiding development culture, always with an eye to how we might go about developing those special purpose data processing functions.
Following a relocation closer to London (for family reasons), I left Sophos and started working for Thales Optronics. Again fortunate, I found myself working on (very) large scale machine vision applications. Here, during a joyous three year period, I was able to put much of my previous intellectual development and thinking into practice, developing not only the in-flight signal processing, tracking and classification algorithms, but more significantly, the petabyte-scale data handling systems needed to train, test and gain confidence in them. In addition to the technical work, I worked to encourage a development culture conducive to the development of complex systems. This was the most significant, successful and rewarding role I have had to date.
Unfortunately, budgetary constraints led Thales to close their office in Staines, and rather than transferring to the new office, I chose to "jump ship" and join Fidelity Asset Managers in the City of London, partly in an attempt to defeat some budgetary constraints of my own, and partly out of an awareness of the potential non-transferability of defense industry expertise, made more pressing by an impending overseas relocation.
At Fidelity, I used my knowledge of the MATLAB distributed computing toolbox to act as the High-Performance Computing expert in the "quant" team. I gained exposure to a very different development culture, and learned a lot about asset management and quantitative investing, gaining some insight into the accounting and management factors that drive development culture in other organizations. I particularly valued my exposure to the insights that Fidelity had, as an institutional investor, into what makes a successful organization, as well as it's attempts to apply those insights to itself.
At Fidelity, I used my knowledge of the MATLAB distributed computing toolbox to act as the High-Performance Computing expert in the "quant" team. I gained exposure to a very different development culture, and learned a lot about asset management and quantitative investing, gaining some insight into the accounting and management factors that drive development culture in other organizations. I particularly valued my exposure to the insights that Fidelity had, as an institutional investor, into what makes a successful organization, as well as it's attempts to apply those insights to itself.
Finally, in 2011, my family's long expected overseas posting came. Yet again we were incredibly lucky, and got to spend a wonderful year-and-a-half living in the middle of New York city. I was fortunate, and managed to get a job at an incredible Silicon Alley startup, EveryScreen Media, which was riding the wave of interest in mobile advertising that was just beginning to ramp up in 2011 and 2012. Again, finding myself working with incredibly talented and passionate colleagues, I was given the opportunity to broaden my skills once again, picking up Python and Unix development skills, becoming immersed in (back-end) web development, building out the data science infrastructure in an early-stage startup. From this year and a half or so, I particularly value what I learned about how to develop large scale, (scaleable) distributed real-time data processing systems systems and the effective use use of modern internet "web" technology.
Now, back in the UK, I am in search of the next step on my journey of learning and discovery. My focus is, and remains, on the pragmatics of developing complex statistical data processing systems, on how to create and curate large data-sets, how to integrate them into the development process, so that continuous integration and continuous testing, visualisation and monitoring help the development team to understand and communicate the system that they are building, as well as the data that feeds it; to respond to unexpected behaviors, and to steer the product and the project to success and, moreover, to help ensure that the organization remains rightly confident in the team and in the system.
Thursday, 10 January 2013
A Network Model for Interpersonal Communication
Modeling interpersonal communication within an organization as a network of reconfigurable topology composed of high capacity data stores connected by limited bandwidth communication channels.
The Model:
The amount that we know on any given topic of interest vastly outweighs our practical ability to communicate that information in a reasonable time-frame. We simply do not have the time or the available bandwidth to communicate everything that we need to in the detail that the subject deserves. Our model reflects this - The data storage capacity at each node is immense, and contrasts sharply with the exceedingly limited bandwidth available for communication between nodes. The difference between the two is many orders of magnitude in size. For a visual analogy, we should not look to buckets connected by hosepipes, but rather half-million ton supertankers connected by thin cocktail straws. Transmitting even a gallon of knowledge is a challenge.
Chinese Whispers:
When communicating with distant nodes with messages routed through intermediary nodes, the information being transmitted is compressed to an incredibly high degree with a very lossy and low-quality compression algorithm. The poor quality of the communications channel is particularly evident when the network encompasses a diverse range of backgrounds, cultures and terminological-linguistic subtypes. In many such cases the intent of the message can easily be inverted as relevant details are either dropped or misinterpreted in transmission.
Systematic Factors impacting efficacy of communication:
The options available for compression are greater when two neighboring nodes already have a great deal in common, where shared datasets, terminology, mental models, and approaches to communication can be used to elide parts of the message, reducing bandwidth requirements, and allowing for communication that is both more reliable and more rapid. As a result of this, communication within an organization that has a strong, unified "culture" (common knowledge, terminology and practices) will be far more effective than communication within an organization that has a less cohesive "culture", purely because the options for message compression are greater, irrespective of any other measures that the organization might put in place to improve the available bandwidth. It is worth noting that, whilst this does improve the situation considerably, the problem itself is fundamental and always presents a significant challenge.
Organizational Optimization for Effective Communication:
Given that there is nothing inherently fixed about the topology of the communications network within which we are embedded, one simple response to this problem is to remove intermediary steps from source to destination nodes, and to allow the source node to connect directly with the destination node, permitting a direct and relatively high bandwidth exchange. This is even more effective if the exchange is bidirectional, permitting in-situ error correction. This argues for a collaborative approach to organization - with technical experts communicating with one another directly, with no intermediary communications or management specialists.
Another approach is to optimize for effective compression of the messages being transmitted through the network. As noted above, this relies on a common terminology, a common knowledge base, and a common set of practices and approaches to problems. In other words, most of the communication is moved out-of-band -- communicated though various channels prior to the point where it is actually needed. Again, this is well aligned with the collaborative approach to organization - where the technical experts within an organization continually educate one another on their own area of expertise so that when they need to communicate quickly, they can do so both effectively and reliably.
A Common Antipattern: Spin Doctors and Office Politicians:
Of course, the approaches outlined above are not the only solution to this fundamental problem. However, I argue that at least some of these approaches are anti-patterns, detrimental to the long term health and development of the organization.
Many individuals craft the messages they send extremely carefully to minimize the probability of corruption or misinterpretation en route, partly by reducing the information content of the message, and partly by crafting the emotional tone to remove ambiguity. This process is colloquially known as "spin", or "crafting the message".
In some situations this approach may even be appropriate, for example: where the network is large and cannot be reconfigured so that messages must be transmitted over more than one "hop"; where the environment for communication is particularly disadvantageous, with little common culture, terminology, or background technical knowledge; and finally, where long-term systematic improvements must be subordinated to short-term goals.
However, there are a couple of significant drawbacks to this approach. Firstly, by restricting knowledge transfer, opportunities to grow a base of common knowledge and understanding are squandered. Secondly, this approach is in direct conflict with cultural norms that emphasize honesty and transparency in interpersonal communication, the adherence to which builds a basis of trust and mutual understanding that further enhances communication.
The Model:
The amount that we know on any given topic of interest vastly outweighs our practical ability to communicate that information in a reasonable time-frame. We simply do not have the time or the available bandwidth to communicate everything that we need to in the detail that the subject deserves. Our model reflects this - The data storage capacity at each node is immense, and contrasts sharply with the exceedingly limited bandwidth available for communication between nodes. The difference between the two is many orders of magnitude in size. For a visual analogy, we should not look to buckets connected by hosepipes, but rather half-million ton supertankers connected by thin cocktail straws. Transmitting even a gallon of knowledge is a challenge.
Chinese Whispers:
When communicating with distant nodes with messages routed through intermediary nodes, the information being transmitted is compressed to an incredibly high degree with a very lossy and low-quality compression algorithm. The poor quality of the communications channel is particularly evident when the network encompasses a diverse range of backgrounds, cultures and terminological-linguistic subtypes. In many such cases the intent of the message can easily be inverted as relevant details are either dropped or misinterpreted in transmission.
Systematic Factors impacting efficacy of communication:
The options available for compression are greater when two neighboring nodes already have a great deal in common, where shared datasets, terminology, mental models, and approaches to communication can be used to elide parts of the message, reducing bandwidth requirements, and allowing for communication that is both more reliable and more rapid. As a result of this, communication within an organization that has a strong, unified "culture" (common knowledge, terminology and practices) will be far more effective than communication within an organization that has a less cohesive "culture", purely because the options for message compression are greater, irrespective of any other measures that the organization might put in place to improve the available bandwidth. It is worth noting that, whilst this does improve the situation considerably, the problem itself is fundamental and always presents a significant challenge.
Organizational Optimization for Effective Communication:
Given that there is nothing inherently fixed about the topology of the communications network within which we are embedded, one simple response to this problem is to remove intermediary steps from source to destination nodes, and to allow the source node to connect directly with the destination node, permitting a direct and relatively high bandwidth exchange. This is even more effective if the exchange is bidirectional, permitting in-situ error correction. This argues for a collaborative approach to organization - with technical experts communicating with one another directly, with no intermediary communications or management specialists.
Another approach is to optimize for effective compression of the messages being transmitted through the network. As noted above, this relies on a common terminology, a common knowledge base, and a common set of practices and approaches to problems. In other words, most of the communication is moved out-of-band -- communicated though various channels prior to the point where it is actually needed. Again, this is well aligned with the collaborative approach to organization - where the technical experts within an organization continually educate one another on their own area of expertise so that when they need to communicate quickly, they can do so both effectively and reliably.
A Common Antipattern: Spin Doctors and Office Politicians:
Of course, the approaches outlined above are not the only solution to this fundamental problem. However, I argue that at least some of these approaches are anti-patterns, detrimental to the long term health and development of the organization.
Many individuals craft the messages they send extremely carefully to minimize the probability of corruption or misinterpretation en route, partly by reducing the information content of the message, and partly by crafting the emotional tone to remove ambiguity. This process is colloquially known as "spin", or "crafting the message".
In some situations this approach may even be appropriate, for example: where the network is large and cannot be reconfigured so that messages must be transmitted over more than one "hop"; where the environment for communication is particularly disadvantageous, with little common culture, terminology, or background technical knowledge; and finally, where long-term systematic improvements must be subordinated to short-term goals.
However, there are a couple of significant drawbacks to this approach. Firstly, by restricting knowledge transfer, opportunities to grow a base of common knowledge and understanding are squandered. Secondly, this approach is in direct conflict with cultural norms that emphasize honesty and transparency in interpersonal communication, the adherence to which builds a basis of trust and mutual understanding that further enhances communication.
Friday, 4 January 2013
Dialectic vs. Tribalism .... Fight!
One of the problems that we have is that there is a very human instinct to stereotype and denigrate any (real or imagined) opposition community. It is part of our tribalistic nature, and it infects every sphere of human thought, even when we are not aware of it. (Especially when we are not aware of it.)
The truth of the matter is that other people are not simple and they are not stupid. They may be preoccupied with other battles, but they are generally smarter and more complex than we are willing to give credit for.
Fortunately, we have centuries of philosophical learning to help us tackle this behavioral bias. Unfortunately we do not have the time to absorb all of this scholarship so we will just jump on the whole "Hegelian" [sic] thesis-antithesis-synthesis meme as a quick-and-dirty fix for our ignorance.
We have to discard our tribalistic myopia and become aware of the other battles that people are fighting - the preoccupations that they are focused on, and within which they frame their arguments and their notions of right and wrong. Any synthesis solution will incorporate elements of these other concerns:- "In such-and-such a situation, you need to worry about risk A, and take action B, in this other situation, you need to worry about risk C and take action D."
In other words, a sign of maturity in the debate over a discipline is the presence of increasingly fine-grained recipe books, within which increasingly tightly defined specialist sub-disciplines emerge with their own concerns and heuristics. The division of labour becomes ever more fine-grained as the economy around a discipline matures and grows.
The truth of the matter is that other people are not simple and they are not stupid. They may be preoccupied with other battles, but they are generally smarter and more complex than we are willing to give credit for.
Fortunately, we have centuries of philosophical learning to help us tackle this behavioral bias. Unfortunately we do not have the time to absorb all of this scholarship so we will just jump on the whole "Hegelian" [sic] thesis-antithesis-synthesis meme as a quick-and-dirty fix for our ignorance.
We have to discard our tribalistic myopia and become aware of the other battles that people are fighting - the preoccupations that they are focused on, and within which they frame their arguments and their notions of right and wrong. Any synthesis solution will incorporate elements of these other concerns:- "In such-and-such a situation, you need to worry about risk A, and take action B, in this other situation, you need to worry about risk C and take action D."
In other words, a sign of maturity in the debate over a discipline is the presence of increasingly fine-grained recipe books, within which increasingly tightly defined specialist sub-disciplines emerge with their own concerns and heuristics. The division of labour becomes ever more fine-grained as the economy around a discipline matures and grows.
A (few) words to the wise
Here are the conclusions (categorized, mildly edited & slightly extended) from Boehm's retrospective of the past 50 years of software development.
Some of these are contradictory. Such is the way of wisdom.
Some of these are contradictory. Such is the way of wisdom.
Skepticism and Critical Thinking:
- Be self-aware. Know your own strengths and weaknesses, and how to manage them.
- Don’t believe everything you read. Be wary of the downslope of the Gartner "hype-cycle" roller-coaster.
- Be skeptical about silver bullets, and one-size-fits-all solutions.
- Avoid falling in love with your slogans. YAGNI (you aren’t going to need it) is not always true.
Total Commitment to Quality:
- Avoid cowboy programming. The last-minute all-nighter frequently doesn’t work, and the patches get ugly fast.
- Eliminate errors early. Even better, prevent them in the future via root cause analysis.
- Be quick, but don’t hurry. Overambitious early milestones usually result in incomplete and incompatible specifications and lots of rework.
Flexibility and Craftsmanship:
- Avoid using a rigorous sequential process. The world is getting too tangeable and unpredictable for this, and it’s usually slower.
- Avoid Top-down development and reductionism. COTS, reuse, IKIWISI, rapid changes and emergent requirements make this increasingly unrealistic for most applications.
- Look before you leap. Premature commitments can be disastrous (Marry in haste; repent at leisure – when any leisure is available).
- Keep your reach within your grasp. Some systems of systems may just be too big and complex.
Clarity and Communication:
- Determine the system’s purpose. Without a clear shared vision, you’re likely to get chaos and disappointment. Goal-question-metric is another version of this.
- Make software useful to people. This is the other part of the definition of “engineering.”
- Consider and satisfice all of the stakeholders’ value propositions. If success-critical stakeholders are neglected or exploited, they will generally counterattack or refuse to participate, making everyone a loser.
- Have an exit strategy. Manage expectations, so that if things go wrong, there’s an acceptable fallback.
The Skill and The Discipline:
- Respect software’s differences. You can’t speed up its development indefinitely. Since it’s invisible, you need to find good ways to make it visible and meaningful to different stakeholders.
- What’s good for products is good for process, including architecture, reusability, composability, and adaptability.
- If change is rapid, adaptability trumps repeatability.
- Don’t neglect the sciences. This is the first part of the definition of “engineering”. It should not include just mathematics and computer science, but also behavioral sciences, economics, and management science. It should also include using the scientific method to learn through experience.
Performance:
- Time is money. People generally invest in software to get a positive return. The sooner the software is fielded, the sooner the returns come – if it has satisfactory quality.
- These are many roads to increased productivity, including staffing, training, tools, reuse, process improvement, prototyping, and others.
- Think outside the box. Repetitive engineering would never have created the Arpanet or Engelbart’s mouse-and-windows GUI. Have some fun prototyping; it’s generally low-risk and frequently high reward.
Wednesday, 2 January 2013
Maslow's Hierarchy of Software Requirements
Agile development practices mandate an ongoing dialog with the customer, in which requirements are raised and met in a sequential manner.
This naturally causes a prioritization, as some requirements only become important once others have been fulfilled. This is closely analogous to Maslow's hierarchy of needs (following the point raised by Boehm in his paper: A View of 20th and 21st Century Software Engineering.
For example, once basic functionality has been implemented, it's basic reliability must be assured, and trust must be built in it's operation. After that requirement has been met, the security of the software must be ensured. Only after these tasks have been done does one (generally) prioritize the offering of metrics to "add value" with insight into the business.
Insight
Security
Trust & Reliability
Essential (Base) Functionality
Tuesday, 1 January 2013
Personal Development Process - End of Day
For the past couple of months I have been working from my home office. It has been a great experience - particularly in comparison with a workday that involves 2-3 hours of commuting, and is something that I would strongly recommend to anybody who is lucky enough to get the opportunity.
On the other hand, working from home definitely presents it's own challenges. Communication with coworkers (now clients) is much, much harder, and requires deliberate effort to get right. Discipline and record-keeping is also much more important. It is something that I was never particularly good at before, (I am a get-lost-in-the-work kind of guy) but have been really making a conscious effort to improve since the transition.
One thing that I have been trying to establish (without too much success so far, admittedly), is to incorporate a review & planning session at the end of each evening. To encourage me to stick at it, I thought that I would try to increase the value of the session by incorporating some post-mortem techniques along the lines of those suggested a couple of months ago by a (very astute and experienced) colleague:
End of day review activities:
Identify one thing that went wrong, and one thing that went right that day.
For both, do a "Five Whys" root-cause analysis.
This should give 10 "causes" at various levels, from proximal to distal.
Pick two "causes" to address with tasks to be performed on the next day, one task to improve and consolidate something that went right, and one task to address something that went wrong.
Over the week, try to spread the tasks over the entire proximal-to-distal spectrum, so that work is balanced between immediate (proximal) concerns and long-term (distal) improvements.
In this way, a better balance is maintained between the urgent and the important.
On the other hand, working from home definitely presents it's own challenges. Communication with coworkers (now clients) is much, much harder, and requires deliberate effort to get right. Discipline and record-keeping is also much more important. It is something that I was never particularly good at before, (I am a get-lost-in-the-work kind of guy) but have been really making a conscious effort to improve since the transition.
One thing that I have been trying to establish (without too much success so far, admittedly), is to incorporate a review & planning session at the end of each evening. To encourage me to stick at it, I thought that I would try to increase the value of the session by incorporating some post-mortem techniques along the lines of those suggested a couple of months ago by a (very astute and experienced) colleague:
End of day review activities:
Identify one thing that went wrong, and one thing that went right that day.
For both, do a "Five Whys" root-cause analysis.
This should give 10 "causes" at various levels, from proximal to distal.
Pick two "causes" to address with tasks to be performed on the next day, one task to improve and consolidate something that went right, and one task to address something that went wrong.
Over the week, try to spread the tasks over the entire proximal-to-distal spectrum, so that work is balanced between immediate (proximal) concerns and long-term (distal) improvements.
In this way, a better balance is maintained between the urgent and the important.
Antifragility
I have recently started reading the book: "Antifragile", by Nassim Taleb, author of "The Black Swan", and although I am only 70 or so pages into the book, I do not hesitate to thoroughly recommend it, although you may find (as I have) the personal, brash and argumentative style somewhat jarring.
As somebody who was weaned on Gleick, and who's bookshelf is packed with pop-sci books with "Complexity" or "Chaos" somewhere in the title, Taleb's central thesis is falling on fertile, and very well prepared ground. It is nice to have a book that brings some new ammunition, and new nuance to old arguments.
For example, in the predictive vs reactive control spectrum, it is clear that Taleb would argue vociferously for the merits of reactive control. His position seems to be based on two observations: The first is a behavioral bias: our propensity to smooth over and absorb past surprises, to rationalise and to confabulate, to fool ourselves into believing that past events were predictable when they were anything but, and to continue reinforcing the (erroneous) belief that we can predict and control the future. The second is the notion that unexpected events are more common that we expect, and that using past behavior to model future events will tend to underestimate the frequency and impact of those events. (A topic I have covered before).
"It is far easier to figure out if something is fragile than to predict the occurrence of an event that may harm it" ... "Fragility is quite measurable, risk, not so at all, particularly risk associated with rare events".
As somebody who was weaned on Gleick, and who's bookshelf is packed with pop-sci books with "Complexity" or "Chaos" somewhere in the title, Taleb's central thesis is falling on fertile, and very well prepared ground. It is nice to have a book that brings some new ammunition, and new nuance to old arguments.
For example, in the predictive vs reactive control spectrum, it is clear that Taleb would argue vociferously for the merits of reactive control. His position seems to be based on two observations: The first is a behavioral bias: our propensity to smooth over and absorb past surprises, to rationalise and to confabulate, to fool ourselves into believing that past events were predictable when they were anything but, and to continue reinforcing the (erroneous) belief that we can predict and control the future. The second is the notion that unexpected events are more common that we expect, and that using past behavior to model future events will tend to underestimate the frequency and impact of those events. (A topic I have covered before).
"It is far easier to figure out if something is fragile than to predict the occurrence of an event that may harm it" ... "Fragility is quite measurable, risk, not so at all, particularly risk associated with rare events".
Predictive vs Reactive management
Nothing in this article is really new, but I needed a document to which I could point people whenever I make use of Predictive-vs-Reactive terminology.
The Agile-vs-Waterfall debate is old, and, arguably, has been won. (Depending on who you ask);
However, I like to frame this dichotomy in other terms, which, I believe, offer both a superior perspective and (Hegel-like) the opportunity for synthesis.
Predictive (Waterfall) vs Reactive (Agile)
Traditional management techniques put the emphasis on predictive management, so that power may be consolidated in the hands of the decision maker. Planning and specification activities are important, leading naturally to a waterfall-style, gated development process. This is not so much a development methodology as a manifestation of the exercise of dominant political power within an organization.
The variance-minimizing aspects of predictive control trace their roots back to Deming's teachings in factory management, and are expressed through the TPS, six-sigma, lean sigma and so on -- Although in some situations, it can be argued that this philosophy is abused (More on this later).
Agile management techniques on the other hand put the emphasis on reactive management and feedback loops; the devolvement of power and responsibility to the individual developer, and the consequent restructuring of information flows to enable the organization to react to new information as it is discovered, and new events as they happen.
With predictive management everything is focussed around a small number of decision points and authority figures. A manager with authority for the project will either stop the project, or give the "green light" for work to continue at one of a handful of project gates. Predictive management requires extensive plans, forecasts and specifications to inform high-impact, long-lasting decisions. Predictive control is sensitive to changing environments, unexpected events, poor planning capacity, behavioral biases, and incorrect assumptions. Although appropriate in some situations, it is fragile and error-prone, and where it does occur, the guiding rationale is normally the consolidation of political or financial power.
With reactive management, the number and frequency of decision points is increased, so that work is planned out over short time scales only (anywhere from a few hours to a few weeks). Each decision is low-impact and carries only for a short time. Sensitivity to changing environments and unexpected events is reduced, as is sensitivity to poor planning, behavioral biases and incorrect assumptions (thanks to empirical feedback). Additionally, since decisions are greater in number and lower in impact, it becomes advantageous to devolve decision making authority to avoid the creation of decision-making information-flow bottlenecks.
One of the primary advantages of the reactive control approach is the opportunity that it offers for the incorporation of timely and relevant empirical information in the decision making process; the ability to seek feedback, to make mistakes and to recover (and learn) from them. Indeed, a properly functioning reactive control system does not seek to avoid mistakes, but rather to make them quickly, and learn from them, ("Move quickly and break things") although we often call the mistake-making process "experimentation" to disguise it's nature from those who, for political reasons, demand preternatural levels of perfection and clairvoyance from those around them.
The key property to look for, of course, is the flow of lesson-bearing information through the decision-making cycle. Error-feedback requires errors. An Interesting comparison is to be had between this and the CFAR (Constant False Alarm Rate) approach to adaptive signal processing - if you do not get any errors or make any mistakes, you are not trying hard enough!
Returning, for a moment, to Deming, variance-minimization, lean and six-sigma. Deming's argument is essentially the same as mine: To manage effectively, you need to incorporate empirical feedback, and empower individuals to act together in common cause. However, manufacturing is a highly controlled environment, where variance can be modelled as Gaussian, (six sigma) and unexpected "Black Swan" "Unknown Unknowns" can be omitted from the process control model. Software development (and other business activities) on the other hand, operate in a very different environment, where the Gaussian is a misleading and dangerous noise model. We do still need empirical and quantitative feedback, but we are no longer measuring variance in a simple, low-dimensional space, so what we can do with it is quite different. However, the concept of the feedback loop remains valid, and the organizational psychology is the same: Empirical feedback frees people from organizational politics, "gamifies" the work experience, and empowers individuals to work together for a common cause.
The Agile-vs-Waterfall debate is old, and, arguably, has been won. (Depending on who you ask);
However, I like to frame this dichotomy in other terms, which, I believe, offer both a superior perspective and (Hegel-like) the opportunity for synthesis.
Predictive (Waterfall) vs Reactive (Agile)
Traditional management techniques put the emphasis on predictive management, so that power may be consolidated in the hands of the decision maker. Planning and specification activities are important, leading naturally to a waterfall-style, gated development process. This is not so much a development methodology as a manifestation of the exercise of dominant political power within an organization.
The variance-minimizing aspects of predictive control trace their roots back to Deming's teachings in factory management, and are expressed through the TPS, six-sigma, lean sigma and so on -- Although in some situations, it can be argued that this philosophy is abused (More on this later).
Agile management techniques on the other hand put the emphasis on reactive management and feedback loops; the devolvement of power and responsibility to the individual developer, and the consequent restructuring of information flows to enable the organization to react to new information as it is discovered, and new events as they happen.
With predictive management everything is focussed around a small number of decision points and authority figures. A manager with authority for the project will either stop the project, or give the "green light" for work to continue at one of a handful of project gates. Predictive management requires extensive plans, forecasts and specifications to inform high-impact, long-lasting decisions. Predictive control is sensitive to changing environments, unexpected events, poor planning capacity, behavioral biases, and incorrect assumptions. Although appropriate in some situations, it is fragile and error-prone, and where it does occur, the guiding rationale is normally the consolidation of political or financial power.
With reactive management, the number and frequency of decision points is increased, so that work is planned out over short time scales only (anywhere from a few hours to a few weeks). Each decision is low-impact and carries only for a short time. Sensitivity to changing environments and unexpected events is reduced, as is sensitivity to poor planning, behavioral biases and incorrect assumptions (thanks to empirical feedback). Additionally, since decisions are greater in number and lower in impact, it becomes advantageous to devolve decision making authority to avoid the creation of decision-making information-flow bottlenecks.
One of the primary advantages of the reactive control approach is the opportunity that it offers for the incorporation of timely and relevant empirical information in the decision making process; the ability to seek feedback, to make mistakes and to recover (and learn) from them. Indeed, a properly functioning reactive control system does not seek to avoid mistakes, but rather to make them quickly, and learn from them, ("Move quickly and break things") although we often call the mistake-making process "experimentation" to disguise it's nature from those who, for political reasons, demand preternatural levels of perfection and clairvoyance from those around them.
The key property to look for, of course, is the flow of lesson-bearing information through the decision-making cycle. Error-feedback requires errors. An Interesting comparison is to be had between this and the CFAR (Constant False Alarm Rate) approach to adaptive signal processing - if you do not get any errors or make any mistakes, you are not trying hard enough!
Returning, for a moment, to Deming, variance-minimization, lean and six-sigma. Deming's argument is essentially the same as mine: To manage effectively, you need to incorporate empirical feedback, and empower individuals to act together in common cause. However, manufacturing is a highly controlled environment, where variance can be modelled as Gaussian, (six sigma) and unexpected "Black Swan" "Unknown Unknowns" can be omitted from the process control model. Software development (and other business activities) on the other hand, operate in a very different environment, where the Gaussian is a misleading and dangerous noise model. We do still need empirical and quantitative feedback, but we are no longer measuring variance in a simple, low-dimensional space, so what we can do with it is quite different. However, the concept of the feedback loop remains valid, and the organizational psychology is the same: Empirical feedback frees people from organizational politics, "gamifies" the work experience, and empowers individuals to work together for a common cause.
Friday, 28 December 2012
The Political Economics of the Singularity
"The singularity", in some sense at least, is already happening, and has been for the past couple of years.
Look, if you will, at the disconnect between the technology sector (booming; "talent" (labor) in exceedingly short supply, salaries rocketing) and the rest of the economy (tanking, many people out of work, surplus of labor, salaries plummeting).
Technology brings many new and unfamiliar nonlinearities into the economy. Access to the mass market does not (in all circumstances) require mass employment - For example, I am one individual, working from my home office, and I can easily make improvements to and deploy a product used (indirectly) by millions of people with a few keystrokes -- No expensive bureaucracy, no factory, no paperwork, and no infrastructure beyond a handful of laptops, an internet connection, and a few dozen rented Amazon EC2 machines.
The funny thing is this: We all expected the singularity to swing the balance of power firmly towards the side of capital, away from labor. After all, surely capital would simply buy robots instead of employing labor? However, it is not quite panning out as we expected. Developing a new technology product is now ridiculously cheap -- capital costs have all but disappeared. Technology startups now look to investors not so much for capital, but for advice, access to customers, and reputation. Just as the need for labor has (unevenly) diminished, so the need for capital has also (unevenly) diminished.
It is becoming clear that (in some circumstances at least) the old balance of power between labor and capital has been swept aside, with both, in some sense, having been made irrelevant.
What replaces it? The answer to that question is not easy to discern. One thing is clear though: this new world is far more complex, richly textured, baroque and interesting, and the old political battle-lines will need to be redrawn with greater subtlety and nuance than they ever were before.
Wednesday, 26 December 2012
Stupid is better than Smart (A call for humility)
Software development is a funny thing. It is full of nonlinearities and counterintuitive results.
Here is one of them: It is better to think of oneself as stupid (and be right) than it is to think of oneself as smart (and be wrong).
This sounds nonsensical, doesn't it? Surely it is better to be smart than it is to be stupid. Particularly since we spend so much of our time trying to demonstrate to other people just how smart we really are?
Well, if we were to start thinking of ourselves as smart people, relative to the rest of the population, then it is all too easy to start thinking of ourselves as being smart relative to the problems that we are trying to solve.
This is a problem, because *everybody* is pretty stupid in the grand scheme of things, and hubris is dangerous, particularly in the presence of complexity.
Complexity makes systems difficult to understand and manage, and difficult to fix when they go wrong. It is also difficult to gauge complexity, and humans have a consistent tendency to underestimate the complexity of unexplored functionality.
Let us consider a (mis) quote that illustrates the point:
"Debugging a system is harder than designing it in the first place, so if you are as clever as you possibly can be when you are designing the system, you are (by definition) too stupid to debug it."
The well known Dunning-Kruger effect applies here too: Because we tend to think that we are smarter than we really are, we tend to design systems that are too complex for us to debug and maintain.
In cases such as this, it is helpful to take a broader view. We may call somebody "Smart", but this term is defined relative to other humans, not relative to the problems that we need to solve.
We are frequently faced with problems that would be considered difficult by the very best of us. It is not a sign of weakness to acknowledge that; to treat the problems that we are trying to solve with the deference and respect that they deserve.
Here is one of them: It is better to think of oneself as stupid (and be right) than it is to think of oneself as smart (and be wrong).
This sounds nonsensical, doesn't it? Surely it is better to be smart than it is to be stupid. Particularly since we spend so much of our time trying to demonstrate to other people just how smart we really are?
Well, if we were to start thinking of ourselves as smart people, relative to the rest of the population, then it is all too easy to start thinking of ourselves as being smart relative to the problems that we are trying to solve.
This is a problem, because *everybody* is pretty stupid in the grand scheme of things, and hubris is dangerous, particularly in the presence of complexity.
Complexity makes systems difficult to understand and manage, and difficult to fix when they go wrong. It is also difficult to gauge complexity, and humans have a consistent tendency to underestimate the complexity of unexplored functionality.
Let us consider a (mis) quote that illustrates the point:
"Debugging a system is harder than designing it in the first place, so if you are as clever as you possibly can be when you are designing the system, you are (by definition) too stupid to debug it."
The well known Dunning-Kruger effect applies here too: Because we tend to think that we are smarter than we really are, we tend to design systems that are too complex for us to debug and maintain.
In cases such as this, it is helpful to take a broader view. We may call somebody "Smart", but this term is defined relative to other humans, not relative to the problems that we need to solve.
We are frequently faced with problems that would be considered difficult by the very best of us. It is not a sign of weakness to acknowledge that; to treat the problems that we are trying to solve with the deference and respect that they deserve.
BAM! The Combinatorial Explosion and Planning
Solving problems requires understanding.
Understanding is built, in part, by measuring features.
One or two features might be sufficient to describe something simple, but describing something complex often takes many more.
With one or two features, a handful of measurements may suffice to capture the behavior of whatever-it-is that we are trying to understand. As the number of features that we need to measure grows, the number of measurements that we need to properly and comprehensively capture that behavior grows faster than fast: It grows stupendously, ridiculously, unreasonably, explosively fast. It grows so fast that it is akin to ... BAM! ... hitting a brick wall.
This is the Combinatorial Explosion:- Know it, and Fear it, because it is Important.
In other words, it stops being possible in any reasonable, practicable sense of the word, to understand any system or problem, the description of which involves more than a small handful of features.
This has some terribly important consequences. Consequences that, as our world becomes more complex, and contains more and more complex, interconnected systems, we particularly need to understand and internalize, for we fail to do so at our peril.
The past becomes less useful as a guide for predicting the future; Our intuition becomes less effective; Unintended consequences and unusual, exceptional events become more prevalent; our ability to predict what will happen next is weakened to the point where it disappears, and the utility of planning (in the way that we commonly understand it) diminishes dramatically.
At no point in history did plans ever survive first contact with the enemy, but as (consciously or unconsciously) we become more liberal with complexity, these traits and characteristics will become more and more prevalent.
We need to adapt, and learn to deal with them.
PS:
This line of thinking is particularly interesting when applied to organizations:
http://daltoncaldwell.com/thoughts-on-organizational-complexity
Which, of course, then go on to produce more complex products:
http://en.wikipedia.org/wiki/Conway's_law
Understanding is built, in part, by measuring features.
One or two features might be sufficient to describe something simple, but describing something complex often takes many more.
With one or two features, a handful of measurements may suffice to capture the behavior of whatever-it-is that we are trying to understand. As the number of features that we need to measure grows, the number of measurements that we need to properly and comprehensively capture that behavior grows faster than fast: It grows stupendously, ridiculously, unreasonably, explosively fast. It grows so fast that it is akin to ... BAM! ... hitting a brick wall.
This is the Combinatorial Explosion:- Know it, and Fear it, because it is Important.
In other words, it stops being possible in any reasonable, practicable sense of the word, to understand any system or problem, the description of which involves more than a small handful of features.
This has some terribly important consequences. Consequences that, as our world becomes more complex, and contains more and more complex, interconnected systems, we particularly need to understand and internalize, for we fail to do so at our peril.
The past becomes less useful as a guide for predicting the future; Our intuition becomes less effective; Unintended consequences and unusual, exceptional events become more prevalent; our ability to predict what will happen next is weakened to the point where it disappears, and the utility of planning (in the way that we commonly understand it) diminishes dramatically.
At no point in history did plans ever survive first contact with the enemy, but as (consciously or unconsciously) we become more liberal with complexity, these traits and characteristics will become more and more prevalent.
We need to adapt, and learn to deal with them.
PS:
This line of thinking is particularly interesting when applied to organizations:
http://daltoncaldwell.com/thoughts-on-organizational-complexity
Which, of course, then go on to produce more complex products:
http://en.wikipedia.org/wiki/Conway's_law
Subscribe to:
Posts (Atom)