by Andrew Kuipers
I was reading the article In the Search of Code Quality by Jacek Sokulski today, and it had an interesting overview of the impact of our efforts to improve code quality by technical changes such as different languages or paradigms (eg: functional languages, concurrency via message passing, etc.) as well as development practice changes such as pair programming and code reviews. The results he looks at are surprising: for the most part these efforts have not resulted in improved code quality, although they have generally contributed to developer productivity. According to Sokulski, "The explanation of this phenomenon is based on the concept of target risk from psychology - people behave so that overall risk - called target risk - is on a constant level. When circumstances change people adapt their behavior so that the level of risk is constant." So instead of these technical and practice changes improving the quality of code developers write, it has instead allowed the developers to write code of a relatively similar quality with less effort.
This caused a theory (of sorts) to bubble up in my brain: software permits infinite complexity but software developers can only manage a limited amount of complexity. Any innovation in software development, whether it's through new languages, development practices, etc., does not reduce the infinite potential complexity of software but rather just makes aspects of our current development processes less complex, thereby freeing up the cognitive capacity of developers to explore more complex uses for software. We can see this through innovations such as relational databases or cloud computing resources: developers no longer need to fill their heads with the complexities of managing raw data on disk or running servers, and so they can instead use that freed up cognitive capacity to build more complex applications on top of those established components. Therefore, any innovation in software development will not make the building of software less complex, it just allows us to make our software systems more complex.
Presumably, you could make the development process less complex if you adopted new practices or technologies yet held fixed the complexity of the software systems you're building, however it seems that we rarely exercise this option, but would instead rather push our systems right up to the line of complexity that we're able to wrap our heads around. I suspect that this is the result of evolutionary pressure; once additional complexity is possible, someone will start taking advantage of it, and very quickly others will have to adapt the same approach or risk losing whatever advantage they have. If your competitors start writing sexy new single-page apps in React, it's not sufficient for you to keep writing the same old page-load interaction PHP sites that you've gotten really good at building. As soon as one individual starts doing something new and advantageous, everyone else has to catch up, because in the evolutionary game staying in place is falling behind.
However, embracing a new complexity in software development requires the capacity to do so, and that capacity is the cognitive capacity of the software developers involved. Without reducing in some way the current cognitive load imposed on developers, it is not possible to take on new complexities while maintaining the same pace of development; it can certainly be tried, but the general consequence of a higher load is a reduction in throughput and an increase in defects. By chasing new complexities without having the cognitive capacity to do so, we make ourselves slower and therefore less able to keep up with the evolving landscape of software development.
To wrap this up with some sort of conclusion, I'll offer this: we will never succeed in making software development less complex, because every innovation in the field just enables us to explore more of the infinite potential complexity of software. However, the fundamental limiting factor in our ability to embrace new forms of complexity is the cognitive capacity of the developers involved in the process of building software. Therefore, if we want to optimize our ability to work effectively in the current state of software complexity, as well as to evolve and embrace new technologies, we should focus our efforts on reducing extraneous cognitive load on developers before introducing new complexities.