A high level discussion about a significant problem in the software development industry and how it might be resolved.
This document is work in progress.
The terms: cause, effect, problem, consequence, symptom, solution are often used loosely. For this document their interrelationships are defined as: a cause is the producer of effects, effects may combine to produce a problem, a problem has consequences, an effect may exhibit symptoms. For clarity I have divided the discussion into a section concerning the problem followed by a section on the solution.
Many software development difficulties are associated with a common high-level problem which is difficult to objectify because it is most visible at a level above that of existing industry roles and so the existing roles don’t often consider the problem. To fully understand the problem we should identify the problem cause, its effects and their symptoms.
In existing roles the problem is typically experienced obtusely; for example as frustration or confusion with the complexity of choosing an architectural approach that firstly will not soon be anachronistic and secondly suffers least from mutual constraints imposed by the choices. Simple analogies often help, so I have constructed one using a physical parallel.
You were asked to build a new kind of car from parts made by others. You favoured the newest types of part because they had new features and many other car makers seem to prefer them. Unfortunately it turned out that the new features were not that important to your customer, the parts were difficult to fit together and soon were not the preferred type. Now the parts are no longer made and it is hard to find someone to work with them. Also the parts makers are producing more types of part because of the past success in selling new types of part. However, they are unsure about which are good features for their next new parts, so the new parts tend to be derivative of each other. The current situation is more complex than last time you were asked to make a new type of car and it seems it will be even more complex in the future.
Although a great deal of effort is being expended in the software development industry, a large part of it is wasted and as a consequence progress in delivering improved systems is slower than it should be.
The software development industry as a whole has no single top-level organisation (STLO) providing authoritative strategy guidance to its senior roles. There are many bodies that do provide guidance but no STLO, leaving difficult judgements with a myriad of individuals and organisations. This causal lack of authoritative guidance has two main effects that dominate the form of the problem. Firstly, choices made tend to be skewed to whatever is the latest perceived state-of-the-art approach over the appropriate approach. I will refer to this as the Newest Is Best (NIB) effect. Secondly, choices made tend to constrain other choices that should be independent. I will refer to this as the Mutual Constraint (MC) effect.
To justify my statements of problem, consequence, cause and effects I will delineate the remainder of the problem discussion in terms of architectural trends and system development constraints, because they are familiar concepts directly impacted by the NIB and MC effects respectively.
In this section I will demonstrate some symptoms of the NIB effect with two examples of new architectural approaches supplanting older ones. In each case there are concomitant and often axiomatic benefits to the new approaches, but I will focus on their disadvantages as a device to highlight the NIB effect. After the examples I will analyse how they relate to the NIB effect.
Relative to a traditional aka ‘smart’ client, a web application will provide a user interface that under many circumstances executes more slowly, uses more client host resources, is less functionally rich, requires more server host resources and more network bandwidth, while also scaling badly on adding more clients to the system. However, many system architects would be more likely to choose the more recent and perceived state-of-the-art web application approach. I suspect the term ‘smart client’ was coined in part to overcome the mistaken perception of this as an anachronistic solution.
I want to exchange some data between clients and a service. Current style dictates that I probably use the latest XML derived protocol to mark-up a text based data stream and transfer it using a HTTP POST through a web server to an ASP.net or JSP based web service. For most purposes this approach is more resource intensive in all respects and functionality equivalent to using either a fixed width field or delimited field binary data stream between sockets. However, any architect suggesting either of those less recent approaches would be considered by some of their peers to be passé and consequently inclined to use a newer approach.
Clearly I am suggesting by my weighting toward the negative symptoms in the examples of recent architectural trends, that the more appropriate approaches are sometimes being overlooked in favour of more recent approaches. In each example above there are benefits to using the newer approaches, but they are relatively small non-universal advantages and are accompanied by disadvantages. With the creation of the new approaches the range of scenarios for using the older approaches is certainly narrower, but is not as narrow as it is often perceived to be. The new approaches are too widely applied and the older ones to quickly dismissed when they still have merit. The trend for a rapid procession of new architectural approaches is at least in part because of a lack of authoritative high level direction from a STLO which could simplify making appropriate choices over the assumption that what is newest is best.
Rapid growth in available technology and techniques from the technologists has led to an explosion in the number of solutions created by solutions developers, thus making the choice harder for the systems architect. The systems architect, solution developer and technologist each have their own micro-agenda and a limited higher level knowledge and concern of industry wide issues. This lack of concern for industry wide issues is derivative of not having to justify a development effort in terms of authoritative guidance from a STLO. This lack of concern is seeded by a failure to enforce a requirement on development work that its approach be justified in the context of authoritative guidance from a STLO.
The exciting but undirected sprawl of development effort is typified by the rapid turnover of solutions used in projects and diverts effort from delivering better functionality into an endless stream of new ways to deliver it. In a constant poorly informed search for the new best way forward the industry has become obsessed with how it does things over what it does. That is not to say that no progress is being made or that creative speculation is a bad thing. It is a matter of scale. The amount and rate of change in choice is becoming a problem to manage in itself and is symptomatic of the lack of the authoritative guidance from a STLO being both available and required for justification of development architectures.
To demonstrate the MC effect firstly I will show the existence of the development constraints, then how things would be better without them, before discussing some progress in this area and lastly analysing the MC effect.
In an ideal world, amongst many other things, a systems architect should be able to decide on a scheme for independently managing the important functionality and data issues delineated in the subsections below. However, this is impossible, because the systems architect has to choose and combine a range of solutions, technologies and techniques, each of which address only some of the functionality and data issues and does so in a constraining way. For example, if as an architect my customer mandates that I provide sophisticated zero deployment client behaviour with web based access to their services, then I have a very limited choice of client runtime environment and programming languages for developing client side behaviour. If I need to staff the project quickly and/or cheaply then I narrow the choice further to well established runtime environments, probably just Flash® and the JRE. In another example I choose to use an XML based configuration file. This time I either have to write an XML parser or look for a third party parser or use a runtime with one in its library. It us unlikely that writing an XML parser would be efficient use of time, but third party and library parsers will only be available for a limited set of programming languages, runtimes and platforms. So this time a seemingly simple choice has narrowed down the options quickly.
Choices generally force an immediate constraint on the structure of the whole development effort. Each choice imposes constraints upon another, forcing a rapidly diminishing set of choices. The end result is that we create systems that are extremely inflexible to change. Architects and their customers have come to accept as a fact of life that systems often need to be adapted with extra layers or redeveloped because the mutual constraints imposed by choices resulted in a very rigid solutions. However, these constraints don’t have to exist, they have arisen because of the lack of direction in the industry as we shall see.
Where functionality will execute, what programming languages the software will be written in, where the functionality will be deployed to, how execution of functionality will be initiated, how system load will statically and dynamically be distributed and balanced, the number of instances of the functionality running and deployed, how functionality is deployed, how functionality is loaded into the execution location, inter-thread and inter-process communication, the version of functionality to execute and the version of the runtime environment.
Storage location and distribution, replication, caching, backup, referential integrity, access contention and version control.
At this point I am going to have to assume I have adequately proven there are some constraints imposed by choices, but that you may be thinking: ‘alright there are constraints, but we can live with them’. Well yes we can, but that is not the point. We can all live without washing machines, but given the choice who would! A few examples of how it would be better without the constraints:
Perhaps now you are thinking: ‘alright there are constraints and it would be better if I didn’t have to live with them, but its never going to happen’. Well there is some evidence of progress. Java bytecode has long been able to run as an application or an applet so the same library code may run server or client side. The Mono CLR and Microsoft® .Net CLR both allow multiple programming languages to target the same runtime environment and a recent version of Silverlight contains a CLR that runs as a browser plug-in, so it can run some code that was previously written for server deployment as zero deployment on a client. There is also some interesting work in the LINQ Project toward the generalising of data access with different sources.
The isolated view of development work, where little or no attention is paid to integration with a diverse range of other work is instrumental to the MC effect. Hence many solutions are difficult to integrate and so constrain the choices available. The MC effect can only be reduced by higher level views being taken on development that span issues greater than the delivery of single systems. In the end systems are a composite of functionality and information and features of each should impose as few constraints as possible on each other. The decoupling of functionality and information features to reduce the MC effect is impinged upon by complexity management theories and revolves around the fundamental concepts of boundary and delineation which are too deep to discuss here. However, treatment of the impact of these and other concepts is vital to progress but beyond the scope of any one project and probably beyond the scope of any one company. A STLO is necessary to have sufficient oversight of these pervasive issues and steer the industry forward more rapidly.
System architects work in isolation and can only combine solutions and technologies available to them. They try to pick the useful and popular ones. Solutions developers work in isolation using the technology and techniques available to them. They hope their products will become useful and popular, while trying to pick the useful and popular technologies. Technologists work in isolation developing their products and hoping they will become useful and popular. Clearly there is a lot of investment in software development without it seems an authoritative understanding about the value of that work within the industry as a whole and how to avoid replication of previous effort and failures.
A single independent and unbiased top-level organisation should provide open access to pragmatic strategic level guidance targeted at senior roles. The organisation should collect current thinking, assess it and disseminate best guidance from that. It should try to obtain contributions from the best people it can but not exclude contributions from all levels. It should seek to engage with and gain endorsement from industry, government and other significant organisations to promote its visibility at the highest level. The guidance it produces will need to be accessible in different forms to different roles but should always to be pragmatic and un-opinionated. There a number of organisations that partly fit this profile, for example: the International Association Of Software Architects, the Institute For Enterprise Architecture Developments, the Worldwide Institute of Software Architects, the Enterprise Architecture Interest Group, the Association of Enterprise Architects, the Association of Open Group Enterprise Architects and the International Federation for Information Processing Software Architecture working group (IFIP WG2.10). This diversity is itself a problem. Ultimately a STLO has to exist for software to become a more professional and cost-effective tool for delivering benefit.
The absence of this organisation is manifest by the isolation of each project, so that there is large variation and frequent change in how projects are delivered that don’t exhibit substantial differences in what they deliver. The organisation would not prevent anyone from doing what they want, obviate competition, nor disseminate protected intellectual property, but it would enable a greater level of strategic awareness and so advance the industry. It would also help companies to better differentiate their products from their competition through increased awareness of the benefits of different approaches. Competition needs no sponsor to thrive and will continue to drive progress, but collaboration is less spontaneous and the domain of the visionary mind, it will direct the drive for progress.
Is there anybody out there making efforts on forming a STLO for software development? It would be nice to know. At the minimum it needs to be endorsed as such by the software development industry’s biggest players.
Gary E Byatt