Lecture
A doctor, a builder, and a woman programmer were arguing about whose profession was the oldest. The doctor observed: "The Bible says that God created Eve from Adam's rib. Such an operation could only have been performed by a surgeon, so I can rightly claim that mine is the oldest profession in the world". At this the builder broke in and said: "But even earlier, the Book of Genesis says that God created heaven and earth out of chaos. That was the first and undoubtedly the most remarkable feat of construction. So, my dear doctor, you are wrong. Mine is the oldest profession in the world". At these words the programmer leaned back in her chair and said with a smile: "And who do you suppose created the chaos?"
1.1. The Inherent Complexity of Software
Simple and complex software systems
A star on the verge of collapse; a child learning to read; blood cells attacking a virus - these are just a few of the astonishingly complex objects of the physical world. Computer programs can be complex too, yet their complexity is of an entirely different kind. Brooks writes: "Einstein argued that there must be simple explanations of natural processes, because God is not capricious or arbitrary. The programmer has no such comfort: the complexity he must master is inherent in the very nature of the system" .
We know that not all software systems are complex. There are plenty of programs that are conceived, developed, maintained, and used by one and the same person. Usually this is a beginning programmer or a professional working in isolation. We do not mean to say that all such systems are badly made, still less to question the qualifications of their creators. But such systems, as a rule, have a very limited field of application and a short life span. It is usually better to replace them with new ones than to try to reuse, rework, or extend them. Developing programs of this sort is tedious rather than complex, so studying that process is of no interest to us.
What interests us is the development of what we shall call industrial-strength software. Such software is used to solve a wide variety of problems, for example: feedback systems that control or are themselves driven by events of the physical world and for which time and memory resources are limited; the problem of maintaining the integrity of hundreds of thousands of records of information while allowing concurrent access to them with updates and queries; systems for the command and control of real processes (for example, air traffic or railway dispatching). Systems of this kind usually have a long life span, and a great many users come to depend on their proper functioning. In the world of industrial-strength software we also encounter development environments that simplify the creation of applications in specific domains, and programs that emulate certain aspects of human intelligence.
The essential feature of an industrial-strength program is its level of complexity: a single developer is practically incapable of grasping all the aspects of such a system. Bluntly put, the complexity of industrial-strength programs exceeds the capacity of the human intellect. Alas, the complexity we are speaking of appears to be inherent in all large software systems. By "inherent", we mean that this complexity is unavoidable here: it can be mastered, but it cannot be got rid of.
Of course, there are always geniuses among us who single-handedly can do the work of a team of ordinary developers and achieve, in their field, success comparable to that of Frank Lloyd Wright or Leonardo da Vinci. We need such people as architects, who invent new idioms, mechanisms, and basic ideas that are then used in developing other systems. However, as Peters remarks: "There are very few geniuses in the world, and there is no reason to suppose that their proportion among programmers is any higher than average" . Despite the fact that we are all a little bit brilliant, in industrial programming we cannot constantly rely on divine inspiration to come to our rescue. We must therefore consider more dependable ways of constructing complex systems. To better understand what it is we are setting out to manage, let us first answer the question: why is complexity inherent in all large software systems?
Why is software inherently complex?
As Brooks puts it, "the complexity of software is by no means an accidental property of it" . Complexity is caused by four principal factors:
The complexity of the real world. The problems we try to solve with software often unavoidably contain complex elements, and the corresponding programs are subject to a multitude of different, sometimes mutually exclusive, requirements. Consider the required characteristics of the electronic system of a multi-engine aircraft, of a cellular telephone switching system, and of a robot. It is hard enough to understand, even in general terms, how each of these systems works. Now add to that the additional requirements (often not stated explicitly), such as usability, performance, cost, survivability, and reliability! The complexity of the problem is what gives rise to that complexity of the software product which Brooks writes about.
This external complexity usually arises from a "mismatch" between the users of the system and its developers: users have difficulty explaining, in a form comprehensible to developers, what actually needs to be done. There are cases in which the user has only a vague idea of what he wants from the future software system. This mostly happens not through anyone's fault; it is simply that each of the groups specializes in its own field and lacks the other's knowledge. Users and developers hold different views of the essence of the problem, and they draw different conclusions about the possible ways of solving it. In fact, even if the user knows exactly what he needs, we have difficulty capturing all of his requirements unambiguously. They are usually spread over many pages of text, "diluted" with a few figures. Such documents are hard to understand, are open to various interpretations, and often contain elements pertaining to design rather than to the necessary development requirements.
Further complications arise from changes in the requirements for the software system while development is already under way. Requirements are mostly revised because the very act of carrying out a software project often changes the problem. Reviewing the first results - diagrams, prototypes - and using the system after it has been developed and installed lead users to a better understanding and a clearer formulation of what they really need. At the same time, this process raises the developers' competence in the problem domain and lets them ask more meaningful questions that clear up the obscure places in the system being designed.
A large software system is a major capital investment, and we cannot afford to throw away what has been done every time an external requirement changes. Nevertheless, even large systems tend to evolve in the course of their use: hence the problem arises of what is often incorrectly called software maintenance. To be more precise, let us introduce a few terms:
Unfortunately, experience shows that a substantial proportion of the money spent on developing software systems goes precisely into preservation.
The difficulty of managing the development process. The developers' main task is to create the illusion of simplicity, to shield users from the complexity of the subject or process being described. The size of a software system's source code is by no means among its chief virtues, so we try to make source code more compact by inventing ingenious and powerful methods and by using the development environments of already existing projects and programs. However, new requirements for each new system are unavoidable, and they lead to the necessity either of writing a great deal of code "from scratch" or of trying to use existing code in new ways. Only 20 years ago programs of a few thousand lines of assembly language were beyond our capabilities. Today software systems whose size is measured in tens of thousands or even millions of lines in high-level languages have become commonplace. No one person will ever be able to understand such a system completely. Even if we decompose it correctly into its constituent parts, we still end up with hundreds and sometimes thousands of separate modules. Such a volume of work therefore requires a team of developers, ideally as small as possible. But whatever its size, significant difficulties will always arise in organizing collective development. The more developers there are, the more complex the connections between them and the more complex the coordination, especially if the participants are geographically remote from one another, which is typical of very large projects. Thus, when a project is carried out collectively, management's main task is to maintain the unity and integrity of the design.
The task of the developers of a software system is to create the illusion of simplicity.
The flexibility of software. A house-building company does not usually own a forestry operation to supply it with timber for lumber; it would be quite unusual for an erection firm to build its own plant for making the steel beams for a future building. In the software industry, however, such a practice is commonplace. Programming offers ultimate flexibility, and a developer can supply himself with all the necessary elements at any level of abstraction. Such flexibility is extremely tempting. It leads the developer to create by his own efforts all the basic building blocks of the future construction, out of which the elements of higher levels of abstraction are then assembled. Unlike the construction industry, where there are uniform standards for many structural elements and for the quality of materials, the software industry has almost no such standards. Software development therefore remains a very labor-intensive business.
The problem of describing the behavior of large discrete systems. When we throw a ball upward, we can reliably predict its trajectory, because we know that under normal conditions the known laws of physics apply here. We would be very surprised if, having thrown the ball a little harder, we saw it stop unexpectedly in mid-flight and abruptly change direction [Even simple continuous systems can exhibit complex behavior because of the presence of chaos. Chaos introduces a randomness that rules out precise prediction of the system's future state. For example, knowing the initial positions of two drops of water in a stream, we cannot predict exactly how far apart they will be after some time has passed. Chaos shows up in systems as diverse as atmospheric processes, chemical reactions, biological systems, and even computer networks. Fortunately, a hidden order seems to exist in all chaotic systems, in the form of so-called attractors]. In an insufficiently debugged program simulating the flight of a ball, such a situation could easily arise.
Inside a large application program there may be hundreds or even thousands of variables and several threads of control. The complete set of these variables, their current values, and the current address and call stack for each process describes the state of the application at any given moment. Since our program executes on a digital computer, we have a system with discrete states. Analog systems, such as the motion of a thrown ball, are by contrast continuous. D. Parnas writes: "when we say that a system is described by a continuous function, we mean that it contains no hidden surprises. Small changes in the input parameters will always produce small changes in the output". Discrete systems, on the other hand, by their very nature have a finite number of possible states, although in large systems this number is, by the rules of combinatorics, very great. We try to design systems by dividing them into parts in such a way that one part has minimal effect on another. However, transitions between discrete states cannot be modeled by continuous functions. Every event external to the software system can move it into a new state, and, moreover, the transition from one state to another is not always deterministic. Under unfavorable conditions an external event can disrupt the current state of the system because its creators were unable to anticipate every possible case. Imagine a passenger aircraft in which the flight control system and the electrical power system are combined. It would be most unpleasant if the passenger in seat 38J switching on his personal reading light sent the plane immediately into a steep dive. In continuous systems such behavior would be impossible, but in discrete systems any external event can affect any part of the system's internal state. This is obviously the chief reason why our systems must be tested; but the fact is that, apart from the most trivial cases, exhaustive testing of such programs is impossible. And until we have either the mathematical tools or the intellectual capacity to model completely the behavior of large discrete systems, we must content ourselves with a reasonable level of confidence in their correctness.
The consequences of unbridled complexity
"The more complex the system, the easier it is to bring it down entirely" . A builder would hardly agree to widen the foundation of an already completed 100-story building. It is not merely expensive: doing such things is asking for trouble. Yet, surprisingly, the users of software systems set such tasks before developers without a second thought. It is, they insist, merely a technical matter for the programmers.
Our inability to build complex software systems shows up in projects that overrun their established schedules and budgets and, on top of that, fail to meet the original requirements. We often call this the software crisis, but, to be honest, an ailment that drags on this long becomes the norm. Unfortunately, this crisis leads to the squandering of human resources - the most precious commodity of all - and to a substantial limitation on our ability to create new products. There are simply not enough good programmers at present to supply all users with the programs they need. Moreover, a substantial proportion of the development staff in any organization must often be occupied with maintaining and preserving obsolete programs. Given the direct and indirect contribution of the software industry to the economies of most of the leading nations, we cannot allow the present situation to remain unchanged.
How can we change the state of affairs? Since the problem arises from the complexity of the structure of software products, we suggest first considering the ways complex structures are dealt with in other fields. Indeed, there are many examples of successfully functioning complex systems. Some of them are man-made, for example: the Space Shuttle, the Channel Tunnel, large firms such as Microsoft or General Electric. In nature there are even more complex systems, for example the human circulatory system or a plant.
Comments