Software engineering

Software engineering is a broad engineering topic whose goal is, ultimately, how to assist the production of cost-effective, reliable software. It is described by the Institute of Electrical and Electronics Engineers (IEEE) as "the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software". As an academic topic of interest, the field of software engineering was arguably born with the publication in 1975 of the book The Mythical Man Month by Frederick P. Brooks, Jr. The book, now in its second edition, is still commonly used as background reading in college courses and was the first treatment to challenge the usefulness of adding programmers to an already late programming project.

The use of computers in government, commerce, education and recreation has been expanding rapidly, leading to demand for software which is easy to use, yet capable of performing complex tasks. Many commercial products now contain embedded computers, which often require software of extremely high reliability. All these trends have spurred the urgent need for effective software engineering.

Emergence of software engineering

The term software engineering was first coined in 1968 at the NATO Software Engineering Conference in Garmisch-Partenkirchen, Germany by its chairman Friedrich L. Bauer to address what had become known as the 'software crisis'.

The software industry was considered to be in crisis for several reasons: project schedules and development costs were often vastly underestimated, the quality of software was less than desirable and the demand for good software was outpacing the number of good software developers. Furthermore, the cost of maintaining software (fixing bugs, changing it to meet new requirements) was now outstripping the initial cost of developing it. In many, if not most cases, the cost of the software exceeded the cost of the hardware it ran on.

Since the early 1960s, computer hardware has been increasing in power rapidly. New and powerful machines, made possible by the invention of integrated circuits, enabled scientists and business professionals to make use of computers in ways that were previously unrealisable and unfeasible.

The resulting software that grew out of this hardware revolution was orders of magnitude larger and more complex than any seen before it and the management of that complexity was extremely difficult. Software products were being delivered to customers years late, over-budget and unreliable.

Up until that point, no formal process for building software existed. Software developers often built software on an ad-hoc basis and any processes that did exist were informal and unique to each company or development team. Software practitioners and experts realised that a more formal approach was needed, akin to other engineering disciplines.

Much research, experiment and trial-and-error was conducted by computer scientists and software professionals in an effort to formulate useful theories, methods and techniques to address the problem of the inherent complexity of software. The term 'software engineering' was widely adopted to reflect the increased focus on discovering good software development practices and the fact that many early practices and principles were imported directly from other engineering disciplines such as construction engineering.

Unique problems facing software engineers

One of the major problems with software is its inherent complexity.

In 1987, respected software expert, Frederick P. Brooks, Jr., wrote about the problems faced by software engineers in his seminal paper "No Silver Bullet: Essence and Accidents of Software Engineering". In this paper, he describes the problem as being that the nature of software is, in essence, highly and arbitrarily complex, conformable to any number of differing human institutions, systems and interfaces, easily changeable (and, thus, the thing most likely to be changed when change becomes necessary), invisible and unvisualizable (in its entirety). He argued that because these problems are (in the Aristotlean sense) an essential aspect of software, rather than an accidental aspect, there can be no all-encompassing solution to the problems of software engineering (although there may be solutions to the accidental problems).

Ian Sommerville points out that, unlike other engineering mediums, software is not constrained by physical materials, governed by physical laws or manufacturing processes; as such there are no physical limits to the potential of software. This results in software becoming extremely complex and difficult to manage or understand; this often leads to the software becoming unreliable.

Despite the seemingly pessimistic outlook, all is not as bleak as it seems: despite its young age, much progress has been made in the field of software engineering and many of the problems (if only the accidental ones described by Brook) have been addressed by the invention and adoption of formal software development processes and other techniques.

Modern-day software engineering

As a result of research and experience in the field, a number of developments have led to improvements in the field of software engineering. The next sections describe just a few of these.

The software development process

Formal software development processes have proven to have marked benefits to software practitioners who adopt them. An in-depth description of each processes is beyond the scope of this article (see the 'List of topics in software engineering' section below for more information); however, four activities have been identified as being common, indeed, essential to any successful software process (as with other engineering disciplines, adopting a process of 'divide and conquer' makes the process more manageable):

1. Software specification. The discovery, and definition, of the software's functionality and constraints on that functionality.
2. Software development. The production of the software to meet the specification.
3. Software validation. The process of ensuring the software correctly meets the specification and customer satisfaction.
4. Software evolution. The ongoing maintenance of the software and implementation of any requested changes or additions.

Each activity in the software process produces a deliverable of some kind, whether it be documentation describing the software, or functioning aspects of the software itself or, most likely, a combination of both documentation and software (in software engineering, both the documentation describing the software, the source code and the working computer software itself are all considered part of the software product).

How, and in what order, these activities are carried out differs from process to process but each tends to follow one or more of the following approaches (or paradigms) identified by Sommerville :

1. The waterfall approach. Each of the above activities are treated as separate time-delineated stages of the development process, with each stage feeding into, and informing the one following. In reality, there is often a back-and-forth between stages but, generally, one stage flows into the next in a uniform direction. Forms of this approach are commonly used in other engineering disciplines.
2. Evolutionary development approach. Each activity is carried out in an interleaved manner. A prototype is developed rapidly based on an abstract (and incomplete) specification. The prototype software is validated against the specification and the customer and the specification is then refined based on the feedback from the validation. The software product is then refined (or recreated from scratch in the early stages) based on the updated specification. The specification, development and validated continues in an iterative manner until the software product meets the final specification and is accepted by the customer as complete.
3. Formal transformation approach. Uses a mathematical system of specification and transformation of the specification into the final software product. This approach ensures, through precise mathematical methods, that the product fits the specification. This approach is not as common as others.
4. System assembly from reusable software components. This approach reuses and integrates existing software components to build the software. It is often combined, in part, with the other approaches.

The waterfall method has generally fallen out of favour with software engineers for all but the most simple projects for being far too simple and inflexible to deal with the complex and rapidly changing nature of software. The evolutionary development approach is the most commonly used and supported approach due to its flexibility and ability to handle rapid changes in specification and early prototyping. System assembly from reusable components is fast becoming a very popular approach due to the increasing availability of off-the-shelf, third-party software components and the speed with which it enables rapid application development.

In reality, a software engineering team will adopt and adapt one, many, or all of these process paradigms during the lifecycle of a software project time depending on the nature of project and the particular problems currently being faced. No software process is meant to be blindly followed as set of inflexible rules (with the exception of some safety-critical systems processes) and the software engineer is expected to use his or her intelligence and experience when following a particular software process and to adapt it as necessary.

Other developments

In addition to software processes, many other programming language paradigms, technologies and techniques have improved the field of software engineering over the course of its history. Some examples are:

• Time-sharing
• High-level languages
• Unified programming environments
• Object-oriented programming
• Rapid application development (or software prototyping),
• Buy versus Build
• Software modelling languages (e.g. Unified Modelling Language)
• Software testing

Software engineering organisations

The following organisations are involved in research into the improvement of, or specification of, current software engineering practices:

• Carnegie Mellon Software Engineering Institute (SEI) [1]. Established the Capability Maturity Model scale of measuring the maturity of any software engineering organization.
• British Computer Society (BCS) [2]. The BCS has a world-wide membership of over 60,000 members in over 100 countries. It is the qualifying body for Chartered IT Professionals (CITP).