The Clean Room - The Ultimate Structured Development Process?

'Computer scientists are defrocked Mathematicians'. That is a saying that was quite common in the early days. Software development at its core was viewed as a science (or a technology) not an art form. The goal of a software process was to have a series of 'correct' steps that began with requirements and ended with 'correct' code - just like a mathematician would begin with a description of a problem and arrive at a solution. 

The programmer was viewed as the weak link in the development process chain. The software development process was the scaffolding put up to minimize coding errors. While this may still be true in some settings, Agile development has moved the needle towards the view of the programmer as an artisan/practitioner.

'Formal' approaches to software process improvement abounded. Improvement in manufacturing quality during the late 80's early 90's was an area that was mined for gems that could be used to improve software quality.  A big goal of manufacturing quality improvement was to form enough of a statistical base for the overall assembly line that spot testing could be done rather than complete testing. This drastically reduced the time needed in the testing step of the production process. Also, if problems were found, the step in the assembly process causing the problem could be fixed, and the process could continue. These ideas were applied to the software development domain. The Clean Room development combined ideas from computer science and manufacturing process improvement and applied them to software development. It's goal was not to have a 'coding step', but use step-wide refinement to arrive at code from a decomposition of the requirements, thereby moving the 'correctness problem' from the coder to the process.

The Clean Room process was created by Dr. Harlan Mills from IBM. Dr. Mills' description of the process can be found in this 1987 article in IEEE Software: Cleanroom Software Engineering.  

The driving idea behind the Clean Room process was that software was to be produced by a series of correctness-preserving steps. So if there as an error produced in the software, the process step that produced the error was  changed so that error could not longer be produced, then the process was re-run. The software was never 'fixed'. The process was 'fixed'. In a strict version of the process, the coder never compiled the code. That was part of the testing group's effort. If the code did not compile, it was determined how code was generated that did not compile. The code would then be thrown out, and the process would be re-run to allow the 'coder' to generate new code using a new process step(s) that would not produce the uncompile-able code.

At the core of the original Clean Room was something called Box Structured Development. It is partially described in: Box-Structured Requirement Determination Methods. It would take one or more blog entries to completely go through box-structured analysis and design, but here's a section from 'Box-Structured Requirement Determination' to give a feel for what happens:

Requirements gathering procedure 

Step 1: Identify Inputs - Via information gathering tasks, a list of system inputs is generated. It should be recognized that these inputs will be at various levels of abstraction, from databases and files to simple data variables and physical signals (e.g., a clock pulse). All potential and available inputs should be listed. An analysis of the necessity and sufficiency of the inputs will be performed later. We define the list of inputs as ]= (11, 12 .... ,Ii).

Step 2: Identify Outputs - A list of required outputs from the system is generated. Close interaction with the customer and users is needed to develop a complete output list. The output list is defined as 0 = (01, 0 2, .... Form Black Box Transactions - The black box behaviors that relate the input history to the required outputs are described. A set of black box transactions is generated, (Pl, P2, ...,Pk). Each transaction is a function from the input history to a set of required outputs, i.e., pro(I*) O m where O" c O. All required outputs must be produced by one or more transactions.

End of requirements gathering procedure. 

Once requirements gathering is complete, a black box hierarchy is created by refinement of the initial black boxes into lower-level black boxes until 'code' is reached. Each black box is decomposed into a triple of (another) black-box : state-box : clear-box.  The clear-box defines the black-box at the next level of iteration.

Example Clear Boxes

To my knowledge, there were never many projects that attempted this process. The one usually touted was part of the IBM CICS project. What really killed this approach to development was the fact that no self-respecting coder wanted to submit themselves to this. It conjures up images of the human computers Richard Feynman assembled during the Manhattan Project. However, one could imagine tools being built that automatically converted high-level problem statements into box-structured statements and then into code.

Completely top-down development processes (where the solution is some sequence of iterations from the requirements) always has the problem that the structure of the requirements imposes itself on the solution - which is probably not optimal. Top-down solutions that only tag the fulfillment of a requirement(s) design/code artifacts allows the structure of the solution to differ from the structure of the requirements and follow, say, design patterns or structure imposed by the solution platform.

So how is this the ultimate structured development process? Well, it takes practically all development decisions away from the coder and distributes them between the requirements and design engineers. It also targets the process itself as the source of bugs. If you have followed my other posts, you know I make a distinction between projects that must be 'good' and projects that must be 'right'. This process assumes all projects must be 'right'. So arguably, this fulfills the dream of structured development.

The Process(es) of Music Creation

Creating software is an art. It is also an engineering discipline. Much has been written about the engineering side of software creation. Not so much has been written about the artistic side. I'd like to try to fill in some of that space by talking about music creation. Why talk about music creation? There's not much question that music is an art. It's an art that has been around for a long time. Focusing on software creation as an art is relatively recent. So talking about software as art would be largely speculative and opinionated. Hopefully by looking at the process(es) of music creation first we can apply what we find there to the realm of software creation. Also, I'm not thinking about aesthetics - either in art or software. I'm thinking about the creation processes.

The picture at the top of this blog attempts to diagram my thoughts on the processes of music creation. While I think I can pass myself off as a musician, this picture should remove any questions about me being an artist 😏. For the purposes of this blog, the process of music creation is what happens between the time the musician has a new idea for a composition and a performance of the composition occurs. I'm leaving out a lot of detail to focus on the higher-level flow between idea and performance.

When an idea strikes, many musicians go right to their instrument and start working out the idea. They may just 'learn' what they work out and perform it without ever writing anything down.  They might play it immediately to some friends to get their feedback. Maybe they take it to an open mic night and get more feedback. This is one end of the music creation process spectrum indicated by the lower arrow in the picture. This is the 'by ear' process of music creation. With this process, even if there is a group of musicians learning a new composition, they sit together and listen to what's happening, then add their part. Or someone may say 'play this' and plays a part for someone else. But nothing is written down. If the next performance is a little different, or parts shift, it doesn't matter. To some in this camp, not being able to read music notation is a badge of honor. 'I just play by ear' is a boast. In fact, a lot of the musicians that I work with don't know anyone that does music any other way. Expressing yourself with an instrument and your voice becomes second nature. Everything else just gets in the way of the muse. A lot of good music has been created this way.

Let's talk about the other end of the music creation process spectrum. What if you are creating music for an orchestra? With this music creation process, you will need to write something down. I'm calling this the 'notational' process. It is shown as the upper arrow flow in the picture.  To be successful with this process you will need to understand how what you write will be interpreted by other musicians who you may never meet. When your music is performed, you may not even be acknowledged directly. The conductor and the orchestra will be the focus of what the audience experiences. Whether the performance is successful may have nothing to do with the person that created the music.

What you write down during the notational process becomes 'the music'.  It's called the 'score'. If you're lucky, people will look at your score for years after you're dead and debate what you meant by a particular sequence of notes. Or how to play one instrument's melody against that of another instruments melody. Or how loud the 'forte' marking should be. Or if you ever changed your mind about how fast the piece should be played. Even if you conduct the piece yourself, that performance will be discounted in the future - since you're a composer not a conductor. 

I've studied with maestros that could look at the score directly and 'hear' what's there.  In the past, those that couldn't do that (or wanted other folks to hear a preview of the composition) might write a two-piano reduction of the  composition.  Thankfully today, to hear a full rendition of what you create with the notational process doesn't require finding an orchestra. There is software that will play what you write down.  It also aids those that can't play an instrument. They can learn music notation and get the computer to do the performance. As a side note, this what happens in a lot of sample-based music. The composer (producer) may not use the traditional music notation, but there is something - maybe a graphic notation that is manipulated to get the computer to generate a performance.  In any case, the score-playing software bends the process of creating orchestral music closer to the process of creating more immediate art. But there is still the requirement to write something down in some notation. Good music is being created this way. Films have 'scores'. Twenty-first century orchestral (and chamber) music is amazing. DJs are amazing!

Jazz sits in the middle of these two processes. Most jazz (big) band music has a score. Other jazz has a 'lead sheet' which includes notations for the melody, a shorthand for the harmonic material, and probably some hints for the drummer. The lead sheet is enough to sketch out how the song should go but doesn't tie you to specific notes like the score. To do a good performance, the performer probably knows the composition, or knows another composition by the same composer or at least knows the musical idioms hinted at by the lead sheet. It would be hard for someone without that background to take a lead sheet and do a satisfying performance.

The process for creating jazz could begin with the 'by ear' process and move towards the notational process. Or not. Sometimes it begins with the notational process and ends up back at the lead sheet. Arguably, a jazz composition may never really be complete. The creation process extends into the performance. In free jazz, the performance is the composition. 

Even in the creation of art we can find processes. And just like in the engineering space, sometimes 'doing something right' can get in the way of 'doing something good'. I know it took me about 2 years to recover from studying music theory. On the other hand, it ultimately made me a better musician.

In future blogs, I plan to formalize these process descriptions and marry the ideas from the music creation to those of software creation.

Introduction

Why a blog about the process of software development/creation? I've been a student of software development processes since the early 80's. There are still books and papers from the history of this subject stacked in various corners of the house. As an example, here's one that I picked up fairly recently in a used bookstore on my way to NYC.

This book was published in 1982. Apparently, there was a revolution in software engineering underway then. What kind of revolution was it? Let's have a look. Here's a few items from the Table of Contents.

• A Technique for Software Module Specification - D. L. Parnas
• Program Development by Stepwise Refinement - N. Wirth
• Design and Code Inspections to Reduce Errors - M. E. Fagan
• Notes on Avoiding "GO TO" Statements - D. E. Knuth & R. W. Floyd
• Mathematical Foundations for Structured Programming - H. D. Mills

These titles sound very formal on one hand, and somewhat basic on another hand. How does an article on avoiding 'goto' end up beside an article on the mathematical foundations of structured programming? Since this is an introduction to the blog, I won't talk about these articles here. I may hit on some of them later. I will quote from the N. Wirth article to give a taste of what was 'in the air'.

"In each step, one or several instructions of the given program are decomposed into more detailed instructions. This successive decomposition or refinement of specifications terminates when all instructions are expressed in terms of an underlying computer or programming language..."

It would be interesting to look at writing an ex post facto manifesto of the Structured Programming 'revolution' so that it could be compared with the Agile Manifesto. Many of us were caught flat-footed by the Agile Revolution. I know I was working to fix some of the problems with 'traditional' top-down development approaches. I very quickly realized that this work had been 'overcome by events'.

I have worked successfully on large and small projects in the Agile world. Agile has really advanced the state-of-the-art and given us many new tools that will be in software development processes for a very long time. Maybe the greatest achievement of Agile was stopping the 'software is 90% done for 50% of the time' problem. It certainly cut the analysis paralysis problem, too.

And yet, the movement to Agile seemed more like a religious conversion than a thought-through move to something better. The Agile Manifesto was to be accepted not rationalized. Past 'non-Agile' successes somehow disappeared. In religious terms, the moral agent in the process shifted from the designer to the coder (will explore this in a future post). Software - which was once thought of as being developed by defrocked mathematicians, is now the creation of the Coder-Artist.

So I'm writing this blog because I want to explore this territory. Certainly I want to document and understand the inflection from software-as-math to software-as-art; but also to consider what's next. I have some ideas.