Oddly Influenced

Welcome to Oddly Influenced, a podcast for people who want to apply ideas from *outside* software *to* software. Episode 47: Oops! The Winston W. Royce Story

Here’s a bit of an experimental episode. It describes a historically-important example of the relevance of metaphors to software. I claim metaphor helped lead our field astray. I also assign some blame to the classic node-and-arrow diagram, a tool that I think is frequently misused.

Now, that may just be sour grapes from someone with almost no visual imagination, but maybe my deficit allows me to, uh, see problems with diagrams that the visually besotted don’t.

The topic presents a problem. I’m going to talk about diagrams that you can’t see. They’re from Winston W. Royce’s 1970 paper “Managing the Development of Large Software Systems.” There’s a link in the show notes, if you want to skim them first or view them as you listen.

Alternately, I’ve put a version of this presentation up on a wiki of mine, where I can juxtapose the images with the commentary. This is part of my starting to explore another growing obsession of mine: the contrast between linky hypertext and linear, narrative story-telling as idea-making and idea-presenting tools. I’m biased toward narrative, so I’m going to spend more time on non-narrative. I’d be interested in anything you think about the two versions.

I’ll end with six recommendations for how to use – or not misuse – metaphors and diagrams.

≤ music ≥

Royce’s paper has been widely credited with inventing what later came to be called the “waterfall model of software development.” As Laurent Bossavit points out in his 2014 book, /Leprechauns of Software Engineering/, that’s bitterly ironic, because Royce’s paper says that while the as-yet-unnamed waterfall model is “fundamentally sound,” it is also “simplistic and invites failure.” Because of that, his paper “presents five additional features that must be added to this basic approach to eliminate most of the development risks.” His actual model was considerably more complicated than waterfall, and advocated for both iteration and a kind of rapid prototyping. And all that got ignored.

There’s an internet meme where a tech company announces their new product, “the Torment Nexus,” inspired by the classic science fiction novel “Don’t Create the Torment Nexus.” Royce is that poor forgotten classic author.

So let’s walk through Royce’s paper to diagnose what went wrong.

It starts with figure 1, “Implementation steps to deliver a small computer program for internal operations.” It has a box labeled “Analysis.” An attractive curved arrow leads to a box labeled “Coding”, which lies diagonally down and to the right. That’s a natural direction for a native English speaker because analysis precedes coding. Since we write left-to-right and top-to-bottom, we’re inclined to think of time as flowing in those directions. See your nearest calendar for details.

“Analysis” is an uncommon task name these days, I think. Royce worked on software for spacecraft. So part of the work of “analysis” is producing algorithms for “orbit mechanics, spacecraft attitude determination, mathematical optimization of payload activity, and so forth.” It was based on such analysis that you sized, for example, on-board memory. Remember, memory was fantastically expensive in 1970, so you didn’t want to build more than you needed. You also had to budget how much time key operations could take so that the software’s model of the spacecraft and its movement could keep up with reality. Also, during analysis you worried about something called “data flux.” I don’t know what that is.

Royce says that figure 1 only suffices for small, in-house projects. When you build serious software, you need more phases to do the kinds of work that reduces risk. He starts with seven phases, in this order:

System requirements come first. Since Royce was working in combined hardware/software systems, this phase explained what the whole spacecraft should do. The next phase, Software Requirements, specializes the system requirements to just the software part. Then comes analysis, as described above.

Analysis doesn’t lead directly into coding, though. There’s a Program Design phase between them. This phase starts by dividing the work. There will be N parts to the software. Those have to be named and their interfaces with each other defined. Then each part has to be designed. Nowadays, we’d do that “design” by writing code in Java or C or Elixir or whatever, and let the compiler translate our files full of classes or functions or whatever into executable code. Back in those days, though, actual code was a thin veneer over raw machine instructions. So people would write a “detailed design” that would look – extremely roughly – like modern code. And then some person (perhaps a different one) gets to play compiler and fill in the details. That happens in the “Coding” phase.

Then you do testing. Royce leans heavily on visual inspection and manual analysis because, for some things “the computer is too expensive.” For example, it seems he wants people to hand-execute “every logic path in the computer program at least once with some kind of numerical check.” Testing on actual hardware – trying out the program – is a shorter process, which he describes as “checkout” – implying that it’s on the cursory side.

Then the system exits development and goes into operations. There’s no mention of anyone ever touching the code again. Nowadays, software isn’t usually finished: immediately after releasing code, you expect to turn around and modify the same code for the next release, which might come the next day. Back then, you’d generally move on to a new project or switch from making the product to running it throughout its life. That might involve debugging and patching (as has been made famous by the Voyager probes, which have been kept running by hook or by crook far beyond their expected lives), but it’s not really new development.

This separation between development and maintenance is partly a hangover from government contracting practices – traditionally, pure hardware (trebuchets, howitzers, and the like) were designed, then manufactured, by one contractor. After that, keeping the weapon systems working was someone else’s job. The Army, for example, employed tank mechanics.

It took a *long* time for the “softness” of software to have an effect on process. Even when yearly releases of commercial software were the norm, people still clung to the separation between development and maintenance. For example, the people fixing bugs in Version 2 might be completely separate, organizationally, from the people designing and building Version 3. Which seems crazy now.

Royce illustrated that seven-step process in his worldview-defining Figure 2. It shows seven named boxes aligned diagonally from the top left to the bottom right, connected with those attractive curvy arrows.So the first arrow goes down and right from System Requirements to Software Requirements, which in turn emits an arrow going down and right to Analysis. And so on. It’s easy to see why the process came to be called “waterfall.” It’s a nice metaphor to see the curvy arrows as like water flowing over a series of ledges labelled “requirements”, “analysis,” “design,” “coding,” and so on.

That was a horrible mistake. Water flows easily and naturally downhill. Indeed, a lot of jobs involving water are about *preventing* it from flowing downhill, as a pipe in our upstairs bathroom failed to do last week. But hard-to-stop flow is not what happens in software: every new phase is *work*. A better way to draw his diagram would have been for work to go *up* and to the right.

Whoever first attached the word “waterfall” to figure 2 (not Royce) could have still used that metaphor. But instead of the project floating down the waterfall, it would be struggling up the waterfall, against the flow of the water, which maps *way* better onto the reality of software development.

I strongly suspect that the reason Royce drew his diagram along a diagonal was because fitting it horizontally on a page would have forced the labeled boxes to be too small, and I can even believe that the choice of down-and-to-the-right was arbitrary. But it led to a metaphor with connotations of ease and naturalness that are absolutely misleading and inappropriate.

I speculate the diagram also led to the misapplication of another metaphor.

In their /Metaphors We Live By/, Lakoff and Johnson make much of how, generally speaking, we humans think of UP as GOOD, probably for reasons like growth of not just people but also, say, trees from down near the ground upward. Growth is good, so up is good.

But when people get sick, they lie *down* to recover. That’s a bad state to be in until you get better and “rise from your sickbed.” In the worst cases, sick people never get up. They’re dead, and dead is bad.

The idea of UP being good and DOWN being bad can be seen not just in sayings like “he finally grew up” or “let your ideas soar like an eagle,” but also in our visual system. In the immediate aftermath of the 9/11 attacks on New York’s World Trade Center, Donald Trump remarked that he now had the tallest building in New York City. (Unsurprisingly, that wasn’t actually true.) Height seems a natural thing to boast about, but have you ever heard someone boast that they have the *widest* building in New York City? I doubt it – because upward is the direction of goodness, whereas spreading horizontally outward doesn’t map onto the goodness/badness spectrum.

This bias also shows up in charts and graphs. Consider a stock that sells for 10 dollars a share last week and 11 dollars now. What that literally means is that you’ll get more money today than if you’d sold last week. But, because “more” is “good” and “good” is “up”, we shorthand that to saying that the stock went up. So when you plot stock prices, the direction of “more” is up. A satisfied investor is one who watches his stock’s chart and watches it climb steadily upward (in the direction of health and growth).

That’s unlike the standard waterfall chart, where progress moves inexorably downward, in the direction of badness. That resonates with all kinds of metaphors: the Judeo-Christian notion of the Fall of Man, Plato’s Metaphor of the Cave, where we’re all stuck down here in the near-dark, whereas the world of Ideal Forms is above us (and is, therefore, the better world).

My point is that Royce’s figure plays into all sorts of metaphorical assumptions that made it easy for me to say, around 1982, that “The code isn’t real. Only the specification is real.” That’s a totally Platonist thing to say: the *idea* of the thing matters more than the actual thing. It took me well over a decade to admit – even to myself – that, actually, I like code. It’s not merely the byproduct of the *real* intellectual work.

After all, the government wasn’t paying Royce’s company (TRW) because it wanted system requirements specifications: it wanted *spaceships*. *That’s* what should be at the top of the diagram. That, like a high stock price, is presumably the sign of and the reward for good work.

≤ short music ≥

As Royce continues with the paper, he complicates the model and his graph.
He’s still on the very first page when he writes, “as each step progresses and the design is further detailed, there is an iteration with the preceding and succeeding steps.” He considered this normal. If you discover a mistake in the requirements during analysis, it’s not such a big deal to go back and fix it. He describes such changes as “scoped down to manageable limits.” To illustrate this, he added back arrows to figure 2 to produce figure 3. Each arrow goes backward – that is, up and left – to the preceding phase. They’re the same pleasant curvy shape as the forward arrows, just rotated 180 degrees, emphasizing a symmetry between moving forward and moving back.

In contrast, figure 4 shows what Royce is trying to avoid: big backwards movement. In it, a problem found in testing requires a fix two-levels up in Program Design, which reveals the need for a change two further levels up, in the Software Requirements. Because of that, “the development process has returned to the origin and one can expect up to a 100 percent overrun in schedule and/or costs.”

The graphic design here is nice. Instead of being pleasantly curvy, the multi-level back arrows are harsh and angular, emphasizing badness. Still, it would have worked better if the flow had been pictured up and to the right. As it is, the project is presented as naturally progressing smoothly downhill, with backtracking being the effortful and unnatural part. After all, how often does water in a waterfall turn around and go back uphill? Whereas, in fact, having to retreat and iterate – to go back up the process waterfall – is the most natural thing in the world.

I’ll even get a little fanciful and say the project metaphor should be of salmon leaping step by step up the waterfall. I’ve never seen that happen in real life, but I have watched fish making their way up a fish ladder near the Gerald R. Ford Presidential Museum in Grand Rapids, Michigan, USA. There’s a picture of part of that ladder on this episode’s web page.

It’s interesting. There’s a sequence of something like six ledges, with a deepish pool of water on each ledge. You can look down on the fish resting in a pool, gathering their strength. When one is ready, it attempts the leap up to the next ledge. Typically, it lands on the edge and wriggles wildly to move forward into the deeper water. More often than not, it fails and gets swept back into the previous stage.

*That’s* a better, more realistic metaphor for waterfall development, and it supports Royce’s central claim: that falling back one level is natural and normal and “scoped to manageable limits.” It’s having the poor fish be washed all the way back to the bottom of the ladder that would be the big tragedy.

When you see a fish being washed back one level, you don’t think it’s careless, or stupid, or undertrained, or not behaving professionally – like so many assume when software people make errors in one phase that are detected in the next. You think “life’s tough all over.” You can even wax evolutionarily and say that there must be some non-obvious cost that caused natural selection not to create fish that succeed on every jump.

Wish I’d thought of this back in the ‘80s when the waterfall metaphor was all the rage. Probably somebody did, but it sure didn’t catch on.

≤ short music ≥

Leaving aside the orientation of the diagram, Royce made a grievous error going between figure 3 and figure 4. Figure 4 adds those harsh-looking back arrows that go several phases back, but it *omits* the pleasant, natural, single-level back arrows. And Royce leaves them out of all the subsequent figures, including the final overall summary picture.

Out of sight, out of mind.

On page 3 of his paper, iteration vanished from the reader’s awareness, and it apparently stayed gone, judging by the next two decades.

I have a modestly detailed critique of the remaining figures, but I’ve stuffed that in the wiki, and I’ll just summarize here. But first, I want to quote something from a famous 1956 paper by George A Miller: “The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information.”

“A man just beginning to learn [Morse] code hears each dit and dah as a separate chunk. Soon he is able to organize these sounds into letters and then he can deal with the letters as chunks. Then the letters organize themselves as words, which are still larger chunks, and he begins to hear whole phrases. I do not mean that each step is a discrete process, or that plateaus must appear in his learning curve, for surely the levels of organization are achieved at different rates and overlap each other during the learning process. I am simply pointing to the obvious fact that the dits and dahs are organized by learning into patterns and that as these larger chunks emerge the amount of message that the operator can remember increases correspondingly.”

Our brain loves to chunk things in order to increase its capacity. I want to claim that Royce’s diagonally downward connected boxes have the sort of symmetry and order that make it easy for the brain to make it into a perceptual chunk. That chunk – I’ll call it the waterfall – appears at the centers of figures 5 through 9, each of which adds an elaboration around the waterfall.

Perhaps you’ve seen those old-fashioned flip books that perform an animation? If not, Wikipedia to the rescue:

“A flip book is a booklet with a series of images that very gradually change from one page to the next, so that when the pages are viewed in quick succession, the images appear to animate by simulating motion or some other change. […]

“Rather than ‘reading’ left to right, a viewer simply stares at the same location of the images in the flip book as the pages turn. The booklet must be flipped through with enough speed for the illusion to work, so the standard way to ‘read’ a flip book is to hold the booklet with one hand and flip through its pages with the thumb of the other hand. The German word for flip book—Daumenkino, which translates to ‘thumb cinema’—reflects this process, the photographic progression integral to film.”

A flip book version of figures 5 through 9 would show the waterfall at the center of the image, with other shaped boxes and arrows chaotically appearing and disappearing around that unmoving chunk. The final image, figure 10, juxtaposes the figure 4 waterfall - the one with the angular back links – a picture representing the problem – to Royce’s solution, which has 14 rectangles, seven stylized document binders, three types of circles, and a total of 26 or 27 arrows (I kept losing count). The back links are still omitted, as are some of the other elements of figures 5 through 9.

I know what he’s doing. He’s showing the thoroughness of his solution, how it covers so many contingencies and risks. However, the unity of the waterfall is broken up – some of its formerly connecting arrows make detours – and the overwhelming impression (to me) is of barely-controlled complexity. The waterfall – the *incorrect* waterfall – on the other side of the diagram looks *way* more appealing, angular arrows demonstrating its big problem be damned.

≤ music ≥

H.L. Mencken famously said, “For every complex problem, there is a [solution] that is clear, simple, and wrong.” Royce’s paper would have been better without diagrams. By visually emphasizing the complexity of the necessary solution alongside the clear, simple wrong solution, he produced what is called, in US Law, “an attractive nuisance”:

“[The doctrine] states that a landowner may be held liable for injuries to children trespassing on the land if the injury is caused by an object on the land that is likely to attract children. The doctrine is designed to protect children who are unable to appreciate the risk posed by the object by imposing a liability on the landowner.”

Now, it’s vaguely insulting to compare adult professionals reading a paper to children seduced by an unfenced swimming pool, but it’s just a fact that the brain loves taking shortcuts that influence our judgments in ways not under our conscious control. So I have some advice for document writers or explainers.

1. Implicit metaphors like “UP is GOOD” have influence, so be alert to whether you’re triggering them unintentionally. Lakoff and Johnson’s /Metaphors We Live By/ will provide you with a list.

2. If you don’t assign a guiding metaphor, there’s a good chance your reader will: that’s how we got “waterfall.”

3. Learning happens through repetition and emphasis. If you keep talking about the torment nexus, don’t be surprised if that’s what the reader remembers. Royce kept showing the world the waterfall, so that’s what the world remembered. Royce *didn’t* keep showing a vital part of the solution (iteration), so that’s what the world forgot.

4. There are certain visual patterns humans tend to find appealing, and others we find offputting. In figures, make the solution appealing and the problem offputting, not the other way around. Royce’s solutions have messy solution icons surrounding the pleasant-looking tidy problem.

5. I have no academic support for this claim – I haven’t looked – but I think people handle complexity better in text than in diagrams *when it comes to grasping the totality of an argument or depiction of reality*. The structure of Royce’s argument is clear and linear. It goes like this: There’s a big risk with the simple waterfall process. To minimize risk, do five things. First, add a Preliminary Program Design phase before Analysis. Perform these five tasks in that phase. Next, really lean in on documentation: write it and use it meticulously throughout the project. Next, spend around a fourth of your time making a working prototype. Next, take the planning and execution of testing more seriously, especially the following four aspects, which he then lists. Finally, get customer signoff after the preliminary program design, the individual program designs, and for final acceptance.

All nice and linear. The supporting details are also presented in a linear fashion. They’re not wildly memorable, but neither is their representation in the figures. In either case, the reader who wants to put Royce’s paper to use is going to have to reread the details anyway.

The problem with diagrams is that they’re branchy. There are multiple paths, some of them just there for completeness. Unlike a narrative that’s read linearly, the user of a diagram has to pick the order in which to read it, which means the diagram has more opportunity to fail to convey what was intended. I speculate that’s why hypertext fiction has never really taken off.

If you must use a diagram, I have two suggestions:

1. A diagram should contain *only* the nodes and arrows that are necessary. Zoom in. Forget about progressively building up a complete solution: focus a diagram narrowly on the current topic. Of Royce’s five solution diagrams, only one (figure 8) benefits from showing the waterfall.

2. If you desperately want to culminate in a picture of the whole solution in all its glory, *build* the diagram rather than *show* it. I learned this from Michael Feathers. I was complaining about a complex UML diagram. In response, he built it up gradually on a whiteboard, explaining each new node and arrow as he added them. I understood at the end. That’s something like what Royce was trying to do, but the need to flip pages and rotate some of them because the diagram was too big to fit horizontally, plus the difficulty of locating the nodes under discussion (as opposed to Feathers’ just pointing at them or drawing them in) adds too much friction. Paper just doesn’t have the right affordances for complex diagrams.

≤ short music ≥

OK, that’s it. Next: more on metaphors or, if I get bogged down, perhaps we’ll visit the French Structuralists of the ’60s and ‘70s. I hope to continue posting audio, the transcript, and the new hyperlinked variant transcript as I accumulate episodes. It’ll slow me down, sure, but you probably couldn’t keep up with my frenetic release schedule anyway.

Thank you for listening.