The decommoditization of protocols

by Raph Levien
4 Nov 1998

DRAFT

One of the most interesting things about Microsoft's Halloween Memo is the concept of "de-commoditizing" protocols. This short essay attempts to explain what this means, and what its effects on free software are. I argue that decommoditized protocols are a very effective weapon against free software in the short term, but in the long term will help free software become more fulfilling to users.

I use the term "protocol" in a rather inclusive sense, encompassing APIs, file formats, etc., not just the narrow sense of networking protocols - in short, anything that software modules need to be able to work with each other.

Background: free software and proprietary software

Decommoditized protocols are a marker for some fundamental differences between free software and proprietary software, and the philosophy that goes into them.

In an ideal world, software would be created to fulfill user needs, and would be designed to be maximally usable. However, real software gets created for somewhat different aims, sometimes in line with these user needs, sometimes less so.

Proprietary software is created to make money for its authors. To a first approximation, the best way to make money is to create a highly usable product, so that users will be willing to pay for it. However, at least two other factors come into play:

For proprietary software to be profitable, it must create a proprietary advantage. Thus, simple software that just gets the job done has one serious disadvantage: a competitor can duplicate it. Given the extremely low marginal cost of software, this inevitably drives the price to near-zero. Thus, all truly profitable software has built-in barriers to competition.

And the most effective way of creating those barriers is to exploit network effects. Software doesn't exist in isolation - it's constantly interworking with other modules, loading and storing files in file formats, calling APIs, and communicating over the network. In most cases the usefulness of the software depends more on how well it interworks with other stuff than its own intrinsic merits. For example, even if a new word processor comes along that's better than Word, many people would be reluctant to switch because there are so many existing documents in Word format.

Free software, by contrast, is written for lots of different reasons, including a simple desire for the software on the part of the author, education, and being part of the free software community. However, getting the job done expediently is almost always an overriding concern. Thus, free software tends not to be much more complex than necessary, and making use of existing modules and protocols is often more appealing than reinventing things from scratch.

How to decommoditize a protocol

There are six things you can do:

Of these, only the last really makes the software more useful to users.

Examples of commodity protocols

The very best commodity protocols solve hard problems, but make it look simple. Two examples stand out: TCP/IP and HTTP/1.0. Neither of these protocols is perfect. However, their commodity nature greatly helped them take a foothold.

TCP/IP

TCP/IP is the foundation of the Internet. The protocol dates back to the early days of the ARPANet, and has existed in its present form since September 1981 (the date of RFC 791 and RFC 793). This protocol violates all of the first five principles of de-commoditization.

To say that TCP/IP has been enormously successful would be an understatement.

HTTP/1.0

Another example of a commodity protocol is version 1.0 of HTTP. Like TCP/IP, it solves a hard problem (people had been trying to implement global hypertexts for at least three decades before the Web hit), but is very simple. Indeed, a working HTTP/1.0 server is a weekend hack, and doing a simpleminded client is pretty easy too.

After the success of HTTP/1.0, however, the pressure to make the standard more complex became too great to resist. As a consequence, the HTTP/1.1 spec is about 167 pages long, and is still in the process of revision at the time of this writing. A lot of what's in HTTP/1.1 is good stuff (like pipelining), but a lot is also needless complexity.

Examples of de-commoditized protocols

The Microsoft Win32 API

Perhaps the most classic example of a decommoditized protocol is the set of API's, DLL's, and other stuff comprising the Microsoft Win32 environment. This protocol is extraordinarily complex, incompletely specified (indeed, there are numerous inconsistencies between Microsoft's own implementations), poorly documented, and subject to rapid change.

As a consequence, the Wine project (an attempt to implement the Win32 API within Linux) has found it very rough going. But they will get there.

RealAudio

Real Networks is a classic example of a company that was able to leverage a proprietary protocol into a successful business. Real has been upgrading the protocol continuously, improving quality and compression. They've used patents, undocumented protocols, rapid change, and added value to protect their product. It's also a classic example of the network effect - the more market share the clients have, the more motivation there is to provide content in RealAudio form, and vice versa.

How the IETF resists de-commoditization

IETF process actively resists de-commoditization in a number of ways. Most importantly, they require two or more interoperable implementations. This requirement puts a lot of pressure on the proposed standard to be both simple and completely specified.

The process resists de-commoditization in a number of other important ways, as well. It encourages the use of unencumbered technology when an unencumbered alternative exists. The entire process is conducted entirely in the open, with free availability of all documents. And perhaps most importantly, there is a strong tradition of standardizing technically excellent commodity protocols.

UnixTM in the '80s: a case study of how de-commoditization can kill you

Back in the '80s, many technically oriented people were hoping that Unix would catch on in the PC marketplace. It was widely recognized as a powerful and mature system, with many important features such as networking, multitasking, and protected memory (the last of which is still not completely implemented in MacOS and Win9x). Of course, it required a lot more resources to run than the "toy" PC's of the day were capable of, but it was also clear that PC's were getting more powerful by the month.

None of this happened in the '80s, though. I argue that de-commoditization was a major culprit. All of the Unix vendors wished for their brand of Unix to have a proprietary edge over the others. Thus, it was to their advantage to add "features" that didn't exist on other Unices. The goal was to lock developers into one vendor's brand. If they were successful, the the resulting software just wouldn't work well on other systems.

Unix vendors did ok during this time, but were never able to compete effectively against PC operating systems. It was not until the advent of Linux that Unix really started taking off in the PC world. I believe that the success of Linux is due in large part to its wholeheartedly embracing the essential commodity nature of Unix. Indeed, Linux has fairly few features that were not present in Unices of the '80s. The appeal of Linux is that it implements these features extraordinarily well, and with an eye towards compatibility.

Complexity

The design of software is a constant struggle against complexity. On the one hand, the world is complex, and many difficult problems inherently require complex solutions. On the other hand, it's quite easy to add gratuitious complexity. The key difference is how much of the problem the complex software solves (i.e. how much complexity is exported to the other side of the protocol).

One example of this sometimes subtle distinction is the comparison of GX vs. OpenType layout by Dave Opstad. Dave points out that for an application to support, say, Tibetan using OpenType, it still has to do a lot of the work. In the GX model, it's done for you in the operating system. Thus, even though OpenType and GX are roughly comparable in complexity, from the point of view of the application GX is "simpler". Thus, overall complexity needs to be weighed against how much of the problem is solved.

For a proprietary software organization that has just implemented a complex piece of software, it is tempting to assign a zero cost to the complexity. However, this would be quite wrong. Aside from the simple issue of higher maintenance costs, having more complex protocols makes the software far less agile, i.e. it is much more difficult to adapt it to changing market conditions.

Conclusion

In spite of the strong financial incentives and remarkable market successes of de-commoditized protocols, I believe that the future lies with commodity protocols. Users are becoming frustrated with the complexity, lack of consistency, poor documentation, and lack of choice that de-commoditized systems suffer from. Conversely, systems based entirely on commodity protocols have had their own share of remarkable successes.

The fact that commodity protocols are simpler and more completely specified than proprietary ones gives free software developers an edge. Given the disadvantages that free software developers work under (lack of funding being the most obvious), this edge is critically important if free software is to realize its current promise as a viable alternative to proprietary software as a tool for ordinary people to get their work done.

Thanks to the Gimp developers on #gimp for feedback on earlier drafts.

Free software

* www.levien.com