European Common Lisp Meeting 2008

I had a great time at the European Common Lisp Meeting (ECLM) in Amsterdam, April 19-20, 2008. I met many of the important people in today’s Common Lisp world, an almost completely generation set of folks as compared to 15 years ago. The papers were excellent, and demonstrated that Common Lisp is still a vibrant and uniquely powerful language. (I’m writing this on the plane back home, on my OLPC laptop, as I learn to touch-type with my big hands on the child-friendly little keys.) Arthur Lemmens and Edi Weitz did a great job organizing and running the meeting. Everything went entirely smoothly and I felt that everyone enjoyed it very much.

InspireData
Jeremy Jones of Clozure Associates demonstrated InspireData, an educational application that lets you analyze data and draw conclusions. The user interface and interaction design is superb. It performs very well, even on weak, old PC’s, which matters since that’s what many schools actually have. It has gotten excellent reviews and sales, and is widely used in schools.

The user would have no reason to know that it was written in Lisp. No Lisp is exposed to the user. They used LispWorks, since it runs on all of their target platforms (including Windows 95, as well as more modern Windows and MacOS X), it has a good interactive development environment, it provides a portable library for accessing the platform’s native menus and other widgets (called CAPI), and had favorable licensing terms. The rest of the graphics was done by calling the OpenGL graphics library, using a library from LispWorks. They found all of these technologies to work very well.

They wrote it as a contract job, building it to specs provided to them in a 200-page document. It took 8 person-years of developers plus two person-years of Q/A, who were brought on board from the very beginning. It’s 270K lines of Lisp code plus 470 lines of C.

The primary advantage of using Lisp is that they could produce a prototype in only two months, and then do incremental additions and refinements. You might ask, why was this necessary if they had a 200-page requirements document?

• The specs were vague. It would have taken over a 1000-page spec to really be unambiguous. (In my opinion, that’s absolutely normal for software.)
• The specs kept changing. (That always happens. Always!)
• In particular, the designers would change the specs because of what they saw the program doing. In other words, specifying it in advance would have been impossible in any number of pages. Design and implementation must be interleaved.
• Even if the spec were known in advance. the best implementation techniques are not initially appanrent. Sometimes you have to get pretty far along in the implementation to realize that some atrchitectural decision did not work out well.

Lisp is very malleable. Experience over the years has shown that even large Lisp systems are particulary easy to re-factor and even re-architect. (I have seen this over and over again.) In fact, Jeremy feels that they didn’t re-architect enough! (One usually hears the opposite lament.) He emphasized that iterative development — build, test, refine — was the only way to go and the only way they could have succeeded.

LispWorks performance in this application is excellent. As I could see in the demo, it is extremely responsive. Jeremy says he has never perceived any delay from the garbage collector. InspireData is a shining proof that real applications can be done just fine in Common Lisp.

FEMLISP

Nicholas Neuss (IANM, U. Karlsruhe) presented FEMLISP, a system to do finite-element analysis (FEM). FEM is used for solving partial differential equations. It’s used to model things like convection, diffusion, viscous fluid flow, and so on.

I had thought this would just be a numeric library with some API, and wondered why doing it in Lisp would be helpful or make any difference. But it’s not like that at all. First, choosing a good way to run FEM is a hard problem. I only sort-of understand the issues, but I got a sense of it. There’s a big issue of how you “discreetize” and solve the discrete problem. You also must make decisions about how to set the mesh. Second, you want an interactive environment that lets you display results graphically, and make small changes to the input spec and try again. Ideally, you’d like an expert system to make these decisions, but what he’s done so far he described as “rudimentary”.

He created small domain-specific language to represent of how to run a particular FEM problem. This lives in a Slime buffer and can be edited and recompiled quickly and conveniently. It can display the history of the iterations so you can see what’s going on and refine your input. You can insert Lisp code into the input, for computing or debugging.

It runs on CMUCL, SBCL, LispWorks CL, and Allegro CL. He does graphics with OpenDX (Data Explorer), which was written by IBM and open-sourced. (He is considering switching to VTK.)

Why did he use Lisp?

• Dynamic typing worked out well
• Macros and a few reader macros let him make an embedded domain-specific language easily
• Dynamic testing and debugging (read-eval-print loop, etc.)
• High performance compiled code (as compared to Guile and other Scheme implementations that he tried)
• Common Lisp is stable; once you learn it, you know it
• There is no system/user dichotomy
• It only took 30K lines of code
• There were lots of useful libraries

Performance as compared to other available FEM packages is hard to determine for many reasons. For example, who chooses the benchmark? Can you find an informed third party to spec and judge the procedure? How do you know you’re not comparing apples and oranges? (These are all standard pitfalls of benchmarking.) Also, he has not spent too much time on performance improvements anyway.

Nevertheless, he ran some basic comparisons against their company’s in-house FEM system, called M++, not only to measure speed but to make sure he got the same answers (he did). M++ turned out to be faster on small problems, but FEMLISP was faster on larger problems.

One reason for this is extremely interesting. Apparently there is a certain well-known technique for speeding up FEM. He had implemented it, but they had not yet done so. This illustrates the principle that higher-productivity software development can lead to faster performance! When considering the effects on performance of using Lisp, take this into account.

In other tests he found that FEMLISP is about as fast as a leading commercial product (FEMLAB) for comparable accuracy, and much easier to use.

So far he has not tried to encourage other people to use it, mainly for political reasons (his boss wrote M++). He used FFI for certain existing libraries (e.g. LAPACK).

Large Internet Systems

Stefan Richter of freiheit.com talked about “Using Common Lisp for Large Internet Systems”. His company, freiheit.com technologies (it means “freedom”, in the sense not having to use the Microsoft platform any more!) has built many commercial web sites in Java. They have 60 developers, most using Java, but also a 6-person Common Lisp group. In an unusual twist, the manager of the group had to convince the reluctant programmers to use Common Lisp. Also, the clients had to be convinced that accepting a product in Common Lisp was OK. They have delivered one Common Lisp application so far, a social marketing tool.

By “large internet systems” he mainly means scalable web sites. Unfortunately, he has not actually built such a thing in Lisp yet. The talk suggests approaches to the problem, but he did not have actual experiences to report. He primarly prefers Lisp because he feels that Java is too verbose, and Ruby is basically like Lisp.

He explained a lot about how to build scalable and reliable servers (all of which I was very familiar with from my work at BEA and at ITA Software). Clusters, load balancers, stateless app tier, separate DBMS’s transactions and reporting, shared memcached distributed cache, keeping functionally separate data on separate DBMS’s, plus one idea that’s still new or in the short-term future: shared-nothing database clusters using “shards” with replicated data for reliability. All of this is completely right, in my opinion, and I don’t think any of it is controversial.

Java has many good tools for doing such an archtecture: Tomcat providing a framework for servlets/JSP’s, a memcached client, Hibernate for database access, and even Hadoop (a free MapReduce implementation).

How does Common Lisp compare? We have Hunchentoot (a sophisticated HTTP server), cl-memcached (a memcached client), cl-sql (to invoke relational DBMS’s), and two advanced tools for generating HTML: Weblocks (by Slava Akhmechet, I think), and UnCommon Web (by Marco Baringer).

He also suggests using cl-muproc (a library that provides Erlang semantics in Common Lisp, basically) which he feels could be a good basis for a Common Lisp MapReduce. I don’t know exactly what he has in mind here, but apparently he has implemented this.

He doesn’t like existing conventional technology for generating web pages. Servlets clumsily embed HTML in Java code; JSP’s clumsily embed Java code in HTML. Using Common Lisp has many of the advantages of other popular languages that are being used to write HTML generation, such as Ruby, Groovy, and Python. Lisp has major advantages: you don’t have to write out files in order to compile things; CLOS is very useful, including the MOP; we can avoid the need for XML because programs and data use the same format; and of course macros help in all kinds of ways. (And, I was thinking, Common Lisp implementations typically execute code much faster than Ruby and Python!)

He talked about using continuations to save state between HTTP interactions. (Many papers have been written on this topic.) You want to be able to write a program in a normal style, that can say “do this web interaction” in the middle of any procedure; this makes flow of control much easier to understand. A continuation saves stack and execution state across interactions. He talked about Weblocks and how it uses continuations, as well as many of its other virtues (it sounds great, from what he said; I have yet to learn about it).

He feels that what’s needed now is to put it all together, and then write a good book about how to use it. He points out that Ruby on Rails would never have taken off without the excellent book. (I agree completely!) He encourages us to write books, and help develop the framework libraries.

This all led to a lively discussion of continuations, particularly persistent continuations, and how to best implement them for Common Lisp. Weblocks uses the cl-cont library. Marco Baringer said that cl-cont’s continuation states are extremely large, leading to performance problems, although it would not be hard to improve this.

We also talked about just how reliable a system like this needs to be. It often turns out that in exchange for a very small about of unreliability, you can make big improvements in simplicity and performance. On a web site, it’s often quite acceptable to fail now and then, since the clients are human users who are much better at handling failure and retrying or finding alternatives.

PWGL

Kilian Sprotte described PWGL, a tool for music composition and analysis. It is based on an earlier system called Patchwork, by Mikael Laurson in his 1996 doctor’s dissertation, at IRCAM, the famous music lab in Paris. It is ten years old, and has always been in Common Lisp. Originally it was developed in MCL; now it’s based on LispWorks and runs on both Windows and Mac OS X. It’s now being developed at Sibelius Academy in Finland. It’s currently in beta-test, downloadable, and version 1.0 is expected later this year.

According to the description on the web site: PWGL is a free cross-platform visual language based on Common Lisp, CLOS and OpenGL, specialized in computer aided composition and sound synthesis. It integrates several programming paradigms (functional, object-oriented, constraint-based) with high-level visual representation of data and it can be used to solve a wide range of musical problems.

It’s a visual dataflow functional language; in some ways it’s like doing Lisp by drawing boxes and lines.

It uses OpenGL for graphics, the PortAudio library for recording, playing back, and basic sound synthesis, and the libsndfile library for reading and writing files containing sampled sound. (It was interesting to see how many Lisp systems are capable of using non-Lisp libraries easily. This is another important counter-argument to the objection that Lisp has too few libraries.)

Embeddable Common Lisp (ECL)

Juan Jose Garcia-Ripoll described Embeddable Common Lisp. ECL is not just for embedding: it’s a full Common Lisp implementation. It’s a descendant of Kyoto Common Lisp by Taiichi Yuasa and Masami Hagiya at the Research Institute. Juan is the maintainer.

It is designed for portability. Rather than generating machine code for various processors, it generates C, and then allows the target host’s C compiler to produce machine language. This approach lets it take advantage of the target compiler’s optimizations, and specific knowledge of the target architecture. (However, compilation is not very fast.) All platforms these days include a free C compiler (even Microsoft). It makes minimal architectural assumptions: a pointer can be cast to an int, and C functions can be called with many arguments, and a variable number of arguments.

It supports a wide range of operating systems: Linux, NetBSD, FreeBSD, OpenBSD, Windows, Solaris, and Mac OS X.

The core and the Lisp interpreter are written in C; the rest is in Lisp. It borrows the Boehm-Weiser conservative GC, and provides CLOS with the PCL implementation. It uses native threads. It also contains a byte-code compiler and interpreter (instead of direct interpretation of Lisp as s-expressions). The implementation of subtypep uses the efficient method described by Henry Baker, and works with CLOS types.

It can build standalone executables and dynamically-linked libraries, and this is why it’s called “embeddable”. But it can be used as a regular Common Lisp implementation too, so don’t be put off by the name!

For more details, see his paper.

House Developer

Kristoffer Kvello of Selvaag told us about House Developer, which is basically a CAD system for architects. It allows the architect to draw a very high-level drawing, and it takes care of filling in myriad specifics. It decides where to put windows and doors, and which way the doors should swing. It places electrical outlets and switches. It decides on wall types, wall offsets, wall junctions, heaters, fire exit paths, and so on.

There are many details, all of which must conform to regulations, company rules, and best practices. Doing all this by hand is costly, time-consuming, and error-prone. Automating it reduces errors, and lets the architect try lots of ideas and see their consequences promptly.

This is, in many ways, a classic rule-based expert system. They started writing it in 1994, using Knowledge-Based Engineering (KBE) technology of the time, which was primarily in Lisp. However, the rules are not like classic Artificial Intelligence rules; they are more like constraints. An example:

(define-attribute area (window)
    (* ?width ?height))

This defines a constraint that gets recomputed as necessary. These rules can use the full power of Lisp.

The core of the system is written in Allegro CL. There is a Java-based user interface, that sends S-expressions to the core. The core sends XML replies back to the user interface. It uses many available libraries: asdf, zip, cl-sql, cl-utilities, s-xml, aserve, Expresso Toolkit, and Screamer.

The Expresso Toolkit knows the STEP (Standard Exchange of Product data) and EXPRESS (an ISO standard modeling language), which are important standards in the architecture industry.

Screamer supports “non-deterministic programming; it does constraint satisfaction with mixed systems of numeric and symbolic constraints, based on a substrate that supports backtracking and undoable side effects.

The advantages of using Lisp for this system include:

• Interactive development, with fast recompilation, incremental changes, no need to constantly re-create the global state
• Break loops, with the ability to fix things and then restart
• Reader macros, so that we could customize the syntax
• Advice, so that we could customize behavior
• It’s easy to inspect the image to find out what to customize
• Extensibility in general
• handler-bind, for use on our test framework
• Many available relevant libraries, which worked fine

There have not been a lot of users so far, but they are planning to deliver it to a large customer soon.

High-Performance Architecture

Marc Battyani discussed a high-performance computer architecture, using Field-Programmable Gate Arrays (FPGA’s) that are programmed using a high-level special-purpose language, implemented in Lisp. He has a computer based on a Stratix II FPGA with memory and network. The FPGA has modules on it such as adders, multipliers, I/O pins, memory, and so on. Programming one consists of hooking up the modules up to perform a particular special-purpose function. A problem with FPGA technology is that programming them is so hard; the novel feature here is to use a Lisp-based language, called HPCC, that compiles a high-level description into the FPGA’s program.

They have implemented two applications so far. One prices exotic financial instruments using Monte Carlo simulation. Currently, this kind of thing is done with grids of 10K-10K boxes. The other does multicast networking at 1 million messages per second. They plan to get funding, hire more Common Lisp programmers, and do more applications.

Cells

Ken Tilton talked about his Cells library, a dataflow extension to CLOS. The basic idea is that the values slots are determined by a formula, like the cells in a spreadsheet. Cells tracks dependencies between cells and propagating values. He demonstrated widgets that grow and reshape graphics automatically.

Announcements

Randall Pitts is looking for Lisp programmers to work on a speech understanding project, that would help answer email, help call center agents, etc. They’re dealing with language, grammar, and syntax. You must work in Germany.

Nick Levine is looking for work. He has 20 years of Lisp experience and has been consulting for seven years.

Marty Simmons of LispWorks is looking for applications that use concurrency, to help test their new thread support.

One parting thought

One of the most widespread complaints about Common Lisp is the lack of available libraries. However, in several practical applications, we see here that there are many available libraries for Common Lisp that work well and can be built on.

European Common Lisp Meeting

The European Common Lisp Meeting of 2008 will take place on Sunday, April 20, 2008, with optional dinners on Saturday and Sunday evening. I’ve been to Amsterdam and totally loved it. I’d very much like to attend; I’ll have to see whether it’s possible.

OOPSLA 2007

I’ve just returned from the OOPSLA 2007 conference (www.oopsla.org) in Montreal. This was the single best conference I’ve ever been to. As you’ll see, the conference was much less about “object-oriented languages” (a topic which was used up years ago) than about software development and engineering in general.

Actually I attended three events: the Dynamic Languages Symposium, the Montreal Scheme and Lisp User’s Group Meeting, and OOPSLA itself. This meant that I went to evening activities every night, so it was non-stop and exhausting. On top of that, I developed some kind of stomach virus the day I left and was somewhat sick the whole time I was there! But the conference was so energizing that I mostly forgot that I was sick, aside from the fact that I didn’t eat very much. There wasn’t any time to get to real restaurants anyway so all I had was fast food, although some of it was quite good. There were plenty of other parallel activities that I could not get to, such as a day-long symposium about garbage collection, an APL conference, a Wiki symposium, an educator’s symposium, and on and on.

Two other ITA friends came: Francois-Rene (”Fare”) Rideau, and Alexy Radul who is just finishing an internship at ITA. Alexy had previously been an intern at Google with Arthur Gleckler, and the four of us ate and hung out together a lot. Arthur knows so many people who I know that it’s amazing we had never met before. He was one of Gerry Sussman’s grad students, and has worked with my old friends Charles Frankston and Carl Hoffman. We talked about everything, and I found out more about what life is like at Google, where Arthur works on mobile computing.

Many of the conference luminaries and keynote speakers were old friends of mine. Richard Gabriel of IBM Research, who was the conference chair, I had worked with on the Common Lisp spec, and later when his company, Lucid, was using my company’s Objectstore object-oriented database. I had worked with Guy L. Steele Jr., now of Sun Research, for years at the MIT Artificial Intelligence Lab in the late 70’s. I’ve known Gregor Kiczales, now professor at U. of British Columbia, for years. And so on. Dick Gabriel started working on this conference over two years ago and has done nothing for the last year but arrange the conference. I feel that his work was a huge success.

I met and spent time with several other very impressive people:

Mark Miller, formerly of Xerox PARC, and now of Google Research. He’s open source coordinator for the E programming language, a pioneer of agoric (market-based secure distributed) computing, and an architect of the Xanadu hypertext publishing system. I found him to be a remarkably clear thinker and good explainer.

David Ungar, formerly of Stanford and then of Sun Research and now at IBM, was co-designer of the “Self” language, a particularly simple object-oriented langauge in which there are no classes: objects just inherit from objects. He achieved an amazingly performant implementation.

Richard Tibbetts, formerly of MIT, is a co-founder, architect, and team lead at StreamBase, developing a Complex Event Processing Platform. This is one of the startups based on Mike Stonebraker’s recent research work. We talked about databases, startups, and also about back-home things since we live near each other.

Michael Ernst, an MIT professor working on adding abstract types to Java (JSR-308). This will be one of the major language upgrades in Java 7. It’s great to see that there is interest at MIT in Java in specific and programmer productivity in general. He gave a talk later that I wasn’t able to get to, about his Javarifier (inferring annotations to the code) and Javari (type checker) tools.

One thing I heard from many people was a uniformly negative verdict on the new R6 Scheme standard. This is really too bad, since a lot of work went into that standard and the Scheme community really does need a standard. But there are so many problems with the new standard that nobody seems to like it and many Scheme implementors are not intending to support it.

There were so many interesting papers that I can only hit the high points of the most prominent ones. Here goes.

The Dynamic Language Symposium:

Mark Miller of Google Research talked about security: fighting the malware crisis by running everything with least privilege, using capability principles in languages. He discussed various approaches to retrofitting such concepts into existing languages. JOE-E (Google, HP Labs, U.C. Berkeley) is a subset of Java (so all the tools still work and there is no slowdown), checked by a static verifier. W7 (Jonathan Rees’s thesis) and Cava (at Google) work by inserting runtime checks, which allows more flexibility but there’s a slowdown. I am pretty convinced that capability-based systems are the way of the future and if I ever get to work on them, I’ll follow all the references in this talk.

Alexy gave a shortened version of the talk on Probabilistic Scheme that he gave at ITA recently. I felt that it got a good reception.

Morten Kromberg, of APL-vendor Dyalog, talked about how they have added object-oriented programming to APL. APL is still alive and well, being used by engineers, financial analysts, actuaries, and data miners. Apparently many domain experts can learn to program APL themselves, which gives them a lot of personal power. Insurance companies are particularly APL’s sweet spot. The objects use the Microsoft CLR, and APL can treat the contents of an Excel spreadsheet as an APL array!

Shriram Krishnamurthi, a well-known Scheme and languages professor from Brown who I first met at the original Lightweight Languages Workshop at MIT, talked about Relationally-Parametric Polymorphic Contracts. It was about how to do contracts in your language less painfully without giving up power, involving inferencing at runtime using unification (in the sense of things like Prolog).

Jim Hugunin of Microsoft talked about the new Dynamic Language Runtime, essentially an extension of the Common Language Runtime to make it easier to write .NET compilers for dynamic languages. Jim, who had earlier implemented Jython (Python in the Java Virtual Machine), set out to show that the CLR was no good for dynamic languages by showing how bad Python would run. But after he implemented IronPython for the CLR, he was surprised to see that it actually performed very well! Although it has to do some gross things to work dynamically (e.g. each add has to check both argument datatypes manually), it still runs quickly in comparison to the original Python because the original Python is an interpreter, whereas IronPython takes advantage of the JIT compiler in the CLR. His group is also doing a JavaScript, and they intend for the CLR/DLR to be a suitable platform for Ruby, PERL, and so on. He went into a lot of detail and answered many questions.

The Montreal Scheme and Lisp User Group:

Manual Serrano, from INRIA (the French research institute), maintainer of Bigloo Scheme, talked about “HOP”, his system for allowing you to create highly dynamic web sites with much more elegant code than the usual jumble of HTML, CSS, JavaScript, and so on. His language lets you write code and succinctly flag which parts should run on the client and which on the server, at a very fine grain. He showed some very advanced web behavior in only a few lines of code. Very impressive!

Pascal Costanza of the Programming Technology Lab of the Vrije Universiteit Brussel, Belgium, talked about his work in context-oriented programming based on CLOS. (Pascal is also known for his Common Lisp documentation and for work on aspect-oriented programming in CLOS.) Context-oriented programming lets you create “layers”, which can be turned on and off orthogonally and dynamically within a thread. When a layer is established, an associated set of instance variables and methods become visible in CLOS classes. Amazingly, he was able to build all of this on top of the CLOS meta-object protocol!

Finally, the OOPSLA conference itself:

Kathy Sierra talked about how to create passionate users; that is, how to make products that will get users excited and enthusiastic. This was completely fascinating and I took two whole pages of notes. She has given this talk before, and there is commentary on the web about it already; see:

http://conferences.oreillynet.com/cs/os2005/view/e_sess/7077

http://www.oreillynet.com/onlamp/blog/2005/08/tutorial_review_creating_passi.html

And a blog at:

http://headrush.typepad.com/creating_passionate_users/

She’ll also be writing a book about it soon, to be published by O’Reilley. She’s also the co-author of a series of books called “Head First xxx”.

Jim Purbrick and Mark Lentczner of Linden Labs, the creators of Second Life, explained a great deal about what Second Life is and how it works. The most interesting thing is that 15% of the residents actually do coding, in a language that lets you make active objects. There are 30,000,000 running scripts, 2.5 billion lines of code. Generally there are 15,000 scripts actively running on each “region” (processor), updating at 45 frames per second, and there are 4,000 processors. There are 30,000,000 concurrent threads. The language itself they described as “terrible”; they are working on bringing up the Mono implementation of the CLR so they can provide good languages. There are some complicated issues in which threads must be migrated from one CPU to another as an object moves around within regions of the Second Life world. The most impressive thing is how many people with little or no technical training are doing programming. They also talked about how the Linden Labs developers, eight of them at five physical locations, use Second Life itself as a collaborative development environment; they say it works really well, particularly due to having stereo audio that’s good enough that you can tell where sounds are coming from. During the talk they showed Second Life on the big screen and moved around and interacted with people, so that you could see what it’s like. I, myself, hardly have enough time to live my first life, let alone a second life, but it was quite fascinating to see what’s going on in there.

That evening, Guy L. Steele Jr. and Richard Gabriel did a presentation that I can only describe as a cross between a lecture on the history of programming languages, and piece of performance art. They said exactly 50 things, each in exactly 50 words, so it sounded like haiku a lot of the time. They presented many real languages from the 50’s to the present, including some very weird ones, plus several joke languages (such as the famous Intercal which does all its output in Roman numerals). I was particularly thrilled when they used the xkcd.com cartoon called “Lisp”, as well as a hilarious “filk” song about how God created the universe in Lisp. The performance got a huge standing ovation. These guys are so amazingly clever.

Fredrick Brooks Jr., author of the classic book “The Mythical Man-Month”, talked about telecollaboration. Most of the talk was about collaboration itself, and under what circumstances it’s a good thing: not always! His main point is that collaboration is great for determining system requirements and brainstorming about possible approaches, but that you really need a single system architect in order to achive conceptual integrity. The system architect can delegate parts of the architecture to others (e.g. the user interface czar), but he distinguishes sharply between delegating design (OK) and sharing design (not OK). He had insightful things to say about the open source process (good for some kinds of software but not others), pair programming (he likes it), and object-oriented programming (the most powerful technique to come along so far to improve software productivity). He is an extremely clear speaker with a lot of good points to make. He’ll be writing a book of essays soon.

Then there was a panel session entitled “No Silver Bullet Reloaded”, based on the famous 1986 paper “No Silver Bullet” by Fredrick Brooks. See

http://en.wikipedia.org/wiki/No_Silver_Bullet

and follow the link to the paper. This is one of the best and most important papers ever written about software engineering; if you haven’t read it, you should.

The paper starts: “Of all the monsters that fill the nightmares of our folklore, none terrify more than werewolves, because they transform unexpectedly from the familiar into horrors. For these, one seeks bullets of silver that can magically lay them to rest.” The paper surveys all the major techniques that have been proposed for making software engineering far more productive, considering each carefully and explaining why none is a “silver bullet” for solving the “essential” complexity of software. At the panel, well-known author Martin Fowler converted himself into a werewolf (with a funny mask) and made all his points in the negative, which was hilarious. Of course all the panelists agreed that there are no silver bullets. David Parnas was the most specific about what he meant by a “silver bullet”: there is no technique that’s easy to learn (e.g. a three-day seminar) that’ll solve the hard problems of software design and construction. They also all agreed that it’s hard to talk about what techniqes improve productivity by a factor of ten since there really isn’t any good way to measure productivity. Many good and interesting points were made. Brooks highly recommends the book “Peopleware” (I have a copy but haven’t read it yet; clearly I should.)

The next keynote speaker was John McCarthy, one of the creators of the field of Artificial Intelligence, the creator of the Lisp language, the long-time director of the Stanford Artificial Intelligence Lab, winner of the Turning Award in 1971. Unfortunately, this was the one disappointing talk of the conference. He had very little to say, and he just read his slides.

David Parnas of the University of Limerick, one of the first people to define and expound upon modularity in software, was the next keynote speaker. He defines object-orientation as “designing software by desiging and implementing data-holding objects that make the job easier” and insists that you don’t need any special language to do it: object-oriented is about how you design. His talk stressed our inability to write clear specifications for software. (The EU is currently fining Microsoft $3M/day until Microsoft produces a clear and precise specification for their server [I'm not sure exactly what software we mean here], but it’s just too big and hard to do. Microsoft offered the sources, but nobody wants that, since the sources just say what the software happens to do now, not what it’s defined to do over time.) He advocates much more precise and specific documentation, kept up to date, to the point where if someone asks a question about the software, a programmer would go to the documentation rather than the code to answer the question. It finally turns out that he wants extremely precise specs written in a mathematical notation, which is very possibly appropriate for the kinds of software projects that he was discussing (avionics, nuclear power plant safety) but not for anything I’ve worked on. I’m in sympathy with the call for far better code documentation, though.

Next up was my old friend Gregor Kiczales, one of the creators of the Common Lisp Object System and the father of Aspect-Oriented Programming. He talked about Effectiveness Without Formality. This talk was very deep and philosophical. It was based heavily on the work of two Xerox PARC scientists/philosophers, Lucy Suchman (”Human-Machine Reconfigurations”) and Brian Cantwell Smith (”On the Origin of Objects”, known to be a difficult book to read). Unfortunately I had no familiarity or touch-points for their work, and partly for that reason, Gregor’s talk was, frankly, over my head. Among his key points: There is conflict between the power of programming langauges and the flexibility of design patterns. There is conflict between the mathematical foundations of programming and the way people really work. The world is sloppy and imprecise. There is a big distinction between programming languages that are “effective” (actually make things happen), and formal abstractions which are not “effective”. People never understand each other precisely, but we can deal with this. He talked a lot about “intensionality”, which he described as “the ability to be ‘about’ something”, and the intensional relationships between users, code, bug databases, and all the other entities of software development. The world is as far from formal as you can imagine! Abstractions are transient, shifting, and negotiated. That’s about as much as I got from this. If it were not Gregor, I would have been dubious about the whole thing. I asked whether he had any idea how these ideas might turn into something useful, and he said, not yet. On the way out of the room, Mark Miller said that this was not only the best talk at the conference but the best talk he had heard in years. So if the people smarter than me think that, I should probably try harder to understand.

Jonathan Edwards, a researcher at MIT CSAIL, gave a very interesting talk about conditionals as tables. He presented a two-dimensional graphical notation for representing conditionals, which, as I see it, takes something that was imperative and makes it declarative, always a source of power. This makes it much easier to specify multi-armed conditionals, in that you say what you want rather than how to achieve it, much as in a database query language. So far this only works for functional (no side-effect) programs and it’s not clear yet whether the idea scales to larger programs (he showed a very elegant Fibonacci example). A lot of the smart people in the audience were very impressed.

Sebastian Fleisner from Hong Kong talked about Epi-Aspects: Aspect-Oriented Conscientious Software. By “conscientous” he means “self-sustaining”, continuing to work in the face of failure. This is based on a paper last year by Richard Gabriel and Ron Goldman about “Conscientious Software”. Its abstract is: “Software needs to grow up and become responsible for itself and its own future by participating in its own installation and customization, maintaining its own health, and adapting itself to new circumstances, new users, and new uses. To create such software will require us to change some of our underlying assumptions about how we write programs. A promising approach seems to be to separate software that does the work (allopoietic) from software that keeps the system alive (autopoietic).” The presentation talked about the latter part as having operations like monitor, test, repair, and kill. Their proposal has to do with using aspect-oriented programming to achieve this. What they’re talking about sounds a lot like some of what I’m currently doing in my own work on Hot Upgrade, and I intend to read both papers and see if there are any good ideas I can use.

Pattie Maes, from the MIT Media Lab, was the next keynote speaker. (As you can see, they had a lot of keynote speaker and I spent most of the conference at keynotes!) She talked all about interesting projects being done at the Media Lab to try to give people access to information at their fingertips when they’re not sitting at the computer. This was all quite interesting and fun although it didn’t have much to do with the subject matter of the conference.

Finally, the last talk was by Brian Foote of Industrial Logic Inc.: “Big Ball of Mud: An Introduction to Post-Modular Programming”. The key point was that most real-world systems use a “big ball of mud” (totally unmodular) architecture, and why is this architecture so popular and successful? Maybe we should give up trying to teach people to be modular, and embrace the big ball of mud. The talk was very rambling and often funny, and I can’t figure out whether it was three-quarters tongue-in-cheek or entirely tongue-in-cheek. By this time I was so exhausted that I was yawning through the talk, anyway.

There were plenty of other talks that sounded interesting, but I could only be in one place at a time, and I’m not sure that I could have absorbed even more material in one week anyway! The set of speaker was nothing short of stunning, and I also learned a great deal from casual conversations during the breaks. I’m back home now and nearly over my illness, and ready to read all the “to do” and “read this” notes I wrote during the conference and get started.