Dynamic lisp - dynamic databases?

Has anyone seen or worked on a system that must run all of the time, and could change underlying databases while it is running (and logging/reading/updating)? I would think that the dynamic nature of Lisp would lend itself to that kind of solution. I could almost see a system that has an old DB, and someone puts a new DB next to it, and loads some lisp code (maybe in a separate thread) that updates the functions that read the database so that they get the info from the new DB instead. Perhaps even have a cache, and gradually migrate data from old to new as it is read/updated, and as time goes on.

It would see it as an super-stable, never-needs-restarting kind of platform

Does anyone think that is possible Do people do that but just keep the same DB and fiddle with tables? Although would people even do that while the system is up?

I've been thinking about things like this ever since I heard that the Internet hasn't 'gone down' since it was created back in DARPA days, but none of the original hardware (or protocols?) are part of the Internet any more. It's like any part of it can be upgraded, and eventually was, without taking it offline. It seems so organic.

Jonathan

Yes, Lisp would be good for this sort of thing. Erlang as well. I know that there were some attempts to build non-stop telecom solutions in the 1980s/1990s using Lisp, but I don't know if those ever went into production. Erlang has been used to build such and there are several good case studies of gear that has gone into service using Erlang and achieved essentially 100% uptime. All the right concepts are there in each system.

I'd also caution you a bit on the line about "the Internet has not gone down." While strictly true, it's misleading. It's like saying that ever since the automobile was popularized, people have been driving every minute of every day. While it's true that somebody, somewhere, has been driving at all times, any individual car can be down for repairs for quite a long time. The odds of every car, worldwide, being down for repairs exactly at the same time is low, but that doesn't mean that your own car is reliable. I have seen service failures of huge web sites (yes, even Google properties) and large ISP networks. Even with the general reliability of the Internet as an overall system, we still have a ways to go with the reliability of any individual component. Like you, I'm very interested in that aspect of software engineering and it was one of the things that attracted me to Lisp.

There are various ways to handle the upgrade of a component of a running system. Essentially, they all boil down to:

• Separate the behavior from the state. Loosely, this means teasing apart code and data when you are designing the system. This is necessary because you want to retain the existing state and yet replace the old code with new code, having the new code then start to operate on the old data. You need to have clear boundaries and documentation on things like data formats.
• Once you have written things correctly, when you are ready to upgrade, you essentially persist the data someplace (could be disk, but most often just a place in memory).
• Then, quickly, you reload some new behavior and start executing it, pointing to the old state that you have persisted.

Depending on what you are upgrading and how long you have to do it, you can do this in various ways. If you're just upgrading a function or two, you might simply reload the function in a Lisp image and move on. If you're doing a database update, you might bring up the new database in parallel, sync data with the old database in real time, and then, once things are fully sync'd, start sending all new database queries to the new engine. If you're upgrading an OS kernel, you might need to persist everything to disk and then reboot. You can sometimes hide any disruption of this by using redundant hardware to cover the reboot time. In general, the lower and more fundamental the code to the system, the more traumatic the replacement is going to be.

You raise good points. Thanks for the reply, findinglisp. It seems that there are (at least) two types/categories of persistence:

One is like the internet, or the automobile example, which seem reminiscent of cellular bodies, ant colonies (nature seems to have a lot of these), and distributed human organisations (terrorist cell groups?). All the parts seem to have very similar behaviour, and you can get emergent behaviour. Let's call it the distributed group.

The other 'type' would be well designed systems that have special interfaces (almost unique) that allow them to replace a given component with another component, like a jet being refueled (and repaired) in the air. Let's call this the designed group. Although it seems that nature and economics seem to push towards the first group, made of many cheap, replaceable components. Actually, ant colonies would be an example of this too, when the ants need a new queen they must grow one, and as far as I know, they only have one queen at a time, and if they have two, one will take some workers and start a new colony.

Thinking about Lisp, as a whole to me it seems big and firmly on the way to the second, 'designed', category, with specially designed custom parts. However, if you look inside, you see a dynamic world of functions, classes, bindings, methods, all able to replace and be replaced at any time - to me this seems more organic, and at least partially in the 'distributed' category. Perhaps this is why a lisp machine would appeal to me so much - the monolithic core is part of the platform, not the operation, and so all you see is the cool dynamic internals.

Then again, maybe it's not about "two types of persistence", but more of a spectrum, because the second category starts to look more like the distributed group the more bits there are (I suppose that the parts lose their individuality in the crowd).

Actually, I think I got a bit off of persistence ...

Kohath wrote:You raise good points. Thanks for the reply, findinglisp. It seems that there are (at least) two types/categories of persistence:

One is like the internet, or the automobile example, which seem reminiscent of cellular bodies, ant colonies (nature seems to have a lot of these), and distributed human organisations (terrorist cell groups?). All the parts seem to have very similar behaviour, and you can get emergent behaviour. Let's call it the distributed group.

The other 'type' would be well designed systems that have special interfaces (almost unique) that allow them to replace a given component with another component, like a jet being refueled (and repaired) in the air. Let's call this the designed group. Although it seems that nature and economics seem to push towards the first group, made of many cheap, replaceable components. Actually, ant colonies would be an example of this too, when the ants need a new queen they must grow one, and as far as I know, they only have one queen at a time, and if they have two, one will take some workers and start a new colony.

Thinking about Lisp, as a whole to me it seems big and firmly on the way to the second, 'designed', category, with specially designed custom parts. However, if you look inside, you see a dynamic world of functions, classes, bindings, methods, all able to replace and be replaced at any time - to me this seems more organic, and at least partially in the 'distributed' category. Perhaps this is why a lisp machine would appeal to me so much - the monolithic core is part of the platform, not the operation, and so all you see is the cool dynamic internals.

Then again, maybe it's not about "two types of persistence", but more of a spectrum, because the second category starts to look more like the distributed group the more bits there are (I suppose that the parts lose their individuality in the crowd).

It actually isn't about persistence so much as the time that a binding is made to a resource. I think Alan Kay would describe this as late binding vs. early binding. If things are late-bound, then they are easier to upgrade on the fly because the question of which "object" (may not be an object in the OOP/CLOS sense; can be a function or "object" in the Lisp type sense) will service my request is determined right at before you actually issue the request. This means that you have an opportunity to alter the binding in real time as you upgrade. For instance, the function that gets invoked is the current value of the FOO symbol, not the function it referenced 100 milliseconds ago.

If things are static bound, it's much more difficult. You essentially do very few dynamic lookups and as a result the system is much more tightly coupled. This provides far fewer opportunities to redirect requests to a new object/server.

For instance, imagine the Internet didn't have DNS and that everything worked off of hard-coded IP addresses. You'd see a lot more "outages" because a service would always be bound to a static address. This still happens because DNS uses caching with large timeout values (hours/days). All major sites (Google, Facebook, etc.) also use load balancers to redirect traffic at a fine granularity and rapidly detect server outages and route traffic around the failures. That's simply an example of a proxy-object that is designed to turn a pseudo-static binding (DNS is pseudo-static when compared with millisecond-by-millisecond traffic management performed by a load balancer) into a late binding. All this can help you get around the failure or restart of an individual server.

So, how this relates to programming languages is that with C you're effectively static bound all the time and thus it's very difficult to upgrade a running process on the fly. Yes, you can do it with loadable libraries by unloading and then reloading, but it's a lot of work and depends on the correct operating system support. Most of the time, you take the easy way out and simply restart the process with the new code. In contrast, late-bound languages like Lisp and Smalltalk make this relatively easy at a very fine granularity. This is why it's customary to start up a long-running Lisp or Smalltalk image and then update it on the fly, saving a core file when you're "done" so that you can recover the dynamic running state at a later time.

Systems that are designed for high-availability (e.g. telephone switches) often use redundant hardware to help with upgrades. For instance, it's a requirement to be able to upgrade a telephone switch while calls continue to be processed. Most current designs do this using two call processing hardware modules (which also serve to handle redundancy in the case of hardware failure). They simply boot up the new code on the backup module, and then simulate a "failover" to the new code. The primary is then upgraded while the original backup module handles call processing. You can then either switch back to the primary, or you can swap the roles until the next upgrade. Even with redundant hardware, switchover between the two modules is often tricky. It's hard to capture all the transient state, so typically there is some small hiccup that affects some portion of the system. For instance, in the telephone example, all existing calls at the time of switchover would be preserved, but any call that had just started to be set up and had not yet been established would be failed; we simply expect the user to redial in the unlikely case that they happen to place a call at the exact moment we're performing an upgrade switchover. The main thing is, the possibly thousands of existing calls through the switch remain intact and these users never notice the disruption.