Anyone using Lisp for web in production?

Hi, does anyone around here been using Common Lisp for developing real web applications?

I've been using PHP for all my web programming but since i started learning Lisp i feel like i could really take advantage of this great language, so i started playing around with hunchentoot, cl-who, clsql and parenscript. But i'm worried about how it would work in a production environment. How well does CL scale in terms of performance, can i have different applications binded to different domains on the same server with a single lisp image or do i need multiple images running, etc... So anyone using CL for serious web work please post your experiences, issues and advices.

Thanks in advance to everyone.

I'm using SBCL for http://streamfocus.com/, a recently launched startup for online project and task management.

We're using some old web libs that work well enough, but if I were to start today, I would go with the ones you mentioned, hunchentoot, cl-who, etc. We do us cl-sql, but only the functions, not the CLOS stuff.

Since we're new, we unfortunately haven't hit scaling problems yet. My informal tests seem pretty good. I've gotten good use out of sbcl's built-in profiling functions, so when I find a page that's slow, I can easily find the offending code and fix it. Usually, it's been the normal high-level stuff like hitting the database too many times. In one case, it was pure code, but optimizing it with vectors and declaring integers fixed it up easily.

I'm running my lisp webserver on a different port, then using apache in front proxying to the lisp. The reason for this was SSL support, though you can get lisp to do ssl with the right tools. I don't know about binding multiple domains specifically, but certainly in this type of setup, you could make apache do it. And running a lisp webserver is usually a function call, not a config file for a process, so you can run as many as you want on different ports within the same process.

I find being able to update the running process though a REPL a wonderful advance over my previous experience with Java. However, I don't have a good way to replace an entire application in place like I can with a Java EAR. I'm not sure what the solution is, but I haven't ran into a real problem yet.

I love that html is generated from lisp code (like cl-who), rather than html templatized with code, but that does only work when it's programmers writing the web pages rather than designers.

-Mike

And, Mike? You're in a situation where you have a full server box or a big slice of one? or is it being served through some webhosting company?

Thanks,
Patrick

nklein wrote:And, Mike? You're in a situation where you have a full server box or a big slice of one? or is it being served through some webhosting company?

We run on amazon ec2, Ubuntu specifically (http://groups.google.com/group/ec2ubuntu). There are cheaper options, but I liked the amount of control I had with ec2, and it's working well.

Not quite in production yet; give me a couple of days. Seriously, I've four sites that should be going beta tomorrow.

It's taken me a while to come up to speed, as my programming background before Lisp was SQL and scripting in shell, perl and PHP. Now that I've made progress up the learning curve, it's an awesome language to work in.
I'm using Hunchentoot, cl-who and CLSQL, with apache in front and PostgreSQL behind. Like Mike, I'm not using the CLOS stuff in CLSQL, and I'm looking at moving to postmodern. Oh, and mel-base is handy for email notifications, plus cl-pdf will be in use for prettier invoices before long. With a bit of refactoring, I've abstracted most of the moving parts into a couple of libraries of my own, and I can now spin off a new site in minutes flat - most of the incremental time for a new site is in the graphic design. Handling multiple domains with a single lisp image isn't a problem; I use apache to manage the domains and SSL termination from the outside, and run a separate hunchentoot server with its own port number and dispatch table for the active portion of each site. Because most of the functionality comes from a common library, adding a site adds very little memory overhead until it starts getting serious traffic. Of course, I say this without having seriously profiled or load-tested them

I looked at EC2, but still have an issue with Amazon's abuse of software patents (one-click checkout is patentable?!), so I'm reluctant to give them money. I've found http://www.flexiscale.com/, who provide a similar service. This allows me to scale up very quickly if (hopefully when) the sites mature a bit and come to wide public attention. For now, I want them to stay relatively low-profile, because the single worst thing that could happen is that thousands of potential customers visit in the early days, conclude (probably rightly) that it sucks, and never come back. I'm anticipating a lot of refinement right after they go live, and it's easier to do that when you don't have a large userbase to annoy.

The other thing that will help scaling is application architecture: each site shares a base library with the rest, but doesn't share its database or any state. So I can move any given application to a new server, update apache, and instantly handle more load. I haven't yet checked, but it should even be safe to run multiple instances of a given site against a common database server, and load-balance them via apache. The databases all use the same schema, but are separate, so again can be moved to a different database server if contention becomes an issue; this is much less trivial than moving an appserver, proportional to the amount of data that needs moving, but can still be done.
Initially, everything runs on a single server. As traffic ramps up, I'll move the most demanding server (web, app, database) onto its own virtual host. If one of the sites is hogging everything, it'll get its own set of hosts.

A handy hint about using virtual hosting: go for Xen. Virtuozzo doesn't play nice with SBCL; there's a problem with memory allocation. There was when I tried it, anyway. I've no idea about VMware, never having tried it.

Updating an app is relatively easy, with asdf. I use darcs for version control, which is great for producing tarballs of a release and gives me release versioning for free. I combine that with symlinks into asdf's system directory. Unzip a new version, update the new link, fire up a separate lisp image to pre-compile and smoke-test the new version (you never know for sure), then update the running process. To do that, I use screen: reattach to the image actually serving sites, (use-package :cl-user), and use asdf to reload whichever site that needs upgrading. Of course, I haven't tested what happens when you do this while the site's under load, so I may need to stop that application while I upgrade it (10 seconds' worth of interruption).

Disclaimer: I have a few years of experience as a web- and application-server admin in a high-traffic environment, so I learned a few tricks about architecture and version management

Apache is my choice because I've used it for years, I know it well, and I know from experience that it'll handle fairly high loads. I've heard good things about other webserver/proxy software, but will shop around if and when apache starts cracking under the strain of a single site. It's also battle-hardened, so I'm much more comfortable with that exposed to the internet, than I would be by sticking hunchentoot straight out there.


In case you're wondering: yes, there's an excellent chance that I'll release and open-source the libraries if it looks like a worthwhile contribution. However, they need a lot of refinement before I can do that with a straight face.

JamesF wrote: A handy hint about using virtual hosting: go for Xen. Virtuozzo doesn't play nice with SBCL; there's a problem with memory allocation. There was when I tried it, anyway. I've no idea about VMware, never having tried it.

SBCL works find in VMware. I use it that way all the time. I could see how Virtuozzo might cause problems. There is some interesting memory mapping going on with Virtuozzo. With VMware, all that happens deeper under the covers so it's less of a worry.

...then update the running process. To do that, I use screen: reattach to the image actually serving sites, (use-package :cl-user), and use asdf to reload whichever site that needs upgrading. Of course, I haven't tested what happens when you do this while the site's under load, so I may need to stop that application while I upgrade it (10 seconds' worth of interruption).

What's your experience running things under screen? I use screen a lot for minor interactive use, but if the screen instance dies, all your attached processes die, too, right? Also, how do you handle unplanned reboots, for power outages, say? Does everything come back up correctly using rc.local or something like that, or do you have to restart it all by hand?

Finally, thanks for sharing your experience. Very interesting!

Haven't had a problem with screen so far, so I haven't worried about that. I'd prefer to use slime, but can't stand emacs; long-term, I'm hoping to equip climacs with vim bindings and use that. I've actually been running a lisp server that way for a few months now (demo version of one of the sites), and it's been fine.

Getting it to start automatically is something I haven't actually tried yet, I have to admit. There's an awful lot of work to be done after the initial go-live, and this is one of those little tasks. From a cursory look-over, it should be entirely possible to write an /etc/rc.d/ script that'll run as the appropriate user and kick off a detached screen session, which in turn calls the script that starts up the lisp process.

sbcl's quite happy to run arbitrary commands on startup; I have a script on my laptop that takes advantage of this, which makes life nice and easy for development. It kicks off a couple of xterms, starts sbcl in one and gets it to load the appropriate asdf system before calling (use-package) to get me into the package itself, then fires up gvim and tells it to load up whatever file I'm concentrating on. Lots of moving parts all over the place, which I'm not so happy about, but they're all small and have been reliable so far.

I'm still debating how to handle automated shutdown as well. The leading contender is yet another hunchentoot server process, which functions as a quick-n-dirty RPC server: call the right URL, and it'll shut down each of the servers before telling sbcl to quit. That one will be firewalled from everything except the local host, and invoked via something like curl or wget


I'm always happy to share what I've learned. Apart from anything else, I've gotten so much benefit from FOSS that it's one way of contributing in return.

JamesF wrote:Getting it to start automatically is something I haven't actually tried yet, I have to admit. There's an awful lot of work to be done after the initial go-live, and this is one of those little tasks. From a cursory look-over, it should be entirely possible to write an /etc/rc.d/ script that'll run as the appropriate user and kick off a detached screen session, which in turn calls the script that starts up the lisp process.

Yea, I'd definitely make sure this works well. After the obligatory power outage takes down everything in the data center, you don't want to be faced with restarting oodles of Lisp-based applications all by hand. Not fun.

And yea, I know that the colo always has backup power, so that never happens , but just in case...

After the obligatory power outage takes down everything in the data center, you don't want to be faced with restarting oodles of Lisp-based applications all by hand.

No, no I certainly don't. I've refined it to a pretty quick and simple process, but it'll be automated before long. The reason it hasn't happened yet, and the reason it will before long, oddly enough, are the same: professional experience. Outages like that are unusual, and the Linux platform is nice and stable, so the actual urgency is low. However, I know from painful experience that it's easy to get complacent and forget, which bites hard when you do reboot the thing and find that nothing came back up. Also, the last thing you need to worry about is whether you remember to start up everything, and whether you started it all in the right order.

This last point is another one that can bite, too: getting things to start in the right order, when the dependencies are on different hosts. That's a case of getting the startup script to confirm that the dependency is running before it starts up its own application.

Monitoring and alerting systems are the next layer to go on, followed by performance monitoring and analysis... actually, all this infra stuff will probably be more useful to other people than the application code itself.

JamesF, yup, exactly.

JamesF wrote:From a cursory look-over, it should be entirely possible to write an /etc/rc.d/ script that'll run as the appropriate user and kick off a detached screen session, which in turn calls the script that starts up the lisp process.

I've found it useful to have a startup script that runs something in a detached screen before. The relevant screen options are:

       -d -m   Start screen in "detached" mode. This creates a new session but
               doesn't  attach  to  it.  This  is  useful  for  system startup
               scripts.

       -D -m   This also starts screen in "detached" mode, but doesn't fork  a
               new process. The command exits if the session terminates.

I'd just like to thank you guys for the rich information provided. I'd really like to tryi and incorporate Lisp into my company's production pipeline but i'm still far from managing that, with all the various info provided i believen i'm now a step closer to it. I'm working on some community site and i'm currently prototyping it in Lisp, if i get the guts by the time the prototype is done maybe i'll actually stick with the Lisp version instead of going back and rewrite it in PHP.

Again, my thanks to everyone for the valuable information.

Wodin: thanks! That's exactly what I needed.

JamesF wrote:Wodin: thanks! That's exactly what I needed.

Glad I could help

We use SBCL/UCW for http://paragent.com.

I switched from screen to detachtty to wrap my server lisp processes. Andreas Fuchs wrote a couple scripts to hook that into the normal unix rc.d machinery to get autostart at boot time and more. Read about it here: http://boinkor.net/archives/2006/02/sta ... age_1.html.

I think I had to alter the scripts slighly, as current detachtty includes the equivalent functionality to Marco's patch but with different syntax. Anyway, works great for me.

At ITA, we are convinced that hunchentoot is the best HTTP server in Common Lisp. Previously we used Araneida, which is also quite good, but Hunchentoot is better, and has some things we needed like Unicode support.

We have made enhancements to Hunchentoot, which of course have been or will be incorporated in the mainline open source code. It doesn't hurt that the author of Hunchentoot is currently working for us as a consultant, although someone else has been doing most of the enhancements.

I'd be very interested to hear what people are using to generate HTML, JavaScript, and such. There are several open-source systems running around that are quite cool-looking, such as UncommonWeb (UCW), Weblocks, and so on. Has anyone had experience with any of those?

(However, at ITA the main web sites we are building are actually done in Java, using a great deal of open-source technology such as Struts and advanced Ajax libraries.)

dlweinreb wrote:At ITA, we are convinced that hunchentoot is the best HTTP server in Common Lisp. Previously we used Araneida, which is also quite good, but Hunchentoot is better, and has some things we needed like Unicode support.

We have made enhancements to Hunchentoot, which of course have been or will be incorporated in the mainline open source code. It doesn't hurt that the author of Hunchentoot is currently working for us as a consultant, although someone else has been doing most of the enhancements.

What kind of load having you been putting on Hunchentoot & how has it been scaling? What kinds of things have you been using it for at ITA, if don't mind me asking?

dlweinreb wrote:I'd be very interested to hear what people are using to generate HTML, JavaScript, and such.

I'm using cl-who, which is working well enough. The only gotcha I've found so far is getting the hang of when to use (format nil ...) vs (format t ...). Haven't tried playing with javascript yet.

JamesF wrote:Haven't tried playing with javascript yet.

Parenscript is the way and the light. It can't hide every ugliness of javascript (you have to include return statements more often than I'd like) but you get to create macros to hide away a good chunk of the more ridiculous boilerplate. For example (stealing from my SVG-GUI library), here's how you add a new method in javascript's prototype system:

Transform.prototype.setScale = function (x, y) {
    this.scale = { x : x, y : y };
}

Gah! This works, but masks your intent somewhat. Here's some CL support:

(defpsmacro defpmethod ((name class) args &body body)
  `(setf (slot-value (slot-value ,class 'prototype) ',name)
         (lambda ,args ,@body)))

And here's the parenscript that produced the javascript above:

(defpmethod (set-scale *Transform) (x y)
  (setf this.scale (create :x x :y y)))

The closure-as-protected-namespace stunt is a lot nicer hidden behind a macro, too.