Clozure Lisp CGI Programming

I'm attempting to set up CLozure CL for CGI programming and have run into a snag.

I can run the program perfectly from the command line, but it won't run through the browser. To make sure that the webserver was properly configured I wrote a simple hello world app in C and ran that through the cgi interface and it worked.

Here is the program.

(defun test()
  (format t "Content-type: text/html~%~%")
  (format t "this is a test"))

(ccl:save-application "index.cgi" :toplevel-function #'test :prepend-kernel t)

Like I said everything works fine until it's run in apache. Here is the error log,

Couldn't load lisp heap image from
[Fri May 08 10:51:21 2009] [error] [client 127.0.0.1] Premature end of script headers: index.cgi

Why would the image load properly from the terminal but not from the browser? Permission issues? I ran Opera as root and had the same problem.

Thanks for any help.

Try switching to the apache user and running the lisp program to see what the error is. You could also run the lisp program from a batch file, and redirect the error output to a file.

I gave full permissions to apache (www-data in Ubuntu) and I still have the same problem.

I think that it's highly unlikely that it's a permission problem because I compiled an equivalent C program and ran it without a hitch. I applied the same permissions from the C program to the lisp program and ran it and had the same problem.

I'm going to try compiling it without preprepending the kernel and see if I can load the image that way.

Any further ideas?

I just tried your example with Ubuntu 8.04 (32-bit), Apache2, ccl 1.3, and it worked for me as it should...

If you build the executable with the Lisp source file, lx86cl, and lx86cl.image all in the same directory, does it build the image file differently?

Try loading a simple shell script as your CGI process:

cat <<_EOF > test.sh
#!/bin/sh

echo 'Content-type: text/html'
echo
echo
ulimit -a
_EOF

[The lines between _EOF go in the file test.sh]

I'm wondering whether apache hasn't set a the memory limits too small for clozure. If so, there should apache config settings (e.g. in httpd.conf or cgi.conf) to control this. Otherwise, you might also try asking on the clozure mailing lists or IRC.
http://trac.clozure.com/openmcl/wiki/WikiStart#Support

Hero Doug wrote:Why would the image load properly from the terminal but not from the browser? Permission issues? I ran Opera as root and had the same problem.

Hi

Running Opera as root is as likely to help as is wearing your lucky rocket-ship underwear ;-)

i.e., it's not Opera that is having trouble. It is Apache.

No, I ran it as Opera to see if the root user also had the same problems. I assumed (I'm not a Linux expert yet) that since root has full control over the system that any permission problems my regular account were having wouldn't apply to the root user. However, since the root user it getting Apache to do the work I guess it was a futile test.

Also, here is the result of the shell script.

time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) 0
memory(kbytes) unlimited
locked memory(kbytes) 64
process unlimited
nofiles 1024
vmemory(kbytes) unlimited
locks unlimited

P.S. As a side note, SBCL is working fine. So maybe is has to do with the amount of memory CCL is requesting.

Those limits don't look particularly restrictive. They probably match what you get by typing "ulimit -a" at a normal shell prompt. If so, this isn't your problem. If not, "ulimit -s 8192" and then starting ccl from the same shell should reproduce the problem separately from apache.

Hero Doug wrote:No, I ran it as Opera to see if the root user also had the same problems. I assumed (I'm not a Linux expert yet) that since root has full control over the system that any permission problems my regular account were having wouldn't apply to the root user.

My point is that this is a CGI script that is running on a web server, right? The browser could be on any machine, including, e.g., a Windows machine that does not have a user called "root". So running Opera as root is irrelevant.

Hero Doug wrote:However, since the root user it getting Apache to do the work I guess it was a futile test.

Yes, except most Linux distributions do not run Apache as root. As you mentioned, Ubuntu uses the www-data user for Apache.

Unless you can get more detailed error messages out of Apache it could be something to do with www-data not having access to all the bits of ccl that are needed in order to run your program. This does seem unlikely given that you (presumably as a normal user) can run it from the command line.

Hero Doug wrote:I think that it's highly unlikely that it's a permission problem because I compiled an equivalent C program and ran it without a hitch. I applied the same permissions from the C program to the lisp program and ran it and had the same problem.

There's a difference, though. The C program is more or less self contained. It will most likely depend on the C library, but if there were permission problems with that you would have more problems than just your CGI program. Your lisp program might depend on other bits of CCL that Apache is having trouble finding or does not have access to. (I've never used CCL, so I'm not sure of the details.)

CCL does not appear to be in the Ubuntu repositories, so I suppose you installed it manually, whereas sbcl was installed from the repositories? This could hint at why sbcl works and ccl doesn't.

I hope this helps :-)

Hmm... I responded Friday, but due to 1st post approval issues it is now in the middle of the pack and probably easy to skip.

It might not be an Apache issue. The error message you cite is interesting:

"Couldn't load lisp heap image from "

with it blank after 'from'. This error message is from within CCL's load_image():

http://svn.clozure.com/publicsvn/openmc ... l-kernel.c

Perhaps it's having problems sorting out the internal geography of its own executable. CCL does seem to have some quirks, hence the idea of rebuilding the image file with CCL & the source file in the same directory to see if it builds it slightly differently.

On a probably-not-related note, I encountered a similar sounding situation once where the issue was that the environment the task ran under in Apache was not the same as the environment from the shell prompt. Running a diagnostic dump of the environment under Apache & comparing it with the normal one highlighted the issue, and modifying Apache's environment solved the problem. In particular, I would try making sure that the PATH variable includes your cgi directory.

To be specific, creating some print-env.cgi

#!/bin/sh

echo "Content-type: text/html"
echo ""
echo $PATH

might show something interesting. And under Ubuntu you could try adding something like this to /etc/apache2/envvars

   export PATH=/usr/local/bin:/usr/bin:/bin:/home/test/cgi-bin 

Good luck! You've been admirably persistent.

To check that you have permission issuses or not, you can run ccl under the www-data user:

sudo -u www-data /path/to/index.cgi

To the most recent reply, here are the results of running the app as www-data, it seems as if it's not a permission problem.

doug@doug-desktop:~/software/ccl$ sudo -u www-data /home/doug/dev/trunk/htdocs/index.cgi
[sudo] password for doug:
Content-type: text/html

This is a test

cgsullivan >> The "blank" in the error message you spoke of is filled in if I try to run it "like an interperted script".

#!/home/doug/software/ccl/ccl -I /home/doug/dev/trunk/htdocs/image[/quote]

The error looks like this:
[quote]
Couldn't load lisp heap image from /home/doug/dev/trunk/htdocs/image
[Fri May 08 10:51:21 2009] [error] [client 127.0.0.1] Premature end of script headers: index.cgi
[/quote]

You're indeed right, the two paths are not the same.
[quote]
[i]Terminal[/i]
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

[i]Apache[/i]
/usr/local/bin:/usr/bin:/bin
[/quote]

However, the executable should be self contained since I prepended the kernel, I'm not seeing why it needs a path at all. 

Anyways, I added the path to apache as you suggested and still had no luck.

I'm going to try saving it without prepending the kernel, maybe I'll be able to load the image by passing it to the CCL binary.



Wodin >> I get how the permissions work; running it as Opera was a brainfart indeed. And I downloaded SBCL from the site, not the repo. I've been using it and it works fine.

I shot an email off to the CCL dev list, I'll see if they can add anything to this.

I note that the error message does not appear to contain the .cgi extension.

What happens if you rename your executable to simply 'image', rather than 'image.cgi'? Perhaps CCL is trying to be a bit clever with an extension it doesn't expect.

Ok, I got it working, sort of.

I make an image file and ran it under www-data using a shell script.

#!/bin/sh

/home/doug/software/ccl/ccl -I /home/doug/dev/trunk/htdocs/sitewire.app

I also prepended the image file to the image and ran that as well using the same method.

#!/bin/sh

/home/doug/dev/trunk/htdocs/index.cgi

This seems to have fixed the problem, although I never did find out why I had that other problem. At least i can get to work now.

Now I have the same problems (and more) on my personal computer (I did all that other stuff from my work computer).

I can't save an image with a prepended kernel, and I cant run the image using ANY method.

I've shot an email off the the CCL guys and I'll see what they have to say. I have to say, I'm nearing my wits end.

I received an error when trying to save an image with a prepended kernel not long ago. I sent an Email to the Clozure mailing list and it was resolved surprisingly quickly.

[quote=The Error]
:The Error

> Error: value NIL is not of the expected type REAL.
> While executing: CCL::<-2, in process Initial(0).
> Type :POP to abort, :R for a list of available restarts.
> Type for other options.
1 >
[/quote]

[quote=The Solution]
Supply the full path to the kernel
(save-application "whatever" :prepend-kernel "/full/path/to/kernel")
[/quote]

Funny thing: I'm doing the same right now! CGI isn't very "cool" today, and Lisp isn't supposed to be particularly suited for CGI due to long startup time, and yet there are two persons simultaneously making CGIs with CCL
My Clozure CGI works fine with Apache on Windows XP, but fails miserably with IIS (due to some error in CCL-DIRECTORY or USER-HOMEDIR-PATHNAME).
Debugging it, I've noticed that if I call my CCL cgi from C CGI via EXECL, it fails, and SBCL doesn't. The reason is CCL is trying to load its core from the wrong place:

image_name = argv[1];
argv[1] = NULL;
} else {
process_options(argc,argv);
}
initial_stack_size = ensure_stack_limit(initial_stack_size);
if (image_name == NULL) {
if (check_for_embedded_image(argv[0])) {
image_name = argv[0];
} else {
image_name = default_image_name(argv[0]);

But if I use SYSTEM or POPEN, CCL works. So personally I think it's best to call it with a script - start up time shouldn't matter much, if we use Lisp for CGIs, after all

hrapof wrote:Funny thing: I'm doing the same right now! CGI isn't very "cool" today, and Lisp isn't supposed to be particularly suited for CGI due to long startup time, and yet there are two persons simultaneously making CGIs with CCL

If you save your lisp image, startup time (with no init file, of course) is very short.

hrapof wrote: My Clozure CGI works fine with Apache on Windows XP, but fails miserably with IIS (due to some error in CCL-DIRECTORY or USER-HOMEDIR-PATHNAME).

Some changes to CCL were checked into the trunk several weeks ago to try to address this.

If you're willing, please review http://trac.clozure.com/openmcl/ticket/517 and see if a current trunk still has problems.

I'm very late to post, so this is probably all over, but I've seen the "Premature end of script headers" error before programming cgi in clisp, usually because there's a problem somewhere in the #! line (wrong path to lisp, wrong init options, or any other error). Hope this helps...