The library situation (warning: mini-rant)

Hey,

I decided to jump in and play with the libraries available in the CL community (i.e. ASDF-installable) to try to stop my procrastination in using CL. So I upgraded to the latest SBCL build for OS X and started looking for GUI libraries; something I personally expect from a programming language/setup before I feel it's practically usable, even though I don't write much GUI software.

Doing so was a bit of a nightmare, at least as a Lisp-newbie. It took maybe 4 hours of effort spread over two days before I found one that would nearly build (lispbuilder-sdl) and was able to coax into working with a couple of hacks. Right after that I tried to asdf-install lispbuilder-sdl-ttf (for playing with fonts) and had to modify its Makefile for OS X and specify a library location in absolute terms.
Before those, I tried and failed to install quite a few libraries - cells, cello, mcclim, poly-pen and a few others.

The ones that didn't turn up a 404 or other basic error with asdf-install required hacking to get them to build, so not one of them worked out of the box. This is pretty discouraging - how do you distribute programs when the users would have to modify Makefiles and asd files etc just to even get your code to build?

Am I just unlucky, and some servers are down (a lot of stuff on cliki gave 404s via asdf-install, and the web-cvs interfaces on common-lisp.net are broken), or is this just a result of half-baked libraries/installation scripts (understandable since not many package maintainers have Macs), or is there something more fundamentally wrong with CL?

The main reason is, probably, some bitrot and lack of maintenance. With GUI, I had similar experience on Linux (except that some libraries actually seemed to work).
But other libraries worked quite fine (maybe because I used Gentoo Linux's package repository, in which lisp-overlay is quite well maintained).

Perhaps it is more of an issue with OS X; McCLIM builds for me under Linux, same with most of the ASDF-Install libraries I have, though sometimes in building I get an error and choosing the retry installation restart works.

Additionally, for distributing software to end users, if you are sending them a lisp image, it doesn't really matter how you got it to build.

Exolon wrote:I decided to jump in and play with the libraries available in the CL community (i.e. ASDF-installable) to try to stop my procrastination in using CL. So I upgraded to the latest SBCL build for OS X and started looking for GUI libraries; something I personally expect from a programming language/setup before I feel it's practically usable, even though I don't write much GUI software.

Then why do you need GUI libraries?

Exolon wrote:Doing so was a bit of a nightmare, at least as a Lisp-newbie. It took maybe 4 hours of effort spread over two days before I found one that would nearly build (lispbuilder-sdl) and was able to coax into working with a couple of hacks. Right after that I tried to asdf-install lispbuilder-sdl-ttf (for playing with fonts) and had to modify its Makefile for OS X and specify a library location in absolute terms.
Before those, I tried and failed to install quite a few libraries - cells, cello, mcclim, poly-pen and a few others.

SDL isn't really a GUI library, is it?

I don't generally mess with asdf-install. Nor have I had much trouble getting things to run. But that's possibly a result of good OS choices on my part. I hear LTK is a good mature library, but I haven't done any any Lisp GUI programming beyond a little fooling with McCLIM.

Exolon wrote:The ones that didn't turn up a 404 or other basic error with asdf-install required hacking to get them to build, so not one of them worked out of the box. This is pretty discouraging - how do you distribute programs when the users would have to modify Makefiles and asd files etc just to even get your code to build?

Distribute working library versions? Yes, this is an inconvenience for developers, but I don't see what it has to do with distribution.

Exolon wrote:Am I just unlucky, and some servers are down (a lot of stuff on cliki gave 404s via asdf-install, and the web-cvs interfaces on common-lisp.net are broken), or is this just a result of half-baked libraries/installation scripts (understandable since not many package maintainers have Macs),

A little of both. Half-baked (as you put it) libraries and installers seem to be endemic to small communities of programmers—you end up with crucial fixes floating around in patches spread by word of mouth rather than incorporated into the main project. Considering a lot of these projects are one-man efforts, or are even between developers, I can't complain too much. Until the proportion of Lispers who want to write libraries for public consumption (as opposed to Lispers who are focused on non-public projects) rises, I don't think we can expect much different.

Exolon wrote: or is there something more fundamentally wrong with CL?

Now, what kind of question is that?

The same situation applys for me.

I must admit while CL is a mysteriously fascinating language, but when it comes to library, CL is probably several steps behind other languages, like Python or Perl. It's easier to find cross platform libraries (I use Linux and Windows) for those two languages than for CL.

I think this nature is due to CL is "a specification", not "an implementation".

CMIIW

Exolon,

The library situation is being increasingly acknowledged as somewhere between "painful" and "abysmal." People like Edi Weitz have made fantastic contributions, but there are still some significant barriers to entry for all but the determined.
I'm mostly keeping my head down until I have something concrete to contribute myself, but the reassuring news is that there are a couple of efforts underway to fix it. This doesn't change the frustrations you've had so far, of course.

OTOH, I can honestly say that most of the libraries I've used have compiled well enough, though I'm using Linux rather than MacOS. That sounds a little defensive until you consider that I'll say the same of Perl.

I'm a little confused about one thing, though: why the importance of GUI libraries? I'm genuinely curious, though possibly showing my bias towards using the browser for such things. Me, I use Hunchentoot, CL-WHO and Postmodern - you may want to try those, before leaping into GUI work. Alternatively, look at Lispworks, which I gather has a well-polished GUI API.

It's definitely not a fundamental issue with the language. If it's an endemic issue with anything, it's probably the DIY culture that's still pretty strong: people are generally expected to be inclined to hacking stuff into a working state, which is not quite as tenable a position as it used to be.

Are you trying to accomplish something in particular, or "just" learn the language? I ask because we could be able to save you some time and frustration by pointing you to some known-good libraries.


I disagree with anta40; it's not because it's a spec with multiple implementations, it's just that either people haven't gotten around to writing some of the libraries that are taken for granted in other language, or they've been written (probably several times) and not shared. One of the hazards of a language this powerful is that its users often find that it's easier to re-invent a wheel than go shopping for one. The other part of this problem is that it's hard work to write a polished library - nobody wants to share (or use) a library that's half-written and undocumented, and the motivation for maintaining it generally goes away when the job is complete.
There's also the note that not all libraries actually _need_ writing - it's often possible to "virtually" implement a library by writing a wrapper layer over a library that does something similar, which is only obvious after you've already gotten the hang of the language.

JamesF wrote: I disagree with anta40; it's not because it's a spec with multiple implementations, it's just that either people haven't gotten around to writing some of the libraries that are taken for granted in other language, or they've been written (probably several times) and not shared.

Thanks for your input.

JamesF wrote:The other part of this problem is that it's hard work to write a polished library - nobody wants to share (or use) a library that's half-written and undocumented, and the motivation for maintaining it generally goes away when the job is complete.

I just recently learn CL. And I think I can describe the situation like this :
this library works on this CL on this OS, and that library works on that CL on that OS.
So if i want to write multiplatform code, then I need to write some kind of wrapper, which is at the moment i'm not capable of.

Probably the (temporal) solution would be stick to one OS for a moment and forget the multiplatform issue for a while.
Not a big problem for me, I'm still happy with CL

anta40 wrote:I think this nature is due to CL is "a specification", not "an implementation".

Yeah, I'm sure people working on Lisp software and libraries are thinking the same thing; they are really worried about "what 'The Spec' might think".

Nah. There is no problem like that; there are no road-blocks. The real problem, question and solution is this; "what am I doing, or what can I do?".

If you're unhappy with the GUI-stuff that's out there then do something about it instead of expecting others to do it for you. It's that simple; that's the reality, even if this means actually having to learn Lisp first to do it. It's not harder to do in Lisp than in other languages; you can write from scratch and you have things like CFFI which is great, or you can patch or fork something already out there. It's basically what you're doing already -- as Java or Python or whatever type-developers, right? You're the reason Python has PyGtk+, no?

..or, well, maybe; how much are you guys willing to pay for an ASDF-installable set of bindings to something portable like, say, Gtk+? .. no, but, really; can you compete with what people are paying for web-UIs?

anta40 wrote:I just recently learn CL. And I think I can describe the situation like this :
this library works on this CL on this OS, and that library works on that CL on that OS.
So if i want to write multiplatform code, then I need to write some kind of wrapper, which at the moment i'm not capable of.

That sounds right: making the code portable across implementations and OSes is even more work.

One thing to note: you normally don't need a wrapper to get things to work across platforms; the #+ and #- operators are very useful for telling the compiler to "use this bit of code for Lispworks and SBCL, that bit for Clisp, and this other bit for everything else". This will be useful for understanding the way some code is written, and could be useful in the future if you decide to try making something portable yourself.

anta40 wrote:Probably the (temporal) solution would be stick to one OS for a moment and forget the multiplatform issue for a while.

That sounds like a good plan while you get the hang of it.

dmitry_vk wrote:But other libraries worked quite fine (maybe because I used Gentoo Linux's package repository, in which lisp-overlay is quite well maintained).

Yes, I have had much more luck using various Linux package managers than ASDF. ASDF is pretty bad. I second the recommendation of Gentoo, it's a good distro for managing Common Lisp.

JamesF wrote:The library situation is being increasingly acknowledged as somewhere between "painful" and "abysmal." People like Edi Weitz have made fantastic contributions, but there are still some significant barriers to entry for all but the determined.

Yes, that about sums it up.

Exolon wrote:...with the libraries available in the CL community (i.e. ASDF-installable)...

FIrst, there isn't such a thing as "CL community", which might be a part of the problem... there are several scattered weakly connected communities centred around various communication channels, ie. people here, on freenode #lisp and comp.lang.lisp, to name a few, are usually quite different.

In any case, I have noticed that some time ago many of the library maintainers stopped making releases, which are sort of necessary for ASDF-installability, and just started publishing their source repositories. I think this sort of thing started with Slime, which pretty much required installing CVS head for years...

Anyway, this means that for quite many libraries the version which ASDF-install sees might be old, or, as you have noticed, no longer in existence. Right know I would suggest using clbuild, which automatically tracks many repositories. Of course it requires something like four different version control systems.

Hi guys,

Thanks for your insightful and helpful responses! A few comments on your comments.

qbg wrote:Additionally, for distributing software to end users, if you are sending them a lisp image, it doesn't really matter how you got it to build.

This is great news - I wasn't sure if this would work or not. If I can build a working, distributable image that can be used on machines that don't even have a Lisp installed, then that improves things massively.

Paul Donnelly (and JamesF) wrote:Then why do you need GUI libraries?

• Back when I was finding my love of programming in 1994 or so with GFA Basic on the Atari ST, even with the fairly sparse API it was easy (and gave immediate results and feedback) to do simple drawing, plotting and a bit of animation. This added an extra level of attractiveness to programming which for me was a good thing. So I like to play with GUI code to get a feel for a new language without taking on heavy duty or mundane programming tasks.
• After working professionally with J2EE/Spring/Hibernate apps and all the headaches and verbose pain involved (creating a new 'page' meant writing 1: controller unit test 2: controller 3: model unit test 4: model 5: new entries in servlets.xml 6: new jsp 7: new entry in views.xml 8: a HttpUnit UAT test - even without the unit/UAT tests this was a lot of donkey work) I've been kind of turned off writing webapps for a while - although the little I've seen of CL libraries for these purposes seems much, much less verbose (which is pretty easy, let's be honest) than Java.
• I'd like to create some colourful, impressive, responsive toy programs for my 10 month old daughter, rather than just letting her bash the keyboard in TextEdit
• And maybe write a game or two.

Paul Donnelly wrote:SDL isn't really a GUI library, is it?

It is, but perhaps not strictly in the sense you mean. It provides API for graphics, sound and input which is very sexy and worth fixing to 'just work'. However I would like to try a conventional GUI library as well - Cells/Cells-gtk/Cello looked promising but didn't work with asdf-install. With perhaps a few days/weeks of experimentation I'll install it manually and try it out.

lnostdal wrote:If you're unhappy with the GUI-stuff that's out there then do something about it instead of expecting others to do it for you. It's that simple; that's the reality, even if this means actually having to learn Lisp first to do it. It's not harder to do in Lisp than in other languages; you can write from scratch and you have things like CFFI which is great, or you can patch or fork something already out there. It's basically what you're doing already -- as Java or Python or whatever type-developers, right? You're the reason Python has PyGtk+, no?

I'm not sure if you're referring to me here, but in any case, I certainly intend to work on these issues and try to improve the situation. On day 0, I submitted a patch to the lispbuilder-sdl mailing list (more as documentation of something that helped the build work for me than a suggestion that they include the patch - for all I know it breaks things for all other OS X users).

However, you must admit that such large learning curves and obstacles to newbie Lispers like myself will drive some people (who may have been capable of great contributions had things been a little easier) away.
That's why I wanted to ask you guys how your experiences compared to mine in this area.
(Incidentally, some asdf-install packages hosted on your site fail because it seems your directory layout has changed.)

On the subject of asdf-install and perhaps asdf itself, I understand why a lot of people are saying "it sucks, just do it manually", but the fact is we need something like that - consider how much apt-get/dpkg from Debian have contributed to the popularity of Linux distributions like Ubuntu. Things like this and having working sexy libraries like sdl/cells/etc out-of-the-box would be a boon to Common Lisp and get more programmers using this powerful language, I think. I will try to improve things as best I can, but I'm no expert, and most definitely a newbie when it comes to Common Lisp itself.

Oisín

Shipping a single image may not always work. FFI might require presence of shared object files.

Exolon wrote:

lnostdal wrote:If you're unhappy with the GUI-stuff that's out there then do something about it instead of expecting others to do it for you. It's that simple; that's the reality, even if this means actually having to learn Lisp first to do it. It's not harder to do in Lisp than in other languages; you can write from scratch and you have things like CFFI which is great, or you can patch or fork something already out there. It's basically what you're doing already -- as Java or Python or whatever type-developers, right? You're the reason Python has PyGtk+, no?

I'm not sure if you're referring to me here,

..everyone, I guess.

Exolon wrote: but in any case, I certainly intend to work on these issues and try to improve the situation. On day 0, I submitted a patch to the lispbuilder-sdl mailing list

..exxxxellent; so there is hope after all!

Exolon wrote: However, you must admit that such large learning curves and obstacles to newbie Lispers like myself will drive some people (who may have been capable of great contributions had things been a little easier) away.

Yeah, maybe - I don't know. I think the language "at the core" or out-of-the-box has enough of interest to keep one going through the initial hurdles wrt. "practical obstacles" etc..

(defclass button (widget)
  ())

(defmethod render ((button button))
  (format t "TODO: replace this with code that renders a button at [~A, ~A] later .. apply my own imagination while learning lisp for now~%" (x-pos-of button) (y-pos-of button)))

"..hm, yeah, OOP -- cool".

Exolon wrote: That's why I wanted to ask you guys how your experiences compared to mine in this area.
(Incidentally, some asdf-install packages hosted on your site fail because it seems your directory layout has changed.)

Do you mean SymbolicWeb? It's not ASDF-installable; never was. I'm using a .sh-script to move it around to other servers quickly. I guess I could clean it up and publish that as a part of the source and link to it from the front page .. but, uh, .. later.

It's a plain ordinary chicken-and-egg problem: without a large user/developer community, there aren't enough people writing and maintaining libraries; without lots of good libraries, we can't attract the users. It's as simple as that.

I think we need to be a little careful when we tell newcomers "if you don't like it, fix it yourself". Yes, that is the reality of the situation. But I think we also need to acknowledge that the expectations these people came in with were not unreasonable; the current state of CL just doesn't happen to be able to meet them. And these newcomers may or may not have the resources to invest in overcoming the difficulties they encounter. Most don't, I would guess, and that doesn't make them wimps; it just means they have other priorities.

I certainly would like to encourage everyone releasing libraries for CL to take the time to get your code working on multiple implementations and to make it ASDF-installable. (I have done this with my FSet library (http://common-lisp.net/project/fset/.) If one is going to the trouble of releasing something at all, seems to me, it's worth the extra effort to make it usable for as many people as possible. But, I understand, you have your own priorities too.

Exolon wrote:I like to play with GUI code to get a feel for a new language without taking on heavy duty or mundane programming tasks.

Ah. You may want to try something else, while getting the hang of CL, as there doesn't seem to be much well-polished GUI-related stuff, unless you go for Lispworks. It's not that it doesn't work, I just get the distinct impression that you want to have already climbed the main learning curve of CL before adding those to your load.

Exolon wrote:After working professionally with J2EE/Spring/Hibernate apps and all the headaches and verbose pain involved (creating a new 'page' meant writing 1: controller unit test 2: controller 3: model unit test 4: model 5: new entries in servlets.xml 6: new jsp 7: new entry in views.xml 8: a HttpUnit UAT test - even without the unit/UAT tests this was a lot of donkey work) I've been kind of turned off writing webapps for a while - although the little I've seen of CL libraries for these purposes seems much, much less verbose (which is pretty easy, let's be honest) than Java.

Ah, whereas my day-job is customer support in the midst of a Java shop. While I admittedly don't work directly with the code myself, I spend enough time reading the source-code to wonder why the hell they have to make everything so incredibly complicated.

Do give it a try; chances are you'll find it strikingly simpler, at least for relatively trivial stuff. I qualify that because non-trivial work requires a well-thought-out model, which involves work no matter what language you're implementing in.

If you go for Hunchentoot and CL-WHO, I'll mention two gotchas: first, make sure you really do have the latest versions of the dependencies, to save yourself wondering why the frikkin' page won't display. Second, CL-WHO wants (format nil ...) sometimes, and (format t ...) at other times, and I've yet to spend the time to figure out the reason.

Exolon wrote:However, you must admit that such large learning curves and obstacles to newbie Lispers like myself will drive some people (who may have been capable of great contributions had things been a little easier) away.
That's why I wanted to ask you guys how your experiences compared to mine in this area.

I totally agree. I just finished a large chunk of refactoring in my pet library this morning, so there's hope it'll be released in a month or so. If I've gotten it right, it'll make web programming just a bit simpler again, by providing the utilities that you'd have had to write anyway. I even comment my code!

Exolon wrote:On the subject of asdf-install and perhaps asdf itself, I understand why a lot of people are saying "it sucks, just do it manually", but the fact is we need something like that - consider how much apt-get/dpkg from Debian have contributed to the popularity of Linux distributions like Ubuntu. Things like this and having working sexy libraries like sdl/cells/etc out-of-the-box would be a boon to Common Lisp and get more programmers using this powerful language, I think. I will try to improve things as best I can, but I'm no expert, and most definitely a newbie when it comes to Common Lisp itself.

My next trick is to see if I can improve on asdf/asdf-install. Version constraints are my particular bug-bear, because I'm sick of maintaining multiple different repositories on my development machine.

sburson wrote:It's a plain ordinary chicken-and-egg problem: without a large user/developer community, there aren't enough people writing and maintaining libraries; without lots of good libraries, we can't attract the users. It's as simple as that.

I think we need to be a little careful when we tell newcomers "if you don't like it, fix it yourself". Yes, that is the reality of the situation. But I think we also need to acknowledge that the expectations these people came in with were not unreasonable; the current state of CL just doesn't happen to be able to meet them. And these newcomers may or may not have the resources to invest in overcoming the difficulties they encounter. Most don't, I would guess, and that doesn't make them wimps; it just means they have other priorities.

From the perspective of a small commercial developer, I agree completely. I am certainly a Lisp newb, but am looking at it for future projects because it looks to be a beautiful and productive language as well as having the potential for the performance that we need. I would be perfectly happy contributing useful pieces of code or libraries that we develop as part of a product, but we don't have the resources to put together something like a GUI toolkit of our own.

August wrote:From the perspective of a small commercial developer, I agree completely. I am certainly a Lisp newb, but am looking at it for future projects because it looks to be a beautiful and productive language as well as having the potential for the performance that we need. I would be perfectly happy contributing useful pieces of code or libraries that we develop as part of a product, but we don't have the resources to put together something like a GUI toolkit of our own.

If you're willing to pay for a GUI toolkit, there's one from Lispworks (http://www.lispworks.com/products/capi.html) and another from Franz (http://www.franz.com/products/allegrocl ... ools.lhtml). While the open-source toolkits might have some way to go before they're newb-friendly, it's not quite true that there's nothing available. I can't comment on what they're like to work with, because I don't do GUI programming outside of a web browser, so SBCL does the job just fine for me.

Sorry about the clumsy linking, but I'm not sure how to hyperlink things on these forums. Give me plain HTML any day

JamesF wrote:If you're willing to pay for a GUI toolkit, there's one from Lispworks (http://www.lispworks.com/products/capi.html) and another from Franz (http://www.franz.com/products/allegrocl ... ools.lhtml). While the open-source toolkits might have some way to go before they're newb-friendly, it's not quite true that there's nothing available. I can't comment on what they're like to work with, because I don't do GUI programming outside of a web browser, so SBCL does the job just fine for me.

I guess I was speaking more in the spirit of "how do we get more people to start using Lisp". In the current Lisp implementation world, we would likely purchase some Lispworks licenses as it seems to solve most of our issues (except native threads simultaneously running Lisp). I didn't mean to imply that I thought there was nothing available. I did a reasonable amount of research before starting my slog to enlightenment via SICP and PCL

August wrote:I was speaking more in the spirit of "how do we get more people to start using Lisp".

Ah. In that case, it looks like we'd get most mileage out of a more complete (in both senses) set of libraries to address things that have become much more common in the last 10 years or so, and making it easier to install and use them. Mostly, I suspect, polish (which is the other 90% of the development work).

At least I now have an answer to the "I don't like parentheses, and I want my C syntax!" crowd: Javascript. It's turning out to be a nice stepping-stone to Lisp, once you figure out the recursion, anonymous functions and closures. I think it'll serve as much as a stepping-stone to C for me, but maybe that just reinforces its versatility... or something.

JamesF wrote:Ah. In that case, it looks like we'd get most mileage out of a more complete (in both senses) set of libraries to address things that have become much more common in the last 10 years or so, and making it easier to install and use them. Mostly, I suspect, polish (which is the other 90% of the development work).

Whether or not you like the language, I think Python does a good job in this respect.

August wrote:

JamesF wrote:Ah. In that case, it looks like we'd get most mileage out of a more complete (in both senses) set of libraries to address things that have become much more common in the last 10 years or so, and making it easier to install and use them. Mostly, I suspect, polish (which is the other 90% of the development work).

Whether or not you like the language, I think Python does a good job in this respect.

Wanna re-implement lisp in python? Then we get the libraries, and the axiom that lisp is slow becomes true again

August wrote:

JamesF wrote:Ah. In that case, it looks like we'd get most mileage out of a more complete (in both senses) set of libraries to address things that have become much more common in the last 10 years or so, and making it easier to install and use them. Mostly, I suspect, polish (which is the other 90% of the development work).

Whether or not you like the language, I think Python does a good job in this respect.

I get that impression too; it seems to be a useful reference point.

TheGZeus wrote:

August wrote:

JamesF wrote:Ah. In that case, it looks like we'd get most mileage out of a more complete (in both senses) set of libraries to address things that have become much more common in the last 10 years or so, and making it easier to install and use them. Mostly, I suspect, polish (which is the other 90% of the development work).

Whether or not you like the language, I think Python does a good job in this respect.

Wanna re-implement lisp in python? Then we get the libraries, and the axiom that lisp is slow becomes true again

I'm pretty sure it's been done the other way around, but this looks more like what you're talking about: http://common-lisp.net/project/python-on-lisp/

I'm pretty sure it's been done the other way around, but this looks more like what you're talking about: http://common-lisp.net/project/python-on-lisp/

Now, I have never used it, but python-on-lisp calls out to Python. It is like an FFI for Python (like I said, I haven't really looked at how it works, but I get the idea that it is similar to how LTk communicates with wish).

I have played around with CLPython, http://common-lisp.net/project/clpython/, which is an project to write an implementation of Python in CL, which would mean that you could compile Python to machine instructions using the your favorite implementation. So maybe "Python is slow" could become a myth. Definitely not ready for prime time though, since many of the Python standard libraries are written in C and are inaccessible (until they are ported).

Zach S