Trying to figure out ASDF

I have been learning CL for several months now, loving it, and getting along without asdf or equivalent. I find the asdf documentation (from the asdf project page on commonlisp.net), ahem, horrible. (Most of the other material I have, from e.g. Peter Seibel, Paul Graham, etc. is great, so I don't think I'm a whiner about this. Maybe I am.)

If anyone knows of a good *complete* asdf tutorial point me to it.

I find this passage from what I take to be the official asdf manual quite confusing:

The files are located in the same directory as the file with the system definition. ASDF resolves symbolic links before loading the system definition file and stores its location in the resulting system 3 . This is a good thing because the user can move the system sources without having to edit the system definition.

First, what does "the file with the system definition" mean? Does it mean "the file containing the system definition" or "accompanied by" something or other? "With" is a weird choice of prepositions. And what symbolic links is he talking about? Symbolic links to what? I totally don't get it. And "stores its location" - WTF is "its"? This is a totally appalling mess of a sentence.

I'm sure it will all seem perfectly clear after I have figured it out.

BTW, those are actual questions - not just complaints about the sentence. Help me out if you can.

On a posix system (anything but mswin), you can make symlinks to system1.asd, system2.asd, ... in one central directory (e.g. ~/.sbcl/systems). Then add this directory to asdf:*central-registry*. Due to the way ASDF resolves symlinks, it will look for source files relative to the location of the real system.asd file, not relative to the symlink.

The system definition file is normally a file with extension .asd that accompanies the library which uses it. It contains a system definition which tells asdf what are the source files, in which order they have to be compiled / loaded, how can one test it and so on.

Now, when you want to install a library, you need to tell ASDF where it is so it can load the library to you. You do that by inserting in the variable asdf:*central-registry* the path to the directory where the file .asd is located. But, instead of having to insert a new directory to this variable every time you want to use a new library, you just create a symbolic link in some directory.

For instance, suppose you downloaded a library named foo and installed it in the directory "/home/me/Lisp/libs/foo/". So there should be a file named "foo.asd" in that directory. One way to acknowledge asdf of that library is inserting

(push #p"/home/me/Lisp/libs/foo/" asdf:*central-registry*)

into you Lisp init file (it is .sbclrc for SBCL, .clisprc for Clisp...). But that you would have to do with every library you install.

A simpler alternative would be to choose a directory, e.g. "/home/me/Lisp/systems/" and create a symbolic link there pointing to "/home/me/Lisp/libs/foo/foo.asd". Finally you tell asdf about that:

(push #p"/home/me/Lisp/systems/" asdf:*central-registry*)

and you are done. Every time you download a new lib, you symlink the .asd file there

You can try using asdf-install, which downloads and installs libraries from Cliki automatically. SBCL already comes with it, you just need to

(require :asdf-install)

In other systems, you will have to install it.

By the way, asdf and asdf-install are more or less a pain to work in Windows.

Ok, thanks. Those replies help. I got the symbolic link thing working.

I don't know if I should post additional questions on this thread, or a new one. I guess I'll do it here.

After the initial simple example, the ASDF manual goes on like this:

A more involved example

Let's illustrate some more involved uses of defsystem via a slightly convoluted example:

(defsystem "foo"  
  :version "1.0"  
  :components  
  ((:module "foo"  
    :components ((:file "bar")  
                 (:file "baz")  
                 (:file "quux"))  
    :perform (compile-op :after (op c) (do-something c))  
    :explain (compile-op :after (op c) (explain-something c)))  
   (:file "blah"))) 
The :perform tokens need explaining: essentially, these clauses:

:perform (compile-op :after (op c)  
           (do-something c))  
:explain (compile-op :after (op c)  
           (explain-something c)) 
have the effect of

 (defmethod perform :after ((op compile-op) (c (eql ...)))  
      (do-something c))  
 
 (defmethod explain :after ((op compile-op) (c (eql ...)))  
      (explain-something c)) 
where ... is the component in question.

Ok, this is totally opaque to me. He says that "the :perform tokens need explaining" but I don't see an actual explanation in what follows. The "tokens" apparently translate into an invocation of the defmethod macro. But what in the world does that mean in terms of its intended effect? When does the method get invoked, and what might it do when invoked? Some indication of what the "token" is trying to accomplish would have constituted an actual explanation.

I don't think I have seen the explain thing ever used. Most of those more complex options of ASDF are quite rarely used. It is one of those things that if you cannot figure out from the source (ASDF is a single file of mostly portable Common Lisp), then you don't need.

In general one can hope that an operation is performed at point indicated by its name, that is, compile-op when compiling and load-op when loading, on all defined components, in order determined by the dependency network.

Hi. Sorry for maybe bringing up an old topic, but it seemed like the proper place to me. Pleas, feel free to move this post if you feel like.
I'm trying to figure out how to get several files compiled. I've followed several tutorials, but none worked (let alone following some illusionary good practices and avoiding deprecated API .
So, here I am, the code should explain my struggle, but if it doesn't, the explanation follows:

(defpackage video-rounds-system
       (:use :common-lisp :asdf))
(in-package :video-rounds-system)

(defsystem "video-rounds"
	:description "rounds-web: a testing environment for Rounds project."
	:version "0.0.1"
	:author "wvxvw"
	:components ((:file "./rounds-web/packages.lisp") ;; defines (defpackage :rounds-web)
		(:file "./rounds-web/server.lisp" :depends-on ("./rounds-web/packages.lisp"))
		(:file "./rounds-web/serialization.lisp" :depends-on ("./rounds-web/packages.lisp"))
		(:file "./rounds-web/error-handlers.lisp" :depends-on ("./rounds-web/packages.lisp"))
		(:file "./rounds-web/success-handlers.lisp" :depends-on ("./rounds-web/packages.lisp"))
		(:file "./rounds-web/services.lisp" :depends-on ("./rounds-web/packages.lisp"))
		(:file "./rounds-web/snapshot.lisp" :depends-on ("./rounds-web/packages.lisp"))))
(asdf:oos :video-rounds)
(require :rounds-web)
(rounds-web:start-server)

The error:

* (compile-file "start-server.lisp" :output-file "start-server")

; compiling file "/home/wvxvw/Projects/rounds/rounds-net/test-web-server/start-server.lisp" (written 02 MAY 2011 01:11:18 PM):
; compiling (DEFPACKAGE VIDEO-ROUNDS-SYSTEM ...)
; compiling (IN-PACKAGE :VIDEO-ROUNDS-SYSTEM)
; compiling (DEFSYSTEM "video-rounds" ...)
; compiling (OOS :VIDEO-ROUNDS)
; compiling (REQUIRE :ROUNDS-WEB); 
                                      ; compilation unit aborted
                                      ;   caught 1 fatal ERROR condition
; compilation aborted because of fatal error:
;   READ failure in COMPILE-FILE:
;     SB-INT:SIMPLE-READER-PACKAGE-ERROR at 901 (line 18, column 24) on #<SB-SYS:FD-STREAM for "file /home/wvxvw/Projects/rounds/rounds-net/test-web-server/start-server.lisp" {B0D9419}>:
;       package "ROUNDS-WEB" not found
; compilation aborted after 0:00:00.007
NIL
T
T
* 

Just to make sure I'm sane: this is the ./rounds-web/packages.lisp file

(require :hunchentoot)
(defpackage :rounds-web
	(:use :cl :hunchentoot)
	(:export :start-server))

I have the file that defines a "system", which as far as my understanding permits means to tell SBCL to load specified files and compile them. After that I'm trying to require a package defined in those files, and whoops, no package... How do I even know what (defsystem) did? Am I at all understanding right what it was meant to do? I tried (operate-on-sytem) instead of (asdf:oos) with same results.
I've tried different ways of defpackage (the quote form and just a string, which I don't really understand the difference in the given context to no avail).

My Lisp is SBCL. This is probably not the newest version, but I would be thankful if the answer didn't involve updating and using some more libraries. I know I will have to update, but I'm just not able to do it now . If you may advise any reading on how to compile multiple files in the simplest way (I'm not doing anything complex!) that would help too.

Thanks in advance.

EDIT: For the future reference, I discovered that probably where it says :file, it doesn't really mean file, instead it only wants the file name (please correct me if I'm wrong, I'd really like to be wrong on that!) and the file must be in the same folder as *asd where you defined the system.
I finally got it working. But please tell me if I did it wrong! In this way:

(in-package :cl-user)
(require :hunchentoot)
(require :cxml)
(require :cl-ppcre)
(require :cl-gd)
(defpackage video-rounds-system
       (:use :common-lisp :cl :asdf :hunchentoot :cxml :cl-ppcre :cl-gd))
(in-package :video-rounds-system)

(asdf:defsystem #:video-rounds
	:description "rounds-web: a testing environment for Rounds project."
	:version "0.0.1"
	:author "wvxvw"
	:components ((:file "packages") ;; defines (defpackage #:rounds-web)
		(:file "server" :depends-on ("packages"))
		(:file "serialization" :depends-on ("packages"))
		(:file "error-handlers" :depends-on ("packages"))
		(:file "success-handlers" :depends-on ("packages"))
		(:file "services" :depends-on ("packages"))
		(:file "snapshot" :depends-on ("packages"))))

Moved inside the ./rounds-web directory.

(asdf:oos 'asdf:load-op :video-rounds)
(rounds-web:start-server)

also moved inside ./rounds-web .
I would be very happy to know how would I go about putting files in my project into different directories (there aren't too many of them, but I just like it to be in some order )

The ".asd" file is loaded (that is, executed) by ASDF when scanning for systems and it should optimally not include anything other than (asdf:defsystem ...) form. The exception is when you are extending ASDF functionality in some way, but you shouldn't need that most of the time. The defpackage defining -system package is unnecessary, since ASDF loads the file in temporary package anyway. It appears in some older definitions, since it wasn't true in the past. But in any case if you have only the defsystem form composed from keywords and strings only it doesn't matter anyway, since no part of it depends on the package system, as long as the operator is given fully qualified form.

(asdf:oos :video-rounds)
(require :rounds-web)
(rounds-web:start-server)

This lines are all not supposed to be placed in an asd file.

Furthermore, the first one is broken, since asdf:oos require operation specifier, typically 'asdf:load-op, and the preferred form in modern ASDF2 (installed by default by Quicklisp) is (asdf:load-system :system-name). Or for that matter (ql:quickload :system-name) if you are using Quicklisp, which you should.

The second line is mostly deprecated. REQUIRE is poorly specified. In SBCL it hooks into ASDF anyway. In general it shouldn't be used except as a shortcut in the REPL. It would also be redundant with the line above if it worked. Normally you shoudn't explicitly load systems from source files explicitly at all. The asd should specify dependencies on other systems.

The third line is wrong because you really should not start a server as a side effect of scanning system definitions. And it won't work anyway since neither REQUIRE nor ASDF:OOS have compile time effects, that is, the system wouldn't be loaded even if they were correct until the file was loaded, and it cannot be compiled because at compilation time the package doesn't exist.

Your packages.lisp file should contain REQUIRE, or any other file/system loading form, either. File and system dependencies are supposed to be specified by the asd file.

In asd file component names are names, from which pathnames are derived. It is possible to specify pathnames explicitly, but it is easier to just use something like (although I am fairly certain that you need more complex dependencies):

(asdf:defsystem "video-rounds"
  :description "rounds-web: a testing environment for Rounds project."
  :version "0.0.1"
  :author "wvxvw"
  :depends-on (:hunchentoot)
  :components (:module "rounds-web"
                       :components
                       ((:file "packages") ;; defines (defpackage :rounds-web)
                        (:file "server" :depends-on ("packages"))
                        (:file "serialization" :depends-on ("packages"))
                        (:file "error-handlers" :depends-on ("packages"))
                        (:file "success-handlers" :depends-on ("packages"))
                        (:file "services" :depends-on ("packages"))
                        (:file "snapshot" :depends-on ("packages")))))

I recommend against using ql:quickload, except when intentionally installing packages. It wraps asdf:load-system in a call that will optionally download some version of the package from Quicklisp if it is not found on your system. That may be proper behavior some times, but often it is not. I believe it is likely to result in frankenbuilds with subtle inconsistencies. YMMV.

Oh, thank you! Yes, I sort of figured it out that asd file has to only have the definition and maybe technical stuff related to locating the sources. Shame that's not what the docs say... I think documentation at times assumes too much about the reader
I'll know now to stay away from require too
One question though. I'm not sure if it worked by chance, or was intended. Once I called asdf:oos in the directory where *.asd file lives, it seems to find the system definition and load it. May I conclude that placing *.asd file in the directory with another file that calls asdf:oos is enough / is the right way to do it if that's just my code I'm working on?

ASDF will scan some number of directories looking for asd files. Those directories are specified by, in old ASDF special variable *central-registry* and in ASDF2 by more complicated source registry mechanism. They will include the current directory, where current directory is defined by the *default-pathname-defaults*. At least in ASDF2 case it will be the value of that variable at the time source registry was initialized. It will not be the directory which contains the currently loaded file (unless those are the same). If you have many interdependent systems of your own you should add them to source registry.

Aha, cool, thank you.

I'm also having a problem. I have created a directory called "Center" in my main Clisp folder.
Then:

(require 'asdf)
(push #p"\\Program Files\\clisp-2.49\\Center\\" asdf:*central-registry*)

And then I am pretty much lost
I have downloaded langutils for example, and now I cannot figure out how to load it
Do I have to do something with those .asd files?
Thanks.

P.s: I am running Windows Vista

You seem to be using Windows. Since Windows cannot create symbolic links, you will need to push the directory of each library into asdf:*central-registry*. For instance, if you extracted langutils into the directory called "Center", you should do this:

(require 'asdf)
(push #p"\\Program Files\\clisp-2.49\\Center\\langutils\\" asdf:*central-registry*)

This is not very user-friendly, I know. You should consider using Quicklisp or Lisp Starter Pack, they both run very neatly from Windows, unlike ASDF or clbuild which were designed for Linux and Mac users.

Thanks.
Windows has so many walls

Are you using asdf2? Quicklisp is great for replacing asdf-install but asdf is it's own thing.

With asdf2 it's very simple to set up a project directory that will automatically push your projects to the asdf central registry. Then they're easy to load from Quicklisp.

This is what I do on WIndows and it works great:

1. From your home directory create the file .config/common-lisp/source-registry.conf.d/asd-link-farm.conf -- you can call that file whatever you want as long as it has the conf type. So for example, my link-farm is at "C:\Documents and Settings\george\.config\common-lisp\source-registry.conf.d\asd-link-farm.conf". This is on XP, but you just need to find the home directory on your Vista and do the same thing.

2. Create a directory where you'll put your lisp projects. This can be any directory and can include non-lisp things too.

3. In your conf file add

(:tree "full-path-of-your-lisp-project-directory")

Your Lisp should now scan that directory and its subdirectories. You can add multiple lines for different directories if you wish.

4. When you create a new project put it in one of these directories. Then in your LIsp do (asdf:initialize-source-registry) and the registry will include your new projects loadable via asdf or Quicklisp. Of course if you restart your Lisp it will automatically contain all these projects from the get go (of course you have to load them, but they're accessible).

The creator of Quicklisp has a good post on this topic here, http://xach.livejournal.com/278047.html?thread=674335.

Thanks
But, by "home directory", do you mean the home lisp implementation directory? Do I have to have another directory named "source-registry.conf.d" which contains the .conf file?
Thanks again.

The home directory is your user home directory. The tricky part is that your Windows OS may not call this a 'home directory', but on my Windows XP, for my user 'george' (me), it looks like:

"C:\Documents and Settings\george\"

So the directory for the .conf file is:

"C:\Documents and Settings\george\.config\common-lisp\source-registry.conf.d\asd-link-farm.conf"

Now another trick is that different applications will not always choose your Windows home as their home directory, but if they don't they usually choose a subdirectory of it. For example Emacs on Windows will use the 'Application Data' subdirectory from the user home as its home. But it's likely that your Lisp will choose your Windows home as the 'home directory'.

ASDF2 actually has many rules for where it finds packages, the one I gave as an example is just the simplest. You can read more about it here:

http://common-lisp.net/project/asdf/asd ... or-systems

But the simple explanation is here

http://common-lisp.net/project/asdf/asd ... uring-ASDF

To find your "home directory", run (user-homedir-pathname) in lisp.

Must my ASDF package also be in the home directory?

Indecipherable wrote:Must my ASDF package also be in the home directory?

No. ASDF packages can be anywhere on the filesystem. Only the config files must be in one of the pre-specified locations.

Note that it should be possible to use shortcuts on windows systems, and even symlinks on Vista or 7; but these are not well supported by ASDF or some CL implementations (i.e. breakage would be unnoticed by the main developers).

ASDF's basic procedure to evaluate (asdf:load-system :foo) is
- look in the config files (or asdf:*central-registry* if non-empty) to find locations for *.asd files
- read each asd file in order until one says (defsystem :foo)
- resolve any symlinks or shortcuts to find the "truename" (actual physical location) of the asd file
- look for source files using paths relative to that truename
- check whether precompiled object files (fasls) exist and load them if current or recompile if stale (timestamp compared to source)
- load in the fasls