SBCL Quicklisp & CFFI on OpenBSD

Hi,

I would like to tinker with GSLL on an OpenBSD-5.3 machine. Using quicklisp to load GSLL:

* (ql:quickload "gsll")

results in some errors:

; Loading "gsll"
sbcl:/usr/local/lib/libffi.so.6.1: undefined symbol 'pthread_mutex_unlock'
sbcl:/usr/local/lib/libffi.so.6.1: undefined symbol 'pthread_mutex_lock'
sbcl:/usr/local/lib/libffi.so.6.1: undefined symbol 'pthread_mutex_init'

After some digging in libffi, I found a comment in ChangeLog that says "Link openbsd programs with -lpthread". Looks interesting. Unfortunately, I haven't found any top-level configuration options in CFFI that specifies how libffi is linked. There is a file:

lucidrine:/home/hanzer/quicklisp/dists/quicklisp/software/cffi_0.11.2/libffi$ libffi-unix.lisp 

that specifies compiler options for darwin:

;; When installed through Mac Ports, libffi include files
;; will be found in /opt/local/include.
#+darwin
(cc-flags "-I/opt/local/include/")

I would like to try a Hail Mary stab at linking libffi with -lpthread by adding something like this:

#+unix
(cc-flags "-lpthread")

Easy enough, but being fairly new to quicklisp (<12 hours), I don't know how to make the system evaluate the modified code.

I will continue to explore, of course, but I've been itching to get involved with the forum. Any ideas, tips, or suggestions?

When CFFI tried loading libffi, it appears to fail due to the unresolved symbols.

This indicates that you need to get libpthread.so loaded before loading CFFI.

The easiest way to test this is to set LD_PRELOAD before starting sbcl. The one-line bash syntax is "LD_PRELOAD=/path/to/libpthread.so sbcl".

If that works, then one more permanent solution would be to link SBCL itself against libpthread. That would require rebuilding SBCL and tweaking the build somewhere.

I suspect the best approach is to inject a (USE-FOREIGN-LIBRARY libpthread) right before CFFI loads libffi. Then you could submit a patch to CFFI.

Awesome suggestions! I'm tinkering with the LD_PRELOAD tactic. Here's the skinny on this morning's activities:

I built a recent version of libffi from source and installed it locally. I suspect that OpenBSD's version from /usr/ports will work but the installed name is libffi.so.0.0 - quicklisp/dists/quicklisp/software/cffi_0.11.2/libffi/init.lisp expects a different name:

  (:unix (:or "libffi.so.6" "libffi32.so.6" "libffi.so.5" "libffi32.so.5"))

Easy enough to fix/look-into later but for this morning I stuck with the locally installed recent version. Here's what happened:

$ export LD_LIBRARY_PATH=/home/hanzer/opt/lib
$ export CPATH=/home/hanzer/opt/lib/libffi-3.0.13/include
$ LD_PRELOAD=/usr/lib/libpthread.so.17.0 sbcl --load quicklisp.lisp  
* (load "/home/hanzer/quicklisp/setup.lisp")
* (ql:quickload "gsll")

The LD_PRELOAD of libpthread solved the undefined symbol errors. Woohoo! The compilation proceeded a bit further then:

[package cl+ssl].
debugger invoked on a CFFI:LOAD-FOREIGN-LIBRARY-ERROR:
  Unable to load any of the alternatives:
   ("libcrypto.so.21.0" "libcrypto.so.20.1" "libcrypto.so.19.0"
    "libcrypto.so.18.0")

In OpenBSD-5.3 it is libcrypto.so.22.0. After a little tweak to quicklisp/dists/quicklisp/software/cl+ssl-20130420-git/reload.lisp

(progn
  (cffi:define-foreign-library libcrypto
    (:openbsd (:or "libcrypto.so.22.0"
                   "libcrypto.so.21.0"
                   "libcrypto.so.20.1"
                   "libcrypto.so.19.0"
                   "libcrypto.so.18.0")))
  (cffi:use-foreign-library libcrypto))

Try again and the compilation gets a little further then:

; caught ERROR:
;   READ error during COMPILE-FILE:
;   
;     :ASCII stream decoding error on
;     #<SB-SYS:FD-STREAM
;       for "file /home/hanzer/quicklisp/dists/quicklisp/software/antik-20130720-git/init/generic.lisp"
;       {41EC9219}>:
;   
;       the octet sequence #(194) cannot be decoded.
;   
;     (in form starting at line: 192, column: 18, file-position: 6849)

I haven't really looked into this error yet. Actually, there is a FreeBSD-9.2-RC4 installation DVD sitting in the drive of my OpenBSD development machine; it's just waiting for a reboot. I went through the SBCL build process on OpenBSD last night. It wouldn't run many of the regression tests and seemed to fail some of those it did run. It would be nice to see SBCL build on OpenBSD with the --fancy options and pass all of the regression tests. Working toward that goal might be an interesting way to learn and explore. Although, pressing the reset button and getting on with it would also be fairly gratifying.

Hi hanzer,

I've looked into these issues lately and have sent patches for cl+ssl and cffi projects to fix loading of libraries.
https://github.com/cffi/cffi/pull/37
https://github.com/cl-plus-ssl/cl-plus-ssl/pull/2

With above patches the cffi-libffi library loads just fine for me on OpenBSD-current running on amd64.

If you want to use threaded sbcl you can my previous patch:
http://permalink.gmane.org/gmane.lisp.s ... evel/17092

Threads seem to work on amd64 but are broken still on other archs. There is some work towards fixing i386 and ppc archs.

Ah, forgot to mention in previous message that you can add following to ~/.sbclrc to see if it helps for the encoding issues:
(setf sb0impl::*default-external-format* :utf-8)