Page 1 of 1

how to get clisp to use all available cpu cores

Posted: Mon Jul 25, 2016 3:16 pm
by globaltree
Hello Lisp World.

Over remote ssh connection to a google-compute-engine instance, running slackware64-14.2, I am trying to build sbcl-1.3.6 from source, cross-compiling using clisp.
Here's what I did:

Code: Select all

# wget --no-check-certificate https://sourceforge.net/projects/sbcl/files/sbcl/1.3.6/sbcl-1.3.6-source.tar.bz2
# tar -xSf sbcl-1.3.6-source.tar.bz2
# cd sbcl-1.3.6
# screen
# sh make.sh --prefix=/usr --dynamic-space-size=2Gb --xc-host='clisp -q'
Then I pressed Ctr-A Ctr-D to detach the screen.
Curious about the cpu load, I ran top:

Code: Select all

# top
The output showed cpu usage at only 6%. Clearly the build is not using all of its 16 cpu cores.

Is there any thing I can add to these commands to speed things up?

Thanks in advance.

Re: how to get clisp to use all available cpu cores

Posted: Tue Jul 26, 2016 3:48 am
by pjstirling
sbcl's build has support for building with multiple threads, but it's still quite new so there may still be issues, though it did pass the test suite the last time I built it that way.

export SBCL_MAKE_PARALLEL=16

Re: how to get clisp to use all available cpu cores

Posted: Thu Jul 28, 2016 5:45 am
by sylwester
Why don't you just download a binary distribution of sbcl for 64 bit linux and use that? It's a lot faster!

Re: how to get clisp to use all available cpu cores

Posted: Fri Aug 05, 2016 9:52 am
by globaltree
Hello LispForum, and thanks for your responses.

Sylwester, the reason I compile sbcl instead of using one of the 64-bit binaries already made, is because I use slackware's pkgtools (installpkg and removepkg) to personally manage by hand the packages on my system,(as it is a minimal system, and there are few to manage); and slackware would prefer sbcl core to be in /usr/lib instead of /usr/local/lib, and so by compiling, I can customize the prefix. I've been making sbcl for slackware for years this way, and once it's compiled (and the --fancy option builds it for every architecture it can, including 64-bit), I build a slackware package by customizing the INSTALL_ROOT for my package root, and then packaging that package root into a slackware package that I can install and remove with slackware's pkgtools. But I should think the resulting binary is just as fast as one of the officially distributed binaries.

Based on pjstriling's suggestion, I recompiled sbcl three separate times, with three separate scenarios (all three using old version of sbcl to compile new version--and not using clisp as xc-host, so I can try sbcl-make-parallel flag, which I figured would do nothing for clisp. The first two compilations are on system with 16cpus, one without the sbcl-make-parallel flag, and the other with it. The third compilation is on single cpu system with out any flags. I timed them with "time" and here are the results:

Code: Select all

Compiled without setting any flags, with 16 ivy bridge core i7 cpus, and 60gb ram:
//build started:  Fri Aug  5 06:15:39 UTC 2016
//build finished: Fri Aug  5 06:21:20 UTC 2016
sh make.sh --prefix=/usr --dynamic-space-size=2Gb --fancy  325.66s user 37.96s system 106% cpu 5:41.67 total

Compiled, setting the SBCL_MAKE_PARALLEL flag, with 16 ivy bridge core i7 cpus and 60gb ram:
//build started:  Fri Aug  5 06:32:46 UTC 2016
//build finished: Fri Aug  5 06:35:55 UTC 2016
SBCL_MAKE_PARALLEL=16 sh make.sh --prefix=/usr --dynamic-space-size=2Gb --fancy  389.37s user 49.58s system 231% cpu 3:09.50 total

Compiled, without setting any flags, with 1 ivy bridge core i7 and 3.75 gb ram:
//build started:  Fri Aug  5 15:35:11 UTC 2016
//build finished: Fri Aug  5 15:40:35 UTC 2016
sh make.sh --prefix=/usr --dynamic-space-size=2Gb --fancy  
294.41s user 14.21s system 95% cpu 5:24.20 total

Interestingly, it was faster on the wimpy system, with no flags, than on the one with 16 cpus, and no flags. But they were roughly the same. The compilation with 16cpus and the SBCL_MAKE_PARALLEL flag was over two minutes faster, as should be expected. As before, I detached the screen, and ran top during the compilation. During the compilation with the SBCL_MAKE_PARALLEL flag set, I notice multiple sbcl processes in the output of top--perhaps even 16 of them--leading me to believe that it did use multithreading, and it the multi-threading build completed in less time, as it should.

So thanks to this forum and to pjstirling, I know how to save time when using sbcl to compile.

Thanks again for your responses. Cheers.