Runtime compilation strategy

I'm doing small lisp like compiler (to x86 asm) and what is bothering me is "runtime compilation", it's simple enough to do simple static C-like compiler with basic Lisp stuff (C runtime/gc and Lisp compiler emiting x86 asm) but i'm not sure how to do runtime compilation and all the problems that goes with it.
I assume that I can start extern gcc assembler each time from Lisp REPL and use C lib dynamic linker/loader (not pretty ?), or meaby create directly in memory opcodes ... but in both cases I'm not completely sure how to do dynamic symbols address resolving, heap functions/closures .....
I dont have experience in writing compilers .. so if someone can explain basic runtime compilation/loading tactics i will be grateful
Thanks

Incremental Compiler Project

The goal of this project is to turn GCC into an incremental compiler. GCC will run as a server and maintain a model of the user's program. When a translation unit is recompiled, GCC will re-compile the minimum necessary. Read the announcement and check out the branch at svn://gcc.gnu.org/svn/gcc/branches/incremental-compiler

http://gcc.gnu.org/wiki/IncrementalCompiler

http://en.wikipedia.org/wiki/Incremental_compiler

http://hedgehog.oliotalo.fi/

http://books.google.com/books?id=iCrrcM ... q=&f=false

makia wrote:I'm doing small lisp like compiler (to x86 asm) and what is bothering me is "runtime compilation", it's simple enough to do simple static C-like compiler with basic Lisp stuff (C runtime/gc and Lisp compiler emiting x86 asm) but i'm not sure how to do runtime compilation and all the problems that goes with it.
I assume that I can start extern gcc assembler each time from Lisp REPL and use C lib dynamic linker/loader (not pretty ?),

I don't know if it is pretty or not, but calling gcc to create a shared object and linking to it is what ECL does. A friend of one friend of mine also used to do this in a program written in C# (but nothing to do with Lisp).

Yup, take a look at ECL. When you call COMPILE-FILE in ECL, it effectively spits out a temporary .c file, invokes GCC on it, and then when you LOAD it, it dynamically loads the resulting dynamic lib.

Most other Lisp compilers don't work this way, BTW. Others, like SBCL, have their own compiler/assembler internal to the Lisp image and can generate assembly code directly into memory or to a file without the need of gcc.

It is much more common for scheme implementations to be built on C compilers. See for example Gambit or Chicken.

findinglisp wrote:Yup, take a look at ECL. When you call COMPILE-FILE in ECL, it effectively spits out a temporary .c file, invokes GCC on it, and then when you LOAD it, it dynamically loads the resulting dynamic lib.

Most other Lisp compilers don't work this way, BTW. Others, like SBCL, have their own compiler/assembler internal to the Lisp image and can generate assembly code directly into memory or to a file without the need of gcc.

yes, i know that, but for example in ECL when you type function in REPL i doubt that you get compiled function (it does not invoke GCC), so if I want something like that I need to invoke gcc on every REPL input or implement interpreter beside compiler
I'm more interested in SBCL/CCL way, to assemble things directly into heap but i'm not sure how to implement this, obvious compiler need to have runtime support to push code into heap ... so it looks like that i dont know how to connect runtime and compiler to have working environment

makia wrote:I'm more interested in SBCL/CCL way, to assemble things directly into heap but i'm not sure how to implement this, obvious compiler need to have runtime support to push code into heap ... so it looks like that i dont know how to connect runtime and compiler to have working environment

I played only a little with assembly, but as far as I know you just mark a heap region as executable if necessary, depending on your operating system (see NX bit on Wikipedia, mprotect syscall on Linux) and jump to it. Of course you need an assembler to generate the machine code which can be placed in memory directly. Other than that you don't need "runtime support to push code into heap" other than normal memory manipulation tools.

Ramarren wrote: I played only a little with assembly, but as far as I know you just mark a heap region as executable if necessary, depending on your operating system (see NX bit on Wikipedia, mprotect syscall on Linux) and jump to it. Of course you need an assembler to generate the machine code which can be placed in memory directly. Other than that you don't need "runtime support to push code into heap" other than normal memory manipulation tools.

You'll also need garbage collector support for moving and reclaiming the code. Moving code in memory might (or might not, depending on linkage strategy) cause text relocations of code that references the generated code. If the generated code closes over some variables, then GC also should support walking over closures.

makia wrote:yes, i know that, but for example in ECL when you type function in REPL i doubt that you get compiled function (it does not invoke GCC)

Indeed, when you type a function in the REPL (or just loads a file without compiling it) ECL transforms your code into bytecode, which it interprets later when necessary. ANSI Common Lisp does not require implementations to compile every function, that is why the functions compile-function and compile-file exist. But there is also nothing preventing you to compile every function using gcc either.

Not that I've got any experience, but instead of generating assembly directly, why not use something like http://llvm.org/, and benefit from all of the optimisers and things in that project It mentions some type of run-time compilation too.

Kohath wrote:Not that I've got any experience, but instead of generating assembly directly, why not use something like http://llvm.org/, and benefit from all of the optimisers and things in that project It mentions some type of run-time compilation too.

I looked llvm couple of days ago, problem for me is that runtime JIT compiler API is C++ only and i have zero C++ experience and i'm not crazy for getting it too(i dont have great C experience also, but i can manage with it)
I'm not trying to do full ansi optimizing compiler (i'm far for capable for doing that) .. but basic stuff should work
I'm currently having problem to abstract whole design ...