I dont get Macros

I am new to lisp and I am already glad that I learned the language.

When people complain about the syntax and/or lisp in general I think they are are confused by the concepts and not the language itsself. For example I hardly needed a day for recursion, conditionals and the parenthesis structure, but I needed half a week just to get the applicative operators and to learn why programming with them is a good idea. I think many people are confused by the complex concepts like functional programming, recursion, functions as primary objects, the extended list navigation functions (like first, cdar, rest) and the not always intuitive list building functions (list, append, cons). After I "got" those they are easy to use and effective.

Here is my problem though: Apparently macros are one of the few thing which makes Lisp different and better than other modern high-level languages like Python or Ruby (I am not talking about low-level languages like C). And Macros are "responsible" for the typical lisp code with numerous parens and the compact not line oriented structure. So, all in all, macros seem important.

But I can't figure out why they are a powerful tool. The behave like functions: If lisp finds an object after the opening parens it is assumed to be a function/macro. The symbol-cell and the corresponding call is triggered, the arguments evaluated (functions) or taken (macro) and some list is returned. Why are macros such a good idea?

My teaching book defines two differences between macros and functions:
1) Macros dont eval their arguments
2) Macros return List-Code which is immeaditly evaluated

I dont see any possibilities for the first point to have a effect: Suppose you have some babble input and want them to communicate with your system. So you build a parser which morphs a string input into useful code. Considering the high number of sequence operations I think macros are not really needed. You just take string or list as input, build tokens and modify them. The point here is: You can evaluate your input everytime, because either your input makes sense from a lisp point-of-view (and you want them to be evaluated) or you take a string/list as a selfevaluating object and return them after you parsed them (which is needed anyway).

And the second point makes no sense too: A function returns data. If a macro returns code which is evaluated it must, at the end, produce data.

Last point: Macros can define functions or variables at the top level. I dont think that is not that useful either because functions can return functions and set variables. While it might be nice to write a function defining macro a function which returns a function does a similar thing.

;not tested
(defmacro adder1 (name arg)
   `(defun ,name (input)
        (+ input ,arg))

;function does just as well
(defun adder (arg)
      (lambda (d) (+ d arg)))

Can somebody show me either
a) examples where macros are desperatly needed
b) some patterns where macros are used
c) explain how to build an embedded language on top of lisp like I heard so many times

One thing you don't seem to be aware of: Common Lisp is a compiled language. Most implementation will compile to native machine code applying many optimizations. See the output of DISASSEMBLE function, although some implementations, like Clisp do emit bytecode. Some implementations have direct interpreter mode, but that is not really relevant to the purpose of macros.

Destruct1 wrote:1) Macros dont eval their arguments
2) Macros return List-Code which is immeaditly evaluated

This is not exactly true. A macro is a hook into the compiler. The argument to a macro is a source form, and the returned code is not immediately evaluated, but returned to the compiler for further processing. The point is to avoid processing the macro-input every time and just execute the optimized returned code.

Canonical examples of basic macros are definers, like the standard DEFCLASS and dynamic context establishers, like WITH-OPEN-FILE. For that matter, SETF is a macro.

Obviously, everything that a macro does can be achieved by manually typing in its expansion, but the same is true about compilers, the same thing could be achieved by typing in machine code directly.

One problem with explaining macros is that simple examples are so simple that it is not obvious that they are worth it, but complex examples are incomprehensible to someone who doesn't already understand the system the example is taken from. That said, the most recent macro I had written was in my parser combinator library, which contains nineteen macro definitions in total.

Macros are useful for many purposes. They can make your code faster in many cases by transporting some of the computation from runtime to compile-time, for instance, simplifying the computation when some of the arguments are known. For that matter, there exist compiler-macros, you can learn about them later.

Another example is convenience. 99% of CL libraries come with macros to simplify the use of the library. The macro with-open-file is a canonical example: instead of forcing the programmer to close every file that was open, create a macro that does it for the programmer. Instead of teaching the programmer what functions are needed to initialize and shutdown a library (for instance, SDL or opengl), you can create a macro that accepts some options and does the dirty job for the programmer (e.g. with-sdl or with-opengl).

Those are just the simplest examples, there are many situations where you can create your own syntax that still looks like lisp, but which wouldn't be possible without creating macros or modifying the compiler to accept them. The canonical example is the loop macro, e.g.:

(loop for i from 0 to 200 collect i)

This code would not work if loop was a function: the environment would complain that the symbols for, i, from, to and collect are not defined.

In any case, just give it some time and you will understand and create very useful macros

Destruct1 wrote:1) Macros dont eval their arguments
2) Macros return List-Code which is immeaditly evaluated

I dont see any possibilities for the first point to have a effect

No, this is great. You can transform your input however you like to automate writing basically any code you can dream up. If the arguments were evaluated, everything would have to be quoted. It's always hard to give good macro examples that won't be trivial or too complicated to understand, but (besides the many examples in the standard) metabang-bind (http://common-lisp.net/project/metabang-bind/) might make sense to you. Common Lisp has a lot of forms that establish bindings, and when you need to use a bunch of them it can get hairy. Thanks to macros, it was possible to write a wrapper that can be much simpler to use.

The loop macro is another good example.

Destruct1 wrote:And the second point makes no sense too: A function returns data. If a macro returns code which is evaluated it must, at the end, produce data.

It's not necessarily immediately evaluated. It's substituted into your code in place of the macro call. This means you can do things like creating lexical variable bindings that you couldn't do otherwise except by writing all the code yourself every time. Some people like to say about Lisp that "code is data". Yeah, functions take and return data, but what about when that data is code? You need a macro so you can type the input data verbatim, and you need need a macro so the returned data can actually end up in your code before compilation or evaluation.

(My apologies if I misunderstood your question.)

Actually, the bit about evaluation is a very important distinction. Imagine you wanted to express something like

(unless (zerop x)
  (/ foo x))

Obviously, we don't want to do the division unless we know x is not zero. So, we need something that will not always evaluate (/ foo x). This is why this must be a macro, and cannot be a function. If it were the latter, the division would always get evaluated regardless of the value returned by zerop. (Strictly speaking, that's not true; you could quote the expression, and then call eval when you deem it safe, but eval in application-level software is rare (and often (though not always) bad style).

What's more, many lisp systems only provide one conditional special operator under the hood. Usually, this is cond or if. In those systems, the one is written in terms of the other as a macro. Then, in turn, the (slightly higher) level constructs like when are written in terms of if and progn. Then, finally, the nonstandard-but-you-find-them-everywhere idioms like awhen or let-when are written as macros in terms of those.

Have you played much with macroexpand-1 and macroexpand? Also, nonstandard but very common is macroexpand-all (if your system lacks it, there are variants of it all over the 'net).

I think a practical example would help. Look up the macro aif. There's one in Paul Graham's on-lisp book. It's simple and easy to understand. Once you've got it, try to write something similar, like awhen. Switch to a language you're familiar with and try to implement aif. This will give you a taste of macro power in albeit a simple example.

You're right, the concepts behind lisp are hard. Peter Norvig wrote a great paper called Learn Programming in 10 years (or something like that). I think the concepts behind programming in general are hard, which is why there aren't that many good programmers (IMO there aren't even many mediocre programmers). People think that because some guy can put together a UI or web page that they're a programmer. However I'm often ashamed by the mess that I so often come across and that my profession for more than 20 years is currently in such disarray. One good programmer is an order of magnitude more valuable that most people I come across day to day in my business.

Your questions are intelligent and clearly you're on the good path, ignore what the majority think, persevere, you're needed!

- Paul

The best explanation of macros that I know of is from Practical Common Lisp. See the chapter on writing your own macros, at http://gigamonkeys.com/book/macros-defi ... r-own.html.

Macros are very different from functions. They are a way to write language extensions and domain-specific languages in Lisp.

I'll pitch in with my bit of input, because you haven't already been deluged enough

Macros are evaluated at compile-time, and are never seen at runtime (unless the application itself writes and compiles code that calls on them).
When the compiler is reading the source-code and encounters a call to a macro, it immediately executes the macro, replaces that section of source-code with the result, then compiles the altered source.
To use a really trivial example that shows none of the actual power of a macro, if you write this:

(defmacro foo (zot)
 (list 'format 't "~a" zot))

(defun bar ()
 (foo "boing"))

...the effect will be the same as if you'd written:

(defun bar ()
 (format t "~a" "boing"))

It may help to explain the intent of macros, to make sense of what they do. You know how sometimes you find yourself writing the same pattern of code for the nth time, and think, "man, I should really write a programme that would write this code for me"? That's what they do. Macros are mini-programmes that the compiler runs to generate the source-code; they let you define the pattern, then invoke that pattern and just feed it the interesting bits. Don't let the above example fool you into thinking that you can only use substitution; you can do anything to the input that can be expressed in Lisp.

Where the power and confusion both enter the picture is that you get to call on the full power of Lisp to generate Lisp source-code. On the one hand, you don't have to switch contexts to think in a different language when writing your code-generating programmes, but on the other hand there's not much to remind you that it's the output of the macro call that will be compiled and not the literal source that you're writing.

JamesF wrote: You know how sometimes you find yourself writing the same pattern of code for the nth time, and think, "man, I should really write a programme that would write this code for me"? That's what they do.

The main point is: What is the difference between a function and a macro. What is so special about a macro that cannot be done otherwise?

Lets go through the examples here and compare them with Python code.
I keep the Python Code simple and readable to avoid confusion, but all my examples can be written shorter.

1)

(unless (zerop x)
  (/ foo x))

Thats easy:

def safediv (x, y):
  if y!=0:
    return (x/y)
  else:
    return (False)

2) The DoPrimes Macro from Practical Common Lisp
Its like dotimes but only outputs prime numbers. It uses primep to check if something is a primenumber

def primegen (until_number_reached):
  primegen = (e for e in range (2, until_number_reached) if primep (e) == True)
  return (primegen) # return an generator expression which can be iterated over

In action:

for x in primegen (30):
  # do something

3) The anphoric Macro in OnLisp:

It’s not uncommon in a Lisp program to want to test whether an expression
returns a non-nil value, and if so, to do something with the value. If the expression
is costly to evaluate, then one must normally do something like this:
(let ((result (big-long-calculation)))
(if result
(foo result)))
Wouldn’t it be easier if we could just say, as we would in English:
(if (big-long-calculation)
(foo it))
By taking advantage of variable capture, we can write a version of if which works
just this way.
[..snip..]
and used as in the previous example:
(aif (big-long-calculation)
(foo it))

I probably dont get that, but it is very easy:

def aif (long_calculation, further_func):
  if (long_calucaltion) != 0:
    return (further_func (long_calculation))
  else:
    return (0) # ??

So here is the point: Macros can be easily implemented by other things

Destruct1 wrote:So here is the point: Macros can be easily implemented by other things

To reiterate: macros are hooks for a compiler. They allow you to extend the compiler. Macros can be implemented as other things in the same sense that everything a compiler does can be by writing it manually in assembly. The point of macros is the same point as that of compilers: it allows automation and syntax/semantic extension.

Destruct1 wrote:I probably dont get that, but it is very easy:

def aif (long_calculation, further_func):
  if (long_calucaltion) != 0:
    return (further_func (long_calculation))
  else:
    return (0) # ??

So here is the point: Macros can be easily implemented by other things

Well, aif would be more like:

(aif (some-function)
     (do-something-with it)
     (do-something-else))

==>

(let ((it (some-function)))
  (if it
      (do-something-with it)
      (do-something-else)))

And it is true what you said: all that can be done with macros can be done in another way - just type its expansion and you are done. But that would make your work repetitive and boring. Sometimes you would have to write some code twice, which is obviously error-prone and not the right way. Sometimes you would be obligated to know the internals of some library.

Your suggested workaround for aif (using functions) would not be good enough. The easiest way to avoid using aif is just writing its expansion, which is not ugly at all. But it is too long and repetitive, and it will take your time while you could be working on something else in your code. The whole point about aif is its simplicity. It makes your code more direct and will keep you away from the tiny details of you code you just don't care about - in this case, the declaration of a variable just to hold a value before you can test if it is true and use the result.

The macro aif alone can't do very much with the transparency of your code if it is the only macro you have, but, having a small cosmetic macro here, another one there, and another utility function over there, in the end your code will be much more transparent.

The good Lisp style is about brevity and transparency. You should not write only what you want to do, not more, not less. If you want to do something with an open file, you just use with-open-file, instead of telling the compiler what to do (which would be you saying "declare a variable, open a file and assign the result to that variable, check if the file was successfully open, do something, then close the file"). The interface to the functionality of your code should be clear instead of requiring the user to do small repetitive ugly things to work with it.

In the end, Lisp libraries look much more like a DSL created on top of Lisp than a library itself. Each one have its own syntax to make its use simpler and clearer.

gugamilare wrote: Your suggested workaround for aif (using functions) would not be good enough.

Why?

The Python function takes 3 Parameters and dispatches them.

I do the exact same thing:
I write a block of code as a definition then use that function in actual practice.
Writing a function definition is easier than a macro expansion.
And there is no difference when calling my construct: (my_def_macro argument1 arg2 arg3) <=> my_python_func (arg1, arg2, arg3)

To really make a difference Makros MUST make things possible who are:
Not implemented in the language
Practical
Not already done by functions

And the problem is: I havent seen a single good example
There are tons of examples for closures, aplicative operators, better native datatypes etc.
But not for Macros

Destruct1 wrote:And there is no difference when calling my construct: (my_def_macro argument1 arg2 arg3) <=> my_python_func (arg1, arg2, arg3)

CL-USER> (aif (and (plusp (+ 2 2)) (+ 2 2)) (+ 3 it))
7

>>> aif(((2+2)>0) and (2+2),3+it)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'it' is not defined

The difference is that AIF macro doesn't take parameters, it takes code, and transforms the code such that there is a new binding for variable IT. And your Python function doesn't do it, you have to give function objects as arguments and it creates no new bindings. It is a matter of typing obviously, you could achieve it by adding 'lambda it:' in there, but that is the point, macro allow you to automate adding boilerplate like that.

Destruct1 wrote:And the problem is: I havent seen a single good example

As was written before: any macro trivial enough to explain as an example is most of the time trivial to expand manually, and hence the benefit is not obvious. A somewhat non-trivial macro may look like the one I mentioned earlier, and there is no way to implement it as a function without adding a lot of junk at every call site.

There is also other thing. One typical example of macros is WITH macros. They are so useful that an equivalent was added to Python 2.5. And the point is that in Lisp, it is just a normal macro, and the user could define it, and can define new language extensions, while in Python it had to be done by language implementation, and any similar extensions can only be added by implementation authors.

Destruct1 wrote:

JamesF wrote: You know how sometimes you find yourself writing the same pattern of code for the nth time, and think, "man, I should really write a programme that would write this code for me"? That's what they do.

The main point is: What is the difference between a function and a macro. What is so special about a macro that cannot be done otherwise?
<snip>
So here is the point: Macros can be easily implemented by other things

Except for the "easily" bit, yes, you're quite correct. Macros generate Lisp code, which means they expand into Lisp forms, which can by definition be written by hand. If you want to.

I think you're missing something very important, though: macros are useful in Lisp for the same reason that you use Python instead of programming directly in assembly language (or even machine code). There's nothing you can do in Python that you can't "easily" implement in assembly, but which one would you rather use? Start the comparison with a trivial example, then scale it up to a really big system, and consider how much more of a win you get with the high-level language.

The point of macros is to abstract away pointless repetition, which both saves time and effort, and massively reduces debugging time. Their value is relative to the size and complexity of the system you're building, so if you're accustomed to writing scripts of a hundred lines or so (as I was when I started with Lisp) it can be hard to see what all the fuss is about. However, the first time you halve a thousand-line codebase by writing a couple of things that write half the code for you, the lightbulb really goes on.

I'll tackle it from another angle: what I really love about Lisp is that it allows me to write the most compact, concise code that still expresses what I want to do - all the pointless, repetitive crap can be factored away. In that sense, it's like a compression algorithm. I can write general-purpose utility functions that take other functions as arguments, and abstract away swathes of repetition, which in itself is a huge win. But then I can identify other general patterns and condense them into a single function that can be called on to generate them at compile-time, compressing the whole thing even further. To borrow a term from alchemy, you can reduce your solution to its quintessence.

Something that programming in lisp taught me was to identify the entire class of problem that I'm solving, and to solve that instead of the immediate issue. Then I have a general solution that I can use the next time I run into that problem. Macros do the same thing, but crank it up an order of magnitude. Now repetition in my code looks like a bug, and my projects are mostly in the single-digit thousands of lines. I don't write many macros, but when I do, I win big in either consistent correctness, code size and complexity, or both.

So what a macro saves you is tedium, boredom and Cobol fingers. Do you really want to write the same things over and over again, or would you rather spend your time solving new problems?

You'll likely use far more macros than you write. From DEFUN to LOOP, you already use all sorts of macros someone wrote, particularly those which are so broadly useful that they're well-debugged and documented.

And when you delve into various domains, you'll run into domain-specific macros. Like with HTML: take CL-WHO.

With other languages, you have to wait for the language implementor to develop some handy new for() loop. If you want to play with new abstractions, like nice support for monads, then you're definitely out of luck unless you fork the language, or write a preprocessor or something. With Lisp, this part of the language is more decentralized; someone oblivious to the language's implementation can nevertheless innovate, and package it as a library. And interested others can build upon that innovation, without it necesasrily having to be an officially blessed part of the language.

Perhaps the recent Clojure screencast about parentheses might be interesting to you. They're making fresh new justifications for canonically Lispy features.

Macros help multiply the power of other language features. But that said, Ernst van Waning gave a wonderful talk on the disadvantages of macros, worth keeping in mind. I don't want to breathlessly sing the praise of macros, as they have their place. These are not things you want to write all the time.

Conrad Barski calls macros "SPELS"
http://www.lisperati.com/casting.html

A more direct link: http://www.lisperati.com/no_macros.html

I found it an easy to understand read.

titanium_geek wrote:Conrad Barski calls macros "SPELS"
http://www.lisperati.com/casting.html

A more direct link: http://www.lisperati.com/no_macros.html

I found it an easy to understand read.

I rewrote the Lisp adventure in Python. Although this first solution is
not 100% without overhead it is very practical to handle and solves the problem nicely.

currentloc = "living-room"

map_des  = {"living-room":"You are in the living-room of a wizard's house. There is a wizard snoring loudly on the couch.",\
             "garden":"You are in a beautiful garden. There is a well in front of you.",\
             "attic":"You are in the attic of the abandoned house. There is a giant welding torch in the corner."}

map_go   = {"living-room":[("west", "door", "garden"), ("upstairs", "stairway", "attic")],
             "garden":[("east", "door", "living-room")],
             "attic":[("downstairs", "stairway", "living-room")]}

map_items = {"living-room": ["whiskey-bottle", "bucket"],
             "garden" : ["chain", "frog"],
             "attic" : []}

inventory = []
condition = []

For the general enviroment I used a native datatypes of Python, dictonaries. These are comparable to the LISP Hashtable and map a
key/loaction to a value/possible orutes, descriptions, items. Note that most of these datatypes are global and require me to type a lot of words
later on, but I wanted to stay near the LISP code whereever possible.

def describe ():
    print (map_des[currentloc])
    for e in map_go[currentloc]:
        print ("There is a", e[1], "to the ", e[0])
    for e in map_items[currentloc]:
        print ("A ", e, "is on the floor")
    
def walk (whereto):
    global currentloc
    for e in map_go[currentloc]:
        if e[0]==whereto:
            print ("You go to the "+e[2])
            currentloc = e[2]
            break
    else:
        print ("Not possible")
    
def take (whatitem):
    global currentloc
    if whatitem in map_items[currentloc]:
        inventory.append (whatitem)
        map_items[currentloc].remove (whatitem)
    else:
        print ("Not possible")

The "builtin" functions walk, look, take of this textbased adventure are implemented here. The multiple helper functions that the LISP code uses for the look
command are integrated.

def require (tupleinput, **keydic):
    global condition; global inventory;global currentloc;
    if "obj1" in keydic:
        if tupleinput[0] != keydic["obj1"]:
            return (False)
    if "obj2" in keydic:
        if tupleinput[1] != keydic["obj2"]:
            return (False)
    if "place" in keydic:
        if currentloc != keydic["place"]:
            return (False)
    if "cond" in keydic:
        while keydic["cond"]:
            if keydic["cond"].pop() not in condition:
                return (False)
    if "inv" in keydic:
        while keydic["inv"]:
            if keydic["inv"].pop() not in inventory:
                return (False)
    return (True)

def weld (*input):
    if require (input, obj1 = "chain", obj2="bucket", place = "attic", inv = ["bucket", "chain"]):
        print ("The chain is now welded to the bucket")
        global condition
        condition.append ("chainweld")

Here we come to the important part:
Lisp code uses a macro generating macro to have the following functionality:

gaming-action Name Object1 Object2 Place
Body

defines a possible action Name and restricts execution unless the current location is Place and the attributes of the action are Obj1 and Obj2.
In Python you have to write for the same effect:

def Name (*input):
if require (input, place = Place, obj1 = Object1, obj2 = Object2):
Body

While that is some overhead, it is very minimal. You basically only have to write input twice and add "if required". The additional keyword passing
which makes the code look bloated can be avoided by a restrictive require function which doesnt take keywords. The Python code can then be rewritten as:

def Name (*input):
if require (input, obj1, obj2, place):
Body

But it is possible to recreate the functionality of the gaming-action macro by using string manipulation and the exec command which
executes the given code passed as string

-> next post

Destruct1 wrote:But it is possible to recreate the functionality of the gaming-action macro by using string manipulation and the exec command which
executes the given code passed as string

That is not recreating functionality of a macro, but of EVAL. Of course, when ran in the interpreter the difference is minimal, but that is part of the point: macros are extending a compiler. If you do not even have a compiler, then importance of macros drops and you can go with fexpr or something.

Destruct1 wrote:While that is some overhead, it is very minimal.

The problem with overhead is that it tends to grow exponentially with problem complexity. So for simple problems it may be minimal and just as well done by hand, but for complex ones it becomes such that you have to use code emitters anyway, and macros are almost always vastly superior to that.

The problem with doing string manipulation on source code is that it is notoriously brittle. The point of Lisp is that the source code syntax is essentially a literal syntax for basic datastructures, which then can be manipulated safely by code. On the other hand, syntax of most other languages (many concatenative languages being a major exception) does not cleanly map to structured data, which means that you either have to use a complete parser and manipulate the syntax tree which you cannot easily see, which adds a lot of cognitive overhead, which is a reason why every time someone implements this it never gains any significant popularity, or manipulate strings blindly which only works for simplest examples and even then is asking for trouble (with binary operator precedence rule violations a typical example).

Also, is it not missing the point? Nobody doubts that even a bad implementation of macros, or, as it may be, eval, can do what macros in Lisp can do. See Greenspun's Tenth Rule. But by doing this you already acknowledge that macros are useful, which, I believe, was what started this thread.

And here is the code for the hacked python macro:

This defines a string template machine:

def gaming_action (name, obj1, obj2, place, body):
    b = "def " + name + " (*input):\n" + "    if require (input, obj1 = '" + obj1 + "', obj2 = '" + obj2 +\
          "', place = '" + place + "'):\n        " + body + "\n\n"
    return (b)

And this binds greet with full code to the top-level:

exec (gaming_action ("greet", "hand", "wizard", "attic", '''print ("Hi Wiz!");global condition;condition.append ("wizgreeted")'''))

I think Ramarren said many things valuable, just complementing.

You are creating a text processor to do the functionality of a macro, which is a bit rude. Of course that is possible, but it is error-prone and, for a reason, not popular. If you heard about C++ Templates, you will understand that trying to manipulate strings to create DSLs will eventually cause you a headache - there will be bugs introduced which you won't find easily or it will not work the way you expected. If you want a safe macro, you need access to the syntax tree. Lisp's code is mapped very easily to syntax trees in a obvious way, so you won't have to understand how a complicated set of operators are represented by the compiler or language. And you can construct such trees also in an easy way, specially with the syntax provided by ` ' , and ,@ in Lisp.

By using macros, you will think with macros. For instance, I bet one wouldn't come out with the "casting spells" idea if that person didn't have knowledge about Lisp macros and wasn't familiar with them. Learning a new paradigm - in this case, the so-called meta-paradigm - you teach you new ways of solving your programming problems. But, in any case, it is completely your choice whether you want to learn how to use them or not.

The idea of Lisp macro system has been brought to other languages. The example I know is OCaml. I don't know OCaml, but a first look at it makes you see that its syntaxes are much more complicated than Lisp's, because you have to handle more complicated syntax trees to handle order of precedence of operators, among other things. And my first thought would be that, since its macros are not so simple to use, you many times will feel discouraged to use them, therefore making their use much less frequent than in Lisp.

Destruct1 wrote: Can somebody show me either
a) examples where macros are desperatly needed
b) some patterns where macros are used
c) explain how to build an embedded language on top of lisp like I heard so many times

Example 1 :
Ever heard of Paul Graham new language Arc. It has a very nice conditional operator that works something like:

(if  condition1 result1
     condition2 result2
    result3)

which is roughly like common lisp cond:

(cond (condition1 result1)
         (condition2 result2)
         (t result3))

Arc if is very handy when there is only one result statement after each conditional. In a common lisp without macros
we would be doomed to write all those unnecessary parenthesis for eternity or switch to arc. But we have defmacro
so we have :

(defun pair-them (x)
  (cond ((null x) nil)
        ((endp (cdr x))
         (cons (list t (car x))
               (pair-them (cddr x))))
        (t
         (cons (list (car x) (cadr x))
               (pair-them (cddr x))))))
;;; Grahams' if
(defmacro gif (&rest args)
  `(cond ,@(pair-them args)))

Few tests:

(pprint
  (macroexpand-1
   '(gif (= a 1) (setq a 2)
         (= a 2) (setq a 3)
         (= a a) (floor a 3))))
expands into 
(COND ((= A 1) (SETQ A 2))
      ((= A 2) (SETQ A 3))
      ((= A A) (FLOOR A 3)))


(pprint
  (macroexpand-1
   '(gif (= a 1) (setq a 2)
         (= a 2) (setq a 3)
         (floor a 3))))
expands into :
(COND ((= A 1) (SETQ A 2))
      ((= A 2) (SETQ A 3))
      (T (FLOOR A 3)))

Example 2:
q has a very nice lambda like utility that allows programmer to omit parameter list assuming that first 3 parameters are named x y and z

(f - x y z) ; => (lambda (x y z) (- x y z)) 
(f - z y x) <=> (LAMBDA (X Y Z) (- Z Y X)) 
(f list  x (list y "bobi")  (+ y 2) z) <=> (LAMBDA (X Y Z) (LIST X  (LIST Y "bobi") (+ Y 2) Z))

how would you create this utility using functions? With macros its a piece of cake:

(defmacro f (&rest args) 
 `(lambda ,(remove-if-not (lambda (x) (member x (flatten args))) 
                          '(x y z)) ,args))

And those are just simple macros which saved me a lot of typing and brought me many golfing victories.
Macros basically allows you to capture the patterns in the code that can't be captured with plain functions. Sure you could type
everything by hand but that is boring and error prone. For a good explanations of macros try On Lisp and Let Over Lambda
after you finish on lisp. Paul Graham actually starts creating abstractions using functions in
the first chapters of On Lisp the he switches to more luxurious vehicles, the mighty macros.

happy lisping
Slobodan Blazeski

Lisp Macros have the fullowing use:
1) Basic text Search-and-Replace
2) Building a lexical context
3) SETF Accessor
4) Conditional Evaluation
5) Binding things to the TopLevel


While technically every macro is just a search-and-replace texteditor, many example macros are used the same as c-Macros.
They are used to abbreviate syntax, but have no deeper functionality. Example of this include arc = for setf or syntatic sugar to avoid typing ' or the f function or the graham if. Another example would be to change the first,second, third etc. buildin function to n1 n2 n3 etc.
I think these macros are kind a useless. First you often solve a problem you only have in lisp. Most languages dont require to quote data or use kinda long names. While that may save you a few letters I dont think that is the point. Programming languages should represent deeper and more abstract concepts and not try to make their code especially short from a pure letter standpoint. Otherwise you might end up with a unreadable language like Perl.
For example the Graham-if can be written in Python:

if condition1:
    action1
elif condition2:
    action2
etc.

While this code has the useless "if" and ":" it isnt inconvinient to write. I actually like the clear structure of the python code compared to the g-if where the semantic is dependent on the position in the code. I dont want to count if a certain codeblock is in 5th or 6th position and therefore is a conditional or an action. And I expecially dont want to match the parenthisis in my head to decipher if codeblock X is the last thing in the block (and therefore the default-action) or actually just the 7. but not last thing in the g-if (and therefore the condition3). It doesnt matter how much typing you do, if the cognitive overhead is small and you gain readability.

Another point is that these macros dont use the unique "code is data" and "direct access to the parse tree" Lisp is famous for. They can be imitated by such simple things like C macros or text autocompletion.

The (f body) -> (lambda (x y z) body) is neat and good (especially in golfing tournaments) but can be textcompleted easily.

The third point on the list seems to be the Lisp way of creating setter functions.

Lets get to the second and fourth point where I changed my mind about Lisp macros.
I think the crucial point here is that these macros really need a body or complex expression to work out. If you have a simple expression that evaluates to a datastructure (list, string, number, struct etc.) it is possible to replace every macro with a function. Think about it: If a macro/function takes x arguments, which all evaluate themselves, and maps this input to an output, it is impossible to gain any advanatge with a macro. Mapping data inputs to data outputs is the basic definition of a function. Even sideeffects just expand this definition. A function takes several data inputs, produces sideeffects and returns data outputs. (Note: You might gain performance with macros instead of functions)

On the other hand macros might be useful if at least one part of the macro is a complex expression or body, a "code input". To make things simple on my already stressed mind I go with exactly one body/code argument. While technical it is possible to take 2 or more code inputs I think it strains the human mind to do so. So, that leaves us with a bunch of data arguments like strings, numbers or lists and one code block which is passed to the macro as raw, code input and is declared the &body argument.

Now we have the following options:

We create a lexical enviroment in which the codeblock is executed. Simple examples include the with-open-file macro. In this macro a string and a symbol is passed as data argument and a code block is executed with a file-access as lexical enviroment. The code can use the enviroment provided by the macro to access the file.
Unfortunatly to be really useful such a macro must provide additional functionality. If the macro only provides a binding of a certain variable it is useless because other languages can just use assignment to imitate the Lisp macro. A (witha 7 body) macro that just binds the number 7 to the variable a is useless because a=7;{codeblock} has the same effect. The witha macro is just a let macro with a as the default parameter.
The let macro has the power to bind a variable only for a certain amount of time (the scope of the let/macro) but other programming languages dont seem to miss that possibility. Instad they just define a new variable in the local scope and forget the variable as soon as the scope changes. So the Let macro as the basic context creating macro is just the Lisp way of assining variables.
But the context creating macros have more possibilities than standard assignment. For one they can execute enter and exit code. So a file/database/something can be opened/locked/entered, the codeblock executed and the needed close/unlock/cleanup done automatically.
I am not sure how useful lexical binding really are. Complex assignments can be substitutes with a=function (datainput);{codeblock}. The enter/exit trick can be emulated in Python with the "with x as variable:" construct.

The other option is to use conditional evaluation. And again other languages can substitute basic use of conditional evaluation with their if/case statements or similar control structures. Multiple datainput or codeinput doesnt change the ability of other languages to keep up. Even if a macro in Lisp can combine the datainputs to a complex predicate, other languages can use if (function (datainput1, datainpu2,etc.)==True do Codeinput1. The only interesting uses of this macro-class is in combination with other macro-classes.

The last point is binding things to the TopLevel. A macro can define a function (or other macro) at the top level. And again it is trivially easy for other languages to keep up with the possibilities of Lisp if tasks are easy. a=something binds a in the toplevel, b=lambda x:2*x binds a function to the toplevel. But while other languages need an explicit assignment, Lisp can bind symbols which are determined during runtime. A good example which illustrates this point is this one: You have an external configuration file with the structure { Database1="Hello.dat";Database2="DataisHere.dat";DataZum="DatRer.dat" } and want to bind access functions to all these databases to the top level. It is possible to create a Lisp macro that defines the functions Database1get, Database2get, DataZumget all in one go. Other languages at least need to write Database1get = funcfromconfigfile (1), Database2get=funcfromconfigfile(2), DataZumget=funcfromconfigfile (3). Another example which is illustrated in "Lisp: A gentle introduction" is a finite state automaton where the nodes are represented by closures. These closures/node fucntions are bound to the top level and can be easily accessed. So while other programming languages need to write programms which access data, Lisp can write functions which are bound to the data since their creation.
There are Python hacks with the exec command that can simulate this toplevel binding but they arent that well integrated into the language. The "Right Way (tm)" to implement the above example would be to load the configuration file into a classobject, write a generic access function which takes a databasename / number as input instaead of defining functions only for a certain database. This means writing function which access data instead of writing functions which are bound to the data in the moment of their creation.

I think the basic pattern which we have seen here is that other programming languages can easily replace simple macros both in convinience and function. Also most of the singular uses of macros arent really that useful, rare or can be implemented with other constructs. I think the mainpower of macros is a combined use of lexical context, conditional expression and toplevel binding. The a-if macro demonstrates this: It first checks if the argument is nil (conditional evaluation) and if it is true it bind the symbol "it" while a function body is executed (lexical context). The Defspell from the textbased adventure use a combination of toplevel-binding and conditional evaluation. It bind a new game-action to the toplevel, but only executes it if certain conditions are met.

That was a long post and still I dont understand macros and/or see a general usefullness, but at least it is a start.

I just think you are making too much assumptions without actually having used Lisp macros for yourself. Even if that does not get into your head now, macros will help you making new abstractions in a very friendly way. Functions can't create abstractions, but macros can. But you need to get used to them. If you don't learn to use macros and make some programs that use macros in a smart way, you won't see that. After learning you will probably miss them when you use another language.

Unfortunately for me, I can't force you to learn macros, it is your choice.

Destruct1 wrote:While technically every macro is just a search-and-replace texteditor, many example macros are used the same as c-Macros.
They are used to abbreviate syntax, but have no deeper functionality.

This just isn't the case but, as has been mentioned, it's difficult to demonstrate real-world use of a macro "in anger" without swamping you with code that addresses a complex problem. Trivial examples are enough to show the basic idea, but simply can't indicate the full power. Artificially restricting their use to simple examples, as you've done above, is a way of hobbling them in a way that prevents their full power from showing: you've effectively set them up for failure.
Lisp macros, unlike C macros, are fully-fledged lisp functions that are executed at compile-time - the really powerful applications of them, therefore, are often quite complex, and only make sense to somebody with a deep understanding of the problem domain.

As gugamilare noted, nobody can force you to learn macros. This isn't a big problem, as you can still accomplish plenty in Lisp without them. I suspect the "problem" here is that you simply haven't yet worked on a codebase that's both large enough and complex enough for a macro to make sense. If and when you do think "surely I could write a shorter programme that would generate this code for me," however, that'll be the time to take another look.

Destruct1, you're right - example you gave - construction of the new program code during runtime and evaluation - is actually more expressive (although syntactically more complicated) than macros. The advantage of macros is, as it is already said, they allow compilation. That is the essence. If you accept interpretation, you might be disappointed with macros, and your reaction is adequate. But for those who do not accept interpretation, the macros provide about as much of expressive power in metaprogramming paradigm one can have. And Common Lisp is dialect of Lisp specifically designed to be compiled. However, when this discussion is started, I'd like to ask Common Lisp programmers:

• to give more examples of the most expressive macros, available in publicly available code libraries, books or papers, no matter if these examples are complicated. I'd like to hear what you consider to be the best macros ever written.

• Avoidance of eval (don't use it if it is not necessary) is already mentioned. Many CL-ers accept that position. When and where this rule is mentioned first time or seriously discussed in published literature, i.e. books, articles?

Kazimir Majorinc wrote:Destruct1, you're right - example you gave - construction of the new program code during runtime and evaluation - is actually more expressive (although syntactically more complicated) than macros. The advantage of macros is, as it is already said, they allow compilation. That is the essence. If you accept interpretation, you might be disappointed with macros, and your reaction is adequate. But for those who do not accept interpretation, the macros provide about as much of expressive power in metaprogramming paradigm one can have. And Common Lisp is dialect of Lisp specifically designed to be compiled. However, when this discussion is started, I'd like to ask Common Lisp programmers:

• to give more examples of the most expressive macros, available in publicly available code libraries, books or papers, no matter if these examples are complicated. I'd like to hear what you consider to be the best macros ever written.

• Avoidance of eval (don't use it if it is not necessary) is already mentioned. Many CL-ers accept that position. When and where this rule is mentioned first time or seriously discussed in published literature, i.e. books, articles?

eval (or any program code that is created and executed at runtime) is bad for these reasons:
a) It is inefficient to hold a full interpreter ready. This is especially so for compiled languages like Lisp, but it is also bad for interpreted languages like python and java who normally compile to bytecode. Lisp macros are
b) You open yourself up to injection attacks. If you take user input and execute these statements the user may just use ("import system; system.commandline ("format c:") or something similar. Even if you catch modifications to the filesystem the user may still knock you out with 2^248716876.
c) It is hard to debug. Because code is string (or symbol) data until executed, it doesnt even catch simple syntax errors. Writing code in ("line1\nline2") syntax doesnt help either.

As I wrote the Python code to the SPEL adventure it became clear that exec statements (the python equivalent to eval) are not the way to go. It is very hard to do. Someone in this thread wrote that " you need access to the parse tree to manipulate code". I agree. While it may be fine to do simple things with eval/exec it is a bad hack and hard to do for complicated problems.
Lisp macros fully integrate these code manipulation/creation because it macroexpands before the compile process (thereby avoiding overhead) and it has direct access to the parse tree (which makes writing macros easier and more integrated).

So all in all Lisp implements macros in the best way. However my problem is that I cant find useful purposes.

Either they are simple textediting macros like the previous mentioned (f (body)) <-> (lambda (x y z) (body)). I reject these for two reasons: I think it is actually the job of the programming language to use short, easy to write and read syntax. Most languages modern languages do a good job.
The more important point is that I dont mind writing extra code as long as I dont need to think. In practice a programmer writes very few lines of code (I heard 10 lines per day in big projects) and spends most of his time reading and thinking, so the time saved by typing "(dolist (e 4) .." instead of "(awesomemacro ..)" is negligible. What is important is time saved by not thinking. A good example is a function which returns the distance between a 2d point and (0/0). If I write this code in Python it looks like this:

def distance (point):
  return (sqrt (point.x**2+point.y**2))

Could that be written shorter? Of course. But it doesnt matter because this piece of code is close to my stream of thinking: ".. Alright i need a function which accepts a point (I write "def distance (point)")... hmm.. I can return immediatly ... (write "return")... and just use high school formular (writes rest).... end function... on to the main problem..."
If I write this code in C it looks like this:

double distance (cPoint inputpoint)
{
  return (sqrt (((double) inputpoint.x)^2+((double) inputpoint.y)^2)))
} 

This time the programming language interrupts my thinking: " ... alright I need a function... (interrupt) should return a float not an integer ... (interrupt) and i need max precision so i better use a double.. (write "double distance cPoint inputpoint) ... now on to the body... i can return immediatly (write "return") ... and I just use highschool formula (write "sqrt (inputpoint.x^2+inputpointy^2)" ... (interrupt) I am not sure which dataformat cInput uses though ... (interrupt) also they might give me a high number and i need a double to avoid overflow ... (interrupt) so i better cast the coordinates before the calculation ... (writes "(double) inputpoint.x" and "(double) inputpoint.y" ... now i only need to match the bunch of parenthesis .... on to the main function
In this case the programming language forced me to center my thinking on technicalities (casts, maxrange for numbers and so on) and away from the problem. That is bad. But it isnt bad to write extra code as long as this code is close to your stream of thinking. A example where short code is irrelevant is the (f (body)) <-> (lambda (x y z) (body)) mentioned previously in the thread.
If my thinking is "... I need a function here... (write "def functionname") and it has three parameters... the first two inputs are lists (write "lsta, lstb") ... the third is the necessary operation (write "operation") ... now the body of the function ..." that is fine. I think about the necessary input to my function and write code as I think about it.
The f-bound-with-x-y-z macro isnt helpful here. I still need to think about the necessary parameters and it creates problem later in the function body when I need to map the objects to x,y,z instead of more descriptive names like lsta, lstb, operation.

Writing short code is nice, but what is important to me is the ability to express my thinking in direct code and not care about the computer or underlying structures.

And the most powerful macro I have seen so far is the gaming-action macro in the textbased adventure. It gave me the most stuff to think about but it still didnt convince me because it is very esoteric (a macro writing macro) and I found a good Python hack (altough I didnt post this hack) to implement the macro functionality nice and easy.

Destruct1 wrote:If I write this code in Python it looks like this:

That is still missing the point. In Python, you only have the syntax/semantics given to you by the language designers. Since Python is decades younger than Lisp, it obviously already includes most of the common cases, just like reasonably modern dialects of Lisp like Common Lisp or Scheme contain most of the common cases in preexisting macros.

Where Lisp is superior is in uncommon cases, which are hard to show as an example since it is not even that they themselves are complicated, but require a lot of context, where you can add your own syntax/semantics without writing a complete new language. All your arguments could apply similarly to statement "there is no purpose to Python since everything it does can be done the same or better in C". The reasons for existence of macros are the same as for existence of Python, some problem domain are better expressed in a specialized language which pushes accidental complexity to the background.

The only, although major, difference is that macros lower the barrier to entry for language modifications, making it possible to extend the Lisp language just enough to fit the problem domain without creating an entirely new language. The need for this doesn't appear if your problem domain is either simple or already well fitted by existing languages.

Destruct1 wrote:The more important point is that I dont mind writing extra code as long as I dont need to think.

This is the point at which I channel the late Erik Naggum and politely suggest that Lisp probably isn't for you.
The value of Lisp (macros, first-class functions and all the rest) is precisely in investing effort in thinking to avoid writing extra code.

Destruct1 wrote:A good example is a function which returns the distance between a 2d point and (0/0). If I write this code in Python it looks like this:

def distance (point):
  return (sqrt (point.x**2+point.y**2))

Could that be written shorter? Of course.

Indeed, this is a good example, but in a different way than you might think. The real question is whether this is exactly the same as having to write:

def distance (point):
  return (sqrt (add(raise(point.x, 2), raise(point.y, 2))))

After all, what we have here is a domain-specific language for arithmetic expressions. In such a simple example, the disadvantage of not being able to use a domain-specific language is indeed negligible (as can be seen in the example, which I can read just fine when written using function calls), but the difference that an embedded DSL makes grows with the complexity of the problem, as can be easily seen when trying to write (and, more to the point, read!) more involved mathematical expressions.

Now, most languages nowadays happen to include a domain-specific language for arithmetic, but you can naturally extrapolate this issue to other domains. That's what macros are about: being able to embed DSLs for your own specific problem domain. Just like the special embedded arithmetic language provided by many languages by default, DSLs not only make your programmes more succinct, but also easier to understand and maintain.

All in all, I think it's pretty clear that embedded DSLs are useful. You may, of course, argue that (i) mathematical expressions are somehow a special case that can't be compared to any other kind of domain language, or (ii) macros are not needed to implement DSLs.

I don't think (i) is a feasible proposition. This leaves us with proposition (ii). As shown above (-> arithmetic), functions just don't cut it for DSL embedding. So what does? EVAL certainly does, but it's a pretty ugly solution. FEXPRs do, but they're terribly hard to implement efficiently and specify sanely. Anyway, all of these alternatives are pretty much in the same ballpark as macros, so what do you suggest?

Do you really think Python can embed DSLs as well as Lisp can?

Or is it that you question the usefulness of DSLs in general? Do you believe in proposition (i) above? If either of these apply, JamesF is probably on the mark: Lisp may not be for you, after all.

DSL are important and arithmetic operations are not special. However it is possible to implement this kind of
behavior in Python (and other languages) with operator overloading.

from math import sqrt

class Point ():
    def __init__ (self, x_=0, y_=0):
        self.x = x_
        self.y = y_
           
    def __getattr__ (self, attrib):
        if (attrib == "dist"):
            return (sqrt (self.x**2+self.y**2))
        else:
            raise AttributeError
    
    def __repr__ (self):
        return ("{}, {}".format (self.x, self.y))    
        
    def __add__ (self,other):
        return (Point (self.x+other.x, self.y+other.y))
    
    def __sub__ (self, other):
        return (Point (self.x-other.x, self.y-other.y))

This code implements a functional 2D-point class. Here is an example session from the REPL:

>>> a = Point (5,3)  ; creates a Point object with the given coordinates
>>> a.dist               ; returns the distance from (0/0)
5.830951894845301
>>> a                      ; calls the standard representation of the object
5, 3
>>> b = Point (9,3)
>>> a+b                  ; arithmetic operations are possible
14, 6
>>> (a+b).dist         ; complex expressions are possible
15.231546211727817

This implements a DSL language although not in the Lisp way.

Lisp generally uses data (organised in classes, lists or structures) and applies "verbs" (macros, methods or functions) on them.
A good example is CLOS where the classes only have slotvalues and no inherent functions. The functions come from the outside
and get dispatched depending on the datatype of the argument. If you have abstract data and functions which take these abstract datatypes as arguments
macros come in handy. This is because conditional evaluation and contextmanagement (two big field of macros) are important when dealing with
abstract data. You need to figure out the type and apply certain methods depending on what you just found out.

All other languages have a different concept. They integrate functions with the data. Classdefinitions have data (members) and functions which are associated with the data (methods). You dont need the conditional evaluation of lisp macros if your interpreter figures out the type and calls the appropriate functions on it.

Destruct1 wrote:DSL are important and arithmetic operations are not special. However it is possible to implement this kind of
behavior in Python (and other languages) with operator overloading.

Unfortunately, operator overloading gives you a limited vocabulary (search the Boost devel list for several debates over operator connotation, or justifying an unusual choice due to precedence rules). Furthermore, operator overloading and other function-based approaches cannot perform code refactoring that requires changing evaluation rules, introducing variables, etc. (unless you're wielding monads in an ML variant). C++ has made great strides using templates for metaprogramming; but I dare you to say templates are easier to comprehend than macros. The Boost Preprocessor library would not be necessary if templates were as powerful as macros.

If you want to truly bask in the true glory of lisp and macros you should read
http://www.paulgraham.com/onlisptext.html

And then once you think youve seen it all you can take a good look at continuatiosn:
http://common-lisp.net/project/cl-cont/, partial evaluators (hu.dwim.partial-eval), code walkers (hu.dwim.walker), and customizable reader macros (hu.dwim.reader):

All of which (well not the continuations) expands the power of macros! Hail to the macros!

OK, i think paul graham does a better job of selling lisp in the first few pages of on lisp