I dont get Macros

Discussion of Common Lisp
slobodan.blazeski
Posts: 9
Joined: Tue May 26, 2009 8:27 am

Re: I dont get Macros

Post by slobodan.blazeski » Wed Feb 03, 2010 1:17 pm

Destruct1 wrote: Can somebody show me either
a) examples where macros are desperatly needed
b) some patterns where macros are used
c) explain how to build an embedded language on top of lisp like I heard so many times
Example 1 :
Ever heard of Paul Graham new language Arc. It has a very nice conditional operator that works something like:

Code: Select all

(if  condition1 result1
     condition2 result2
    result3)
which is roughly like common lisp cond:

Code: Select all

(cond (condition1 result1)
         (condition2 result2)
         (t result3))
Arc if is very handy when there is only one result statement after each conditional. In a common lisp without macros
we would be doomed to write all those unnecessary parenthesis for eternity or switch to arc. But we have defmacro
so we have :

Code: Select all

(defun pair-them (x)
  (cond ((null x) nil)
        ((endp (cdr x))
         (cons (list t (car x))
               (pair-them (cddr x))))
        (t
         (cons (list (car x) (cadr x))
               (pair-them (cddr x))))))
;;; Grahams' if
(defmacro gif (&rest args)
  `(cond ,@(pair-them args)))

Few tests:

(pprint
  (macroexpand-1
   '(gif (= a 1) (setq a 2)
         (= a 2) (setq a 3)
         (= a a) (floor a 3))))
expands into 
(COND ((= A 1) (SETQ A 2))
      ((= A 2) (SETQ A 3))
      ((= A A) (FLOOR A 3)))


(pprint
  (macroexpand-1
   '(gif (= a 1) (setq a 2)
         (= a 2) (setq a 3)
         (floor a 3))))
expands into :
(COND ((= A 1) (SETQ A 2))
      ((= A 2) (SETQ A 3))
      (T (FLOOR A 3)))

Example 2:
q has a very nice lambda like utility that allows programmer to omit parameter list assuming that first 3 parameters are named x y and z

Code: Select all

(f - x y z) ; => (lambda (x y z) (- x y z)) 
(f - z y x) <=> (LAMBDA (X Y Z) (- Z Y X)) 
(f list  x (list y "bobi")  (+ y 2) z) <=> (LAMBDA (X Y Z) (LIST X  (LIST Y "bobi") (+ Y 2) Z))
how would you create this utility using functions? With macros its a piece of cake:

Code: Select all

(defmacro f (&rest args) 
 `(lambda ,(remove-if-not (lambda (x) (member x (flatten args))) 
                          '(x y z)) ,args))
And those are just simple macros which saved me a lot of typing and brought me many golfing victories.
Macros basically allows you to capture the patterns in the code that can't be captured with plain functions. Sure you could type
everything by hand but that is boring and error prone. For a good explanations of macros try On Lisp and Let Over Lambda
after you finish on lisp. Paul Graham actually starts creating abstractions using functions in
the first chapters of On Lisp the he switches to more luxurious vehicles, the mighty macros.

happy lisping
Slobodan Blazeski

Destruct1
Posts: 20
Joined: Wed Jan 20, 2010 1:40 am

Re: I dont get Macros

Post by Destruct1 » Wed Feb 10, 2010 4:10 am

Lisp Macros have the fullowing use:
1) Basic text Search-and-Replace
2) Building a lexical context
3) SETF Accessor
4) Conditional Evaluation
5) Binding things to the TopLevel


While technically every macro is just a search-and-replace texteditor, many example macros are used the same as c-Macros.
They are used to abbreviate syntax, but have no deeper functionality. Example of this include arc = for setf or syntatic sugar to avoid typing ' or the f function or the graham if. Another example would be to change the first,second, third etc. buildin function to n1 n2 n3 etc.
I think these macros are kind a useless. First you often solve a problem you only have in lisp. Most languages dont require to quote data or use kinda long names. While that may save you a few letters I dont think that is the point. Programming languages should represent deeper and more abstract concepts and not try to make their code especially short from a pure letter standpoint. Otherwise you might end up with a unreadable language like Perl.
For example the Graham-if can be written in Python:

Code: Select all

if condition1:
    action1
elif condition2:
    action2
etc.
While this code has the useless "if" and ":" it isnt inconvinient to write. I actually like the clear structure of the python code compared to the g-if where the semantic is dependent on the position in the code. I dont want to count if a certain codeblock is in 5th or 6th position and therefore is a conditional or an action. And I expecially dont want to match the parenthisis in my head to decipher if codeblock X is the last thing in the block (and therefore the default-action) or actually just the 7. but not last thing in the g-if (and therefore the condition3). It doesnt matter how much typing you do, if the cognitive overhead is small and you gain readability.

Another point is that these macros dont use the unique "code is data" and "direct access to the parse tree" Lisp is famous for. They can be imitated by such simple things like C macros or text autocompletion.

The (f body) -> (lambda (x y z) body) is neat and good (especially in golfing tournaments) but can be textcompleted easily.

The third point on the list seems to be the Lisp way of creating setter functions.

Lets get to the second and fourth point where I changed my mind about Lisp macros.
I think the crucial point here is that these macros really need a body or complex expression to work out. If you have a simple expression that evaluates to a datastructure (list, string, number, struct etc.) it is possible to replace every macro with a function. Think about it: If a macro/function takes x arguments, which all evaluate themselves, and maps this input to an output, it is impossible to gain any advanatge with a macro. Mapping data inputs to data outputs is the basic definition of a function. Even sideeffects just expand this definition. A function takes several data inputs, produces sideeffects and returns data outputs. (Note: You might gain performance with macros instead of functions)

On the other hand macros might be useful if at least one part of the macro is a complex expression or body, a "code input". To make things simple on my already stressed mind I go with exactly one body/code argument. While technical it is possible to take 2 or more code inputs I think it strains the human mind to do so. So, that leaves us with a bunch of data arguments like strings, numbers or lists and one code block which is passed to the macro as raw, code input and is declared the &body argument.

Now we have the following options:

We create a lexical enviroment in which the codeblock is executed. Simple examples include the with-open-file macro. In this macro a string and a symbol is passed as data argument and a code block is executed with a file-access as lexical enviroment. The code can use the enviroment provided by the macro to access the file.
Unfortunatly to be really useful such a macro must provide additional functionality. If the macro only provides a binding of a certain variable it is useless because other languages can just use assignment to imitate the Lisp macro. A (witha 7 body) macro that just binds the number 7 to the variable a is useless because a=7;{codeblock} has the same effect. The witha macro is just a let macro with a as the default parameter.
The let macro has the power to bind a variable only for a certain amount of time (the scope of the let/macro) but other programming languages dont seem to miss that possibility. Instad they just define a new variable in the local scope and forget the variable as soon as the scope changes. So the Let macro as the basic context creating macro is just the Lisp way of assining variables.
But the context creating macros have more possibilities than standard assignment. For one they can execute enter and exit code. So a file/database/something can be opened/locked/entered, the codeblock executed and the needed close/unlock/cleanup done automatically.
I am not sure how useful lexical binding really are. Complex assignments can be substitutes with a=function (datainput);{codeblock}. The enter/exit trick can be emulated in Python with the "with x as variable:" construct.

The other option is to use conditional evaluation. And again other languages can substitute basic use of conditional evaluation with their if/case statements or similar control structures. Multiple datainput or codeinput doesnt change the ability of other languages to keep up. Even if a macro in Lisp can combine the datainputs to a complex predicate, other languages can use if (function (datainput1, datainpu2,etc.)==True do Codeinput1. The only interesting uses of this macro-class is in combination with other macro-classes.

The last point is binding things to the TopLevel. A macro can define a function (or other macro) at the top level. And again it is trivially easy for other languages to keep up with the possibilities of Lisp if tasks are easy. a=something binds a in the toplevel, b=lambda x:2*x binds a function to the toplevel. But while other languages need an explicit assignment, Lisp can bind symbols which are determined during runtime. A good example which illustrates this point is this one: You have an external configuration file with the structure { Database1="Hello.dat";Database2="DataisHere.dat";DataZum="DatRer.dat" } and want to bind access functions to all these databases to the top level. It is possible to create a Lisp macro that defines the functions Database1get, Database2get, DataZumget all in one go. Other languages at least need to write Database1get = funcfromconfigfile (1), Database2get=funcfromconfigfile(2), DataZumget=funcfromconfigfile (3). Another example which is illustrated in "Lisp: A gentle introduction" is a finite state automaton where the nodes are represented by closures. These closures/node fucntions are bound to the top level and can be easily accessed. So while other programming languages need to write programms which access data, Lisp can write functions which are bound to the data since their creation.
There are Python hacks with the exec command that can simulate this toplevel binding but they arent that well integrated into the language. The "Right Way (tm)" to implement the above example would be to load the configuration file into a classobject, write a generic access function which takes a databasename / number as input instaead of defining functions only for a certain database. This means writing function which access data instead of writing functions which are bound to the data in the moment of their creation.

I think the basic pattern which we have seen here is that other programming languages can easily replace simple macros both in convinience and function. Also most of the singular uses of macros arent really that useful, rare or can be implemented with other constructs. I think the mainpower of macros is a combined use of lexical context, conditional expression and toplevel binding. The a-if macro demonstrates this: It first checks if the argument is nil (conditional evaluation) and if it is true it bind the symbol "it" while a function body is executed (lexical context). The Defspell from the textbased adventure use a combination of toplevel-binding and conditional evaluation. It bind a new game-action to the toplevel, but only executes it if certain conditions are met.

That was a long post and still I dont understand macros and/or see a general usefullness, but at least it is a start.

gugamilare
Posts: 406
Joined: Sat Mar 07, 2009 6:17 pm
Location: Brazil
Contact:

Re: I dont get Macros

Post by gugamilare » Wed Feb 10, 2010 5:19 am

I just think you are making too much assumptions without actually having used Lisp macros for yourself. Even if that does not get into your head now, macros will help you making new abstractions in a very friendly way. Functions can't create abstractions, but macros can. But you need to get used to them. If you don't learn to use macros and make some programs that use macros in a smart way, you won't see that. After learning you will probably miss them when you use another language.

Unfortunately for me, I can't force you to learn macros, it is your choice.

JamesF
Posts: 98
Joined: Thu Jul 10, 2008 7:14 pm

Re: I dont get Macros

Post by JamesF » Wed Feb 10, 2010 6:19 pm

Destruct1 wrote:While technically every macro is just a search-and-replace texteditor, many example macros are used the same as c-Macros.
They are used to abbreviate syntax, but have no deeper functionality.
This just isn't the case but, as has been mentioned, it's difficult to demonstrate real-world use of a macro "in anger" without swamping you with code that addresses a complex problem. Trivial examples are enough to show the basic idea, but simply can't indicate the full power. Artificially restricting their use to simple examples, as you've done above, is a way of hobbling them in a way that prevents their full power from showing: you've effectively set them up for failure.
Lisp macros, unlike C macros, are fully-fledged lisp functions that are executed at compile-time - the really powerful applications of them, therefore, are often quite complex, and only make sense to somebody with a deep understanding of the problem domain.

As gugamilare noted, nobody can force you to learn macros. This isn't a big problem, as you can still accomplish plenty in Lisp without them. I suspect the "problem" here is that you simply haven't yet worked on a codebase that's both large enough and complex enough for a macro to make sense. If and when you do think "surely I could write a shorter programme that would generate this code for me," however, that'll be the time to take another look.

Kazimir Majorinc
Posts: 2
Joined: Mon Jan 26, 2009 3:56 pm
Location: Zagreb, Croatia
Contact:

Re: I dont get Macros

Post by Kazimir Majorinc » Sat Feb 13, 2010 9:05 am

Destruct1, you're right - example you gave - construction of the new program code during runtime and evaluation - is actually more expressive (although syntactically more complicated) than macros. The advantage of macros is, as it is already said, they allow compilation. That is the essence. If you accept interpretation, you might be disappointed with macros, and your reaction is adequate. But for those who do not accept interpretation, the macros provide about as much of expressive power in metaprogramming paradigm one can have. And Common Lisp is dialect of Lisp specifically designed to be compiled. However, when this discussion is started, I'd like to ask Common Lisp programmers:
  • to give more examples of the most expressive macros, available in publicly available code libraries, books or papers, no matter if these examples are complicated. I'd like to hear what you consider to be the best macros ever written.
  • Avoidance of eval (don't use it if it is not necessary) is already mentioned. Many CL-ers accept that position. When and where this rule is mentioned first time or seriously discussed in published literature, i.e. books, articles?

Destruct1
Posts: 20
Joined: Wed Jan 20, 2010 1:40 am

Re: I dont get Macros

Post by Destruct1 » Mon Feb 15, 2010 4:08 am

Kazimir Majorinc wrote:Destruct1, you're right - example you gave - construction of the new program code during runtime and evaluation - is actually more expressive (although syntactically more complicated) than macros. The advantage of macros is, as it is already said, they allow compilation. That is the essence. If you accept interpretation, you might be disappointed with macros, and your reaction is adequate. But for those who do not accept interpretation, the macros provide about as much of expressive power in metaprogramming paradigm one can have. And Common Lisp is dialect of Lisp specifically designed to be compiled. However, when this discussion is started, I'd like to ask Common Lisp programmers:
  • to give more examples of the most expressive macros, available in publicly available code libraries, books or papers, no matter if these examples are complicated. I'd like to hear what you consider to be the best macros ever written.
  • Avoidance of eval (don't use it if it is not necessary) is already mentioned. Many CL-ers accept that position. When and where this rule is mentioned first time or seriously discussed in published literature, i.e. books, articles?
eval (or any program code that is created and executed at runtime) is bad for these reasons:
a) It is inefficient to hold a full interpreter ready. This is especially so for compiled languages like Lisp, but it is also bad for interpreted languages like python and java who normally compile to bytecode. Lisp macros are
b) You open yourself up to injection attacks. If you take user input and execute these statements the user may just use ("import system; system.commandline ("format c:") or something similar. Even if you catch modifications to the filesystem the user may still knock you out with 2^248716876.
c) It is hard to debug. Because code is string (or symbol) data until executed, it doesnt even catch simple syntax errors. Writing code in ("line1\nline2") syntax doesnt help either.

As I wrote the Python code to the SPEL adventure it became clear that exec statements (the python equivalent to eval) are not the way to go. It is very hard to do. Someone in this thread wrote that " you need access to the parse tree to manipulate code". I agree. While it may be fine to do simple things with eval/exec it is a bad hack and hard to do for complicated problems.
Lisp macros fully integrate these code manipulation/creation because it macroexpands before the compile process (thereby avoiding overhead) and it has direct access to the parse tree (which makes writing macros easier and more integrated).

So all in all Lisp implements macros in the best way. However my problem is that I cant find useful purposes.

Either they are simple textediting macros like the previous mentioned (f (body)) <-> (lambda (x y z) (body)). I reject these for two reasons: I think it is actually the job of the programming language to use short, easy to write and read syntax. Most languages modern languages do a good job.
The more important point is that I dont mind writing extra code as long as I dont need to think. In practice a programmer writes very few lines of code (I heard 10 lines per day in big projects) and spends most of his time reading and thinking, so the time saved by typing "(dolist (e 4) .." instead of "(awesomemacro ..)" is negligible. What is important is time saved by not thinking. A good example is a function which returns the distance between a 2d point and (0/0). If I write this code in Python it looks like this:

Code: Select all

def distance (point):
  return (sqrt (point.x**2+point.y**2))
Could that be written shorter? Of course. But it doesnt matter because this piece of code is close to my stream of thinking: ".. Alright i need a function which accepts a point (I write "def distance (point)")... hmm.. I can return immediatly ... (write "return")... and just use high school formular (writes rest).... end function... on to the main problem..."
If I write this code in C it looks like this:

Code: Select all

double distance (cPoint inputpoint)
{
  return (sqrt (((double) inputpoint.x)^2+((double) inputpoint.y)^2)))
} 
This time the programming language interrupts my thinking: " ... alright I need a function... (interrupt) should return a float not an integer ... (interrupt) and i need max precision so i better use a double.. (write "double distance cPoint inputpoint) ... now on to the body... i can return immediatly (write "return") ... and I just use highschool formula (write "sqrt (inputpoint.x^2+inputpointy^2)" ... (interrupt) I am not sure which dataformat cInput uses though ... (interrupt) also they might give me a high number and i need a double to avoid overflow ... (interrupt) so i better cast the coordinates before the calculation ... (writes "(double) inputpoint.x" and "(double) inputpoint.y" ... now i only need to match the bunch of parenthesis .... on to the main function
In this case the programming language forced me to center my thinking on technicalities (casts, maxrange for numbers and so on) and away from the problem. That is bad. But it isnt bad to write extra code as long as this code is close to your stream of thinking. A example where short code is irrelevant is the (f (body)) <-> (lambda (x y z) (body)) mentioned previously in the thread.
If my thinking is "... I need a function here... (write "def functionname") and it has three parameters... the first two inputs are lists (write "lsta, lstb") ... the third is the necessary operation (write "operation") ... now the body of the function ..." that is fine. I think about the necessary input to my function and write code as I think about it.
The f-bound-with-x-y-z macro isnt helpful here. I still need to think about the necessary parameters and it creates problem later in the function body when I need to map the objects to x,y,z instead of more descriptive names like lsta, lstb, operation.

Writing short code is nice, but what is important to me is the ability to express my thinking in direct code and not care about the computer or underlying structures.

And the most powerful macro I have seen so far is the gaming-action macro in the textbased adventure. It gave me the most stuff to think about but it still didnt convince me because it is very esoteric (a macro writing macro) and I found a good Python hack (altough I didnt post this hack) to implement the macro functionality nice and easy.

ramarren
Posts: 613
Joined: Sun Jun 29, 2008 4:02 am
Location: Warsaw, Poland
Contact:

Re: I dont get Macros

Post by ramarren » Mon Feb 15, 2010 4:27 am

Destruct1 wrote:If I write this code in Python it looks like this:
That is still missing the point. In Python, you only have the syntax/semantics given to you by the language designers. Since Python is decades younger than Lisp, it obviously already includes most of the common cases, just like reasonably modern dialects of Lisp like Common Lisp or Scheme contain most of the common cases in preexisting macros.

Where Lisp is superior is in uncommon cases, which are hard to show as an example since it is not even that they themselves are complicated, but require a lot of context, where you can add your own syntax/semantics without writing a complete new language. All your arguments could apply similarly to statement "there is no purpose to Python since everything it does can be done the same or better in C". The reasons for existence of macros are the same as for existence of Python, some problem domain are better expressed in a specialized language which pushes accidental complexity to the background.

The only, although major, difference is that macros lower the barrier to entry for language modifications, making it possible to extend the Lisp language just enough to fit the problem domain without creating an entirely new language. The need for this doesn't appear if your problem domain is either simple or already well fitted by existing languages.

JamesF
Posts: 98
Joined: Thu Jul 10, 2008 7:14 pm

Re: I dont get Macros

Post by JamesF » Mon Feb 15, 2010 4:50 am

Destruct1 wrote:The more important point is that I dont mind writing extra code as long as I dont need to think.
This is the point at which I channel the late Erik Naggum and politely suggest that Lisp probably isn't for you.
The value of Lisp (macros, first-class functions and all the rest) is precisely in investing effort in thinking to avoid writing extra code.

Kompottkin
Posts: 94
Joined: Mon Jul 21, 2008 7:26 am
Location: München, Germany
Contact:

Re: I dont get Macros

Post by Kompottkin » Mon Feb 15, 2010 9:59 am

Destruct1 wrote:A good example is a function which returns the distance between a 2d point and (0/0). If I write this code in Python it looks like this:

Code: Select all

def distance (point):
  return (sqrt (point.x**2+point.y**2))
Could that be written shorter? Of course.
Indeed, this is a good example, but in a different way than you might think. The real question is whether this is exactly the same as having to write:

Code: Select all

def distance (point):
  return (sqrt (add(raise(point.x, 2), raise(point.y, 2))))
After all, what we have here is a domain-specific language for arithmetic expressions. In such a simple example, the disadvantage of not being able to use a domain-specific language is indeed negligible (as can be seen in the example, which I can read just fine when written using function calls), but the difference that an embedded DSL makes grows with the complexity of the problem, as can be easily seen when trying to write (and, more to the point, read!) more involved mathematical expressions.

Now, most languages nowadays happen to include a domain-specific language for arithmetic, but you can naturally extrapolate this issue to other domains. That's what macros are about: being able to embed DSLs for your own specific problem domain. Just like the special embedded arithmetic language provided by many languages by default, DSLs not only make your programmes more succinct, but also easier to understand and maintain.

All in all, I think it's pretty clear that embedded DSLs are useful. You may, of course, argue that (i) mathematical expressions are somehow a special case that can't be compared to any other kind of domain language, or (ii) macros are not needed to implement DSLs.

I don't think (i) is a feasible proposition. This leaves us with proposition (ii). As shown above (-> arithmetic), functions just don't cut it for DSL embedding. So what does? EVAL certainly does, but it's a pretty ugly solution. FEXPRs do, but they're terribly hard to implement efficiently and specify sanely. Anyway, all of these alternatives are pretty much in the same ballpark as macros, so what do you suggest?

Do you really think Python can embed DSLs as well as Lisp can?

Or is it that you question the usefulness of DSLs in general? Do you believe in proposition (i) above? If either of these apply, JamesF is probably on the mark: Lisp may not be for you, after all.

Destruct1
Posts: 20
Joined: Wed Jan 20, 2010 1:40 am

Re: I dont get Macros

Post by Destruct1 » Tue Feb 16, 2010 3:34 am

DSL are important and arithmetic operations are not special. However it is possible to implement this kind of
behavior in Python (and other languages) with operator overloading.

Code: Select all

from math import sqrt

class Point ():
    def __init__ (self, x_=0, y_=0):
        self.x = x_
        self.y = y_
           
    def __getattr__ (self, attrib):
        if (attrib == "dist"):
            return (sqrt (self.x**2+self.y**2))
        else:
            raise AttributeError
    
    def __repr__ (self):
        return ("{}, {}".format (self.x, self.y))    
        
    def __add__ (self,other):
        return (Point (self.x+other.x, self.y+other.y))
    
    def __sub__ (self, other):
        return (Point (self.x-other.x, self.y-other.y))
This code implements a functional 2D-point class. Here is an example session from the REPL:

Code: Select all

>>> a = Point (5,3)  ; creates a Point object with the given coordinates
>>> a.dist               ; returns the distance from (0/0)
5.830951894845301
>>> a                      ; calls the standard representation of the object
5, 3
>>> b = Point (9,3)
>>> a+b                  ; arithmetic operations are possible
14, 6
>>> (a+b).dist         ; complex expressions are possible
15.231546211727817
This implements a DSL language although not in the Lisp way.

Lisp generally uses data (organised in classes, lists or structures) and applies "verbs" (macros, methods or functions) on them.
A good example is CLOS where the classes only have slotvalues and no inherent functions. The functions come from the outside
and get dispatched depending on the datatype of the argument. If you have abstract data and functions which take these abstract datatypes as arguments
macros come in handy. This is because conditional evaluation and contextmanagement (two big field of macros) are important when dealing with
abstract data. You need to figure out the type and apply certain methods depending on what you just found out.

All other languages have a different concept. They integrate functions with the data. Classdefinitions have data (members) and functions which are associated with the data (methods). You dont need the conditional evaluation of lisp macros if your interpreter figures out the type and calls the appropriate functions on it.

Post Reply