Macro expansion, packages, and LET

Hello, friendly and knowledgeable fellow Lispers. I'm trying to write a little library to provide charting capabilities over Vecto. For my first task, I'm trying to mimic Vecto's method of drawing images but at a slightly higher level (for the user). So I've provided a WITH-PIE-CHART macro that the user would utilize like this:

(taylor-chart:with-pie-chart ("test.png" 256)
  (add-slice 25)
  (add-slice 50)
  (add-slice 75)
  (add-slice 100))

As you can see, all of this is inside a package TAYLOR-CHART. My current definition of WITH-PIE-CHART has some flaws:

(defpackage :taylor-chart
  (:use :common-lisp :vecto)
  (:export :with-pie-chart))

(in-package :taylor-chart)

(defmacro with-pie-chart ((file radius) &rest body)
  `(let ((slice-pieces nil)
         (slice-spacing 0))
     (flet ((add-slice (value &optional caption color)
              (setf slice-pieces (cons value slice-pieces))))
       (with-canvas (:width ,radius :height ,radius)
         ,@body
         (format t "~a~%" slice-pieces)
         (translate (/ ,radius 2) (/ ,radius 2))
         (setf slice-pieces (nreverse slice-pieces))
         (let ((idx 0) (sum (reduce #'+ slice-pieces))
               (len (length slice-pieces)))
           (reduce (lambda (x y)
                     (let ((z (* (/ y sum) 2 pi)))
                       (incf idx)
                       (format t "Drawing arc: ~a ~a~%" x (+ x z))
                       (set-rgb-fill (/ idx len) (/ idx len) (/ idx len))
                       (move-to 0 0)
                       (arc 0 0 128 x (+ x z))
                       (fill-and-stroke)
                       (+ x z))) slice-pieces :initial-value 0)
           (save-png ,file))))))

Namely, the bindings created by LET and FLET end up in the TAYLOR-CHART package, where they cannot be found by the user. The expansion is:

(LET ((TAYLOR-CHART::SLICE-PIECES NIL) (TAYLOR-CHART::SLICE-SPACING 0))
  (FLET ((TAYLOR-CHART::ADD-SLICE
             (TAYLOR-CHART::VALUE
              &OPTIONAL TAYLOR-CHART::CAPTION TAYLOR-CHART::COLOR)
           (SETF TAYLOR-CHART::SLICE-PIECES
                   (CONS TAYLOR-CHART::VALUE TAYLOR-CHART::SLICE-PIECES))))
    (VECTO:WITH-CANVAS (:WIDTH 256 :HEIGHT 256) (ADD-SLICE 25) (ADD-SLICE 50)
                       (ADD-SLICE 75) (ADD-SLICE 100)
                       (FORMAT T "~a~%" TAYLOR-CHART::SLICE-PIECES)
                       (VECTO:TRANSLATE (/ 256 2) (/ 256 2))
                       (SETF TAYLOR-CHART::SLICE-PIECES
                               (NREVERSE TAYLOR-CHART::SLICE-PIECES))
                       (LET ((TAYLOR-CHART::IDX 0)
                             (TAYLOR-CHART::SUM
                              (REDUCE #'+ TAYLOR-CHART::SLICE-PIECES))
                             (TAYLOR-CHART::LEN
                              (LENGTH TAYLOR-CHART::SLICE-PIECES)))
                         (REDUCE
                          (LAMBDA (TAYLOR-CHART::X TAYLOR-CHART::Y)
                            (LET ((TAYLOR-CHART::Z
                                   (* (/ TAYLOR-CHART::Y TAYLOR-CHART::SUM) 2
                                      PI)))
                              (INCF TAYLOR-CHART::IDX)
                              (FORMAT T "Drawing arc: ~a ~a~%" TAYLOR-CHART::X
                                      (+ TAYLOR-CHART::X TAYLOR-CHART::Z))
                              (VECTO:SET-RGB-FILL
                               (/ TAYLOR-CHART::IDX TAYLOR-CHART::LEN)
                               (/ TAYLOR-CHART::IDX TAYLOR-CHART::LEN)
                               (/ TAYLOR-CHART::IDX TAYLOR-CHART::LEN))
                              (VECTO:MOVE-TO 0 0)
                              (VECTO:ARC 0 0 128 TAYLOR-CHART::X
                                         (+ TAYLOR-CHART::X TAYLOR-CHART::Z))
                              (VECTO:FILL-AND-STROKE)
                              (+ TAYLOR-CHART::X TAYLOR-CHART::Z)))
                          TAYLOR-CHART::SLICE-PIECES :INITIAL-VALUE 0)
                         (VECTO:SAVE-PNG "test.png")))))

How can I avoid or work around this? When I move everything into the same package (for example by defining the macro in the CL-USER package, explicitly marking the Vecto functions, and trying my example usage from there), it works as expected.

Thanks for any hints or ideas.

Oh my, I suppose I should have realized that of course I should be referencing ADD-SLICE by the package in the code that the user would write. Sorry, programming late at night. Still, I find it a bit odd that I could not find mention anywhere of the accident of having stuff bound in a macro expansion end up in the package where the macro is defined rather than where it is being used. It wasn't immediately obvious to me anyway, so maybe somebody else will benefit from this thread in the future.

Macro generates a piece of AST that contains symbols. Symbol is an object named by its package and name. Macro returns a subtree with symbols in it. Symbols are produced by reader. Reader uses current package to intern them. So it is only logical that symbols end up in the same package with macro (because the decision where to put symbols is done at read-time, not at macroexpansion time).
It is possible to make a macro that returns symbols not from the macro's package but from user's package.
Like this:

(defmacro with-pie-chart ((file radius) &rest body)
  (let ((add-slice-symbol (intern "ADD-SLICE" *package*)))
    `(let ((slice-pieces nil)
	   (slice-spacing 0))
       (flet ((,add-slice-symbol (value &optional caption color)
		(setf slice-pieces (cons value slice-pieces))))
	 (with-canvas (:width ,radius :height ,radius)
	   ,@body
	   (format t "~a~%" slice-pieces)
	   (translate (/ ,radius 2) (/ ,radius 2))
	   (setf slice-pieces (nreverse slice-pieces))
	   (let ((idx 0) (sum (reduce #'+ slice-pieces))
		 (len (length slice-pieces)))
	     (reduce (lambda (x y)
		       (let ((z (* (/ y sum) 2 pi)))
			 (incf idx)
			 (format t "Drawing arc: ~a ~a~%" x (+ x z))
			 (set-rgb-fill (/ idx len) (/ idx len) (/ idx len))
			 (move-to 0 0)
			 (arc 0 0 128 x (+ x z))
			 (fill-and-stroke)
			 (+ x z))) slice-pieces :initial-value 0)
	     (save-png ,file)))))
    ))

We just look at the current package (current at the time of macroexpansion), intern a symbol with name "ADD-SLICE" into it, and the substitute the symbol into the macroexpanded code.

Why not just use a function?

(taylor-chart:pie-chart '(25 70 75 100) "test.png" :radius 256)

dmitry_vk:

Oh, I see. Thanks for the explanation, that makes sense.

death:

Eventually the charts will get more complicated, and I think it will be more useful for the user to have a little more flexibility in how they specify things. For example, I'm already starting to code for one of those donut / ring charts with multiple layers, and if the user wants to have three layers of five elements each, I don't want them to have to specify everything in a single big list straight up. I think it will also make it cleaner for them to specify background color, where the legend goes, the title, other labelling, etc. if I do it this way rather than using a single function with a lot of parameters.

dmitry_vk wrote:We just look at the current package (current at the time of macroexpansion), intern a symbol with name "ADD-SLICE" into it, and the substitute the symbol into the macroexpanded code.

That is fairly horrible: interning symbols into a package you don't own is almost always wrong. What if there is another definition (possibly originating in a third package!) of ADD-SLICE with a compiler macro? Result is that WITH-PIE-CHART will not work, and the user is not even told that the packages conflict!

The correct solution is to export all symbols that are part of your API. That way the user can have his own ADD-SLICE thingie, and still use TAYLOR-CHART:ADD-SLICE.

(defpackage :taylor-chart
  (:use :common-lisp :vecto)
  (:export #:with-pie-chart #:add-slice))

Cheers,

-- Nikodemus

taylor_venable wrote: Eventually the charts will get more complicated, and I think it will be more useful for the user to have a little more flexibility in how they specify things.

I still advise you to use functions. The primary role of macros is to provide for syntactic abstraction. The primary role of functions is to provide for procedural abstraction. Your code, which consists of one big macro, does very little for syntactic abstraction while bringing in lots of procedural abstraction. It's a common newbie mistake. I conjecture that, as you become more experienced in Lisp, you will become more aware of the need to separate concerns, and you will learn to avoid such muddled code. In fact, experienced Lisp programmers sometimes use the following strategy: first define a procedural interface using functions, and then, if so desired, implement a syntactic layer in terms of the procedural interface. I believe this strategy to be relevant and appropriate to your case.

taylor_venable wrote: For example, I'm already starting to code for one of those donut / ring charts with multiple layers, and if the user wants to have three layers of five elements each, I don't want them to have to specify everything in a single big list straight up.

They wouldn't need to. They would build the list as they go. It is just for chart output that they need the whole list, and the user could construct such a list in whatever fashion she fancies. If she wished to not output a chart after all, she could just not call the function (the user of your macro would have to do much more to express her change of mind). There's no reason you couldn't provide a procedural interface for incremental charting if you wanted to and besides, as you recognize, the functionality of adding slices (or data in general) is not unique to pie charts, so why should it be implemented in the definition of a WITH-PIE-CHART macro?

taylor_venable wrote: I think it will also make it cleaner for them to specify background color, where the legend goes, the title, other labelling, etc. if I do it this way rather than using a single function with a lot of parameters.

Key parameters can come a long way, and a full-fledged object to represent a chart is also something to think about. None of that requires you to jump head first into the macro pool. First come up with a good procedural interface, then add the sugar.

nikodemus:

Thanks for the advice; exporting the function (as well as moving the definition outside the macro) is what I decided on doing.

death:

I'll keep all that in mind, it sounds like great advice, but my thinking is that there are certain macros which establish a context in which the body operates; for example, WITH-OPEN-FILE establishes the context of the file in the sense that it opens the file, lets you do whatever you need with that file around, then cleans up for you at the end. I think that WITH-PIE-CHART fulfills basically the same profile: it creates a basic chart, lets you modify or add to it, then at the end when you're done it generates the result. I was thinking along the lines not only of convenience, but also in encapsulating the activity of chart-making. Nothing from a particular chart can "escape from" the WITH-PIE-CHART that defines it. True that you can still do it with a function that supports a lot of key arguments, but I think this fits more into the way such context problems have already been solved. Maybe I'm wrong about this? Anyway, it still makes sense to me, though I always like to hear other approaches.

taylor_venable wrote: I'll keep all that in mind, it sounds like great advice, but my thinking is that there are certain macros which establish a context in which the body operates; for example, WITH-OPEN-FILE establishes the context of the file in the sense that it opens the file, lets you do whatever you need with that file around, then cleans up for you at the end.

Yes, but note that WITH-OPEN-FILE uses OPEN to create the file stream. It doesn't create the file stream inline. It is implemented in terms of a procedural interface. Analogously, if you choose to have WITH-PIE-CHART, it should use a pie chart function. It shouldn't create the pie chart inline. In some cases, a programmer might want to keep the file stream for a while, and she would use OPEN directly. The analogous situation may happen with chart objects.

I'd also recommend looking more closely at the Vecto package that you're using/emulating/etc.

In particulary, the WITH-CANVAS macro sets the graphic state in a dynamic variable.
Then, all of the functions called within the body of the WITH-CANVAS use the *GRAPHIC-STATE*.

In your case, the WITH-PIE-CHART can be a macro that sets up the *CHART*. Then
your ADD-SLICE function can be a top-level, exported function which makes use of the *CHART*.
You can define functions for the user to query the current number of slices, etc. if you like.

ttyl,
Patrick