JSON Streaming API Design

[pjstirling]

In furtherence of my plans for global conquest, I've been pushing more intelligence into my hunchentoot-based webapp library.

I already had support for embedding the results of SQL queries as a (formatted) html table, handling, transparently, column sorting and pagination. I wanted to add support for javascript-enabled browsers to request data via AJAX, so as to not require a full page refresh (this becomes a bigger win when you have more than one such table on a page).

In order to support this I had to generate JSON to represent the table. There are a number of libraries in quicklisp that can generate JSON for you, but the ones I looked at initially expected you to provide data for conversion in the form of lisp data. The natural way of representing the table data is as an array-of-arrays, but that isn't the natural way to generate it; I stream the data from the database row-by-row, generating the strings representing the cell contents, I didn't really want to hang on to all of those strings in vectors, simply to throw them away at the end (the unformatted JSON for one of my test pages comes to 43kb, the in-memory layout would be even worse).

I opted to do the generation by-hand because this way I could stream the output as I streamed the input, but now I would like to reify that code into a library.

Consider the following example of the code I would like to be able to write:

Code: Select all

  (json-arr
    (dotimes (i 10)
      (json-val i))
    (json-arr
      (json-val 'hi-there)
      (json-val 20))))

Obviously JSON-ARR needs to be a macro (so that it can arrange for the opening bracket to be output before any of its contents). Less obviously it must internally redefine JSON-ARR, JSON-VAL and the missing-from-this-example JSON-OBJ so as to conditionally insert separating commas before performing their normal operation. A naive implementation of JSON-ARR might be:

Code: Select all

(defmacro json-arr (&body body)
  (bind ((:symbols seen))
    `(progn
       (write-string "[" *json-stream*)
       (let (,seen)
         (macrolet ((json-arr (&body body)
                      `(progn
                         (if ,',seen
                             (write-string ", " *json-stream*)
                             (setf ,',seen t))
                         (json-arr ,@body)))
                    ;; similar definitions for json-obj and json-val would also go here                                        
                    )
           ,@body))
       (write-string "]" *json-stream*))))

Unfortunately MACROLET behaves more like LABELS than FLET, with regard to expansion: the JSON-ARR inside the PROGN of the expansion also gets expanded with the same expander, and the compiler goes into infinite recursion.

If FLET wasn't in common-lisp you could imitate the effect by using two layers of LABELS and a gensym, where the outer layer uses the gensym to bind a function that calls the global function, and the inner LABELS redefines the named function, using the gensym to invoke the call it is rebinding.

My first attempt to solve my MACROLET problem (foolishly) imitated the above by substituting double MACROLET, this resulted in the same infinite recursion (because, unlike function invocation, macroexpansion is always done with the innermost environment).

I see a few ways to proceed:

• Interpose a different symbol (e.g. JSON-EL) to signal where a comma might be needed, so that JSON-ARR doesn't need to be re-defined. Con: makes client-code more verbose, adds indentation
• Write the macros using symbols that won't be re-bound (e.g. %json-arr), and then copy into the public symbols (via (SETF (MACRO-FUNCTION 'json-arr) ...), or just (DEFMACRO json-arr (&body body) `(%json-arr ,@body))). Con: these other symbols would appear in macro-expansions, MACROLET is also a PITA
• Use a special variable to communicate between macros that they may need to inject a comma (requires manually running the expanders with MACROEXPAND). Doesn't work, because MACROEXPAND doesn't expand subforms, and by the time the subforms get expanded the special var is no longer bound.

Opinions, suggestions?

[David Mullen]

Here's one way—make another macro and progressively shadow that with MACROLET, instead of JSON-ARR and friends—maybe this is what you meant by "Interpose a different symbol," but it's the simplest approach to my eyes.

Code: Select all

(defmacro json-with-comma
    (&body body)
  `(progn ,@body))

(defmacro json-arr (&body body)
  (bind ((:symbols seen))
    `(json-with-comma
       (write-string "[" *json-stream*)
       (let (,seen)
         (macrolet ((json-with-comma (&body body)
                      `(progn (if ,',seen
                                  (write-string ", " *json-stream*)
                                  (setf ,',seen t))
                             ,@body)))
           ,@body))
       (write-string "]" *json-stream*))))

A variation of this, which I like because it obviates the need for nested backquotes, is to take your idea of "Use a special variable to communicate between macros," except using a symbol macro instead. The symbol macro tracks the gensym.

Code: Select all

(define-symbol-macro
    -seen- nil)

(defmacro json-with-comma (&body body &environment environment)
  `(progn ,@(let ((seen (macroexpand-1 '-seen- environment)))
              (when seen `((if ,seen (write-string ", " *json-stream*) (setq ,seen t)))))
          ,@body))

(defmacro json-arr (&body body)
  (bind ((:symbols seen))
    `(json-with-comma
       (write-string "[" *json-stream*)      
       (symbol-macrolet ((-seen- ,seen))
         (let ((,seen nil)) ,@body))
       (write-string "]" *json-stream*)))))

[marcoxa]

My suggestion is to avoid mixing up macros and printing.
Personally I got bitten very hard (or at least that's the way I felt) by the normal "macro-expanding-into-write" HTML generators out there. Looks to me you are doing the same.

I advocate a different approach, which Common Lisp allows you to follow very well. Have JSONOBJ and JSONARRAY as objects with their proper functional constructors and then just control how they get printed (or pretty-printed) depending on the context.

I may have not been fully consistent, but that's the way I now think of the XHTMLambda code. I blogged about it at http://within-parens.blogspot.ch/2011/0 ... -xhtm.html.

MA

[pjstirling]

Pretty-printing (generated) html is an anti-pattern, it has real costs, and ephemeral benefits.

You should treat it like a data-interchange format (like PNG) and use firebug's DOM inspector instead of mentally parsing the html (it's the DOM you care about after-all).

[marcoxa]

Pretty-printing (generated) html is an anti-pattern, it has real costs, and ephemeral benefits.

You should treat it like a data-interchange format (like PNG) and use firebug's DOM inspector instead of mentally parsing the html (it's the DOM you care about after-all).

IMHO it is the macro-expanding into WRITE that is an anti-pattern. What I believe you are missing form the approach is that I *have* the HTML (and hence the DOM) in my hands to start with. Therefore printing (or pretty printing) is TRT to do. Just as "printing/dumping/marshalling/choose-your-buzzword-du-jour" any data structure is, as long as you have your data structure.

Cheers
--
MA