convert a string of numbers to array of numbers?

After about a year and a half of Lisp study, I've been starting to feel comfortable about starting a small project. Perhaps I've been hasty... I am trying to convert a comma separated values file to a 2d array. I looked at other people's code and I can't understand why their code is so long just to parse a csv file. Perhaps my approach is naive, but this is what I did:

(defvar *stream* (open "C:\\yada\\table.csv"))
	(substitute #\space #\, (read-line *stream*))

;Sample result:
"2010-09-30 115.05 115.79 113.59 114.13 287106700 114.13"

I probably should use WITH-OPEN-FILE instead of OPEN, but I couldn't read the next line in the csv file. I know I don't have an iterative array making part yet, but I am stuck at reading the entire string separated by spaces. I used READ-FROM-STRING (and a few combination of its keywords), but it seems to stop at the first space:

(read-from-string (substitute #\space #\, (read-line *stream*))

;Sample result:
|2010-09-27|
10

I understand the first element isn't really a number, but I forgot what those bars mean: |...|
I also tried to substitute the quotation marks but it seems that Lisp doesn't understand this: #\"

speech impediment wrote:I looked at other people's code and I can't understand why their code is so long just to parse a csv file.

The CSV format is described by RFC4180. It is not quite that trivial in general case. Also, you have to spend some code on parsing the values themselves. Lisp reader, which is what READ-FROM-STRING uses, is specialized for reading Lisp forms and reading general data with it is not a good idea.

You should probably use already existing CSV-reading library, or at least a parser generator. CLiki has a page with a list of parsers and generators. I have written a parser combinator library which also can be used for this purpose.

speech impediment wrote:I understand the first element isn't really a number, but I forgot what those bars mean: |...|

Bar syntax is for quoting symbol names. A name of a symbol in Common Lisp can be an arbitrary string, and bars are used when that string could conflict with some other reader syntax.

speech impediment wrote:

(defvar *stream* (open "C:\\yada\\table.csv"))
(substitute #\space #\, (read-line *stream*))

(edited for clarity -- the initial indentation fooled me for a while.

speech impediment wrote:I probably should use WITH-OPEN-FILE instead of OPEN, but I couldn't read the next line in the csv file.

With-open-file is just a macro that closes the file when the macro body completes (try macroexpanding it to see how it works). It is perfectly fine to close the file manually by yourself.

speech impediment wrote: I know I don't have an iterative array making part yet, but I am stuck at reading the entire string separated by spaces. I used READ-FROM-STRING (and a few combination of its keywords), but it seems to stop at the first space:

Yes. Read-from-string just reads one form (symbol, number, string, parenthesized expression, etc.) at a time. You will have to use it in a loop to get everything out of the line.

speech impediment wrote:

(read-from-string (substitute #\space #\, (read-line *stream*))

;Sample result:
|2010-09-27|
10

I understand the first element isn't really a number, but I forgot what those bars mean: |...|
I also tried to substitute the quotation marks but it seems that Lisp doesn't understand this: #\"

This "substitute #\space #\," should work as long as your values don't contain commas or spaces (i.e. no string values). The vertical bars mark symbol names containing nonstandard characters (mixed upper and lower case, spaces, etc.). The quotation marks are added by the printer to make strings usable by READ -- they don't actually exist in your sample data.

Thanks again Ramarren and nuntius. It seems like I have a lot more work to do before I do any useful programming.

Ramarren wrote:

speech impediment wrote:I looked at other people's code and I can't understand why their code is so long just to parse a csv file.

The CSV format is described by RFC4180. It is not quite that trivial in general case. Also, you have to spend some code on parsing the values themselves. Lisp reader, which is what READ-FROM-STRING uses, is specialized for reading Lisp forms and reading general data with it is not a good idea.

Indeed. The CSV format is one of those deceptive formats that seems easy until you discover all the corner cases, protocols for escaping various special characters, etc. In short, it's harder than it looks, which results in the complexity of the code.