Reading formatted data

Hello Lispforum,

what is the closest Lisp equivalent for reading formatted data? (Like C's scanf and Fortran's read).

I have a file with measurement data (doubles) saved in columns, and I want to read those columns
into arrays for further processing.

What ist the easiest way to do this?

I can read the file line by line by "read-line", but this gives me a string og the whole line, which I
cant use without further processing.

"read" requires lisp input, so I cant use it for a sequence of doubles.

I a wondering that processing formatted measured data does seem to be something that is not being
done in Lisp at all, so no standard routines for this exist.

Any suggestions? Thanks.

muuh-gnu wrote:I a wondering that processing formatted measured data does seem to be something that is not being
done in Lisp at all, so no standard routines for this exist.

Common Lisp standard has been created by an ANSI committee in a rather time consuming and expensive process, so the topics it covers are necessarily limited. The formatted input the standard covers are the s-expressions. Other input formats can be read using libraries. Just because something is not covered by the standard doesn't mean that it is not done any more than the fact that ANSI-C doesn't describe sockets means that no networking is done in C.

In the case of numeric column data this is trivially achieved using READ-LINE combined with PARSE-NUMBER and SPLIT-SEQUENCE, both of which are available trough Quicklisp. For more complicated input formats you would use either regular expressions with CL-PPCRE, or a parser generator like CL-YACC or PARSER-COMBINATORS or the like.

You're correct that there's no "standard" system or library for doing this in Common Lisp, which suggests that it's not a problem to which the language is often applied.

I'm afraid that the best way I can suggest is to use read-line, then split the lines into their columns with something like cl-ppcre:split, and then insert each value into their respective array. That's pretty much what a library function would do anyway - either that, or it would employ a parser to read the file character-by-character, which you'd probably only want to do after the first approach exhibited serious performance problems.

This page may also prove useful for parsing numbers from strings.

What's wrong with (untested):

(defun read-tabulated-numbers-from-file (p)
  (with-open-file (f p :direction :input)
    (loop for line = (read-line f nil nil)
          while line
          collect (parse-line line))))


(defun parse-line (line)
  (loop for n = (read-from-string line nil nil)
        while n
        collect n into result
        finally (return (coerce result 'vector))))

This will give you a list of vectors (it is not necessary to do so: it could be a list) organized in rows. Your columns are one transpose away.

Cheers
--
MA

Thanks everybody.

I hoped there would be some "easy" way to read in numeric data files.

I am not able to use quicklisp behind a proxy and I dont know how to install
all those libraries manually, so i'll give it up for now.

Such simple tasks shouldnt be so non-obvious to do in Lisp, from a beginners
point of view, or there should be a "batteries included" implementation delivering
all those "standard" functionalities out of the box like python does.

muuh-gnu wrote:Thanks everybody.

I hoped there would be some "easy" way to read in numeric data files.

I am not able to use quicklisp behind a proxy and I dont know how to install
all those libraries manually, so i'll give it up for now.

Such simple tasks shouldnt be so non-obvious to do in Lisp, from a beginners
point of view, or there should be a "batteries included" implementation delivering
all those "standard" functionalities out of the box like python does.

I don't understand what you're asking for. In what way is READ not what you want?

muuh-gnu wrote:Thanks everybody.

I hoped there would be some "easy" way to read in numeric data files.

I am not able to use quicklisp behind a proxy and I dont know how to install
all those libraries manually, so i'll give it up for now.

Such simple tasks shouldnt be so non-obvious to do in Lisp, from a beginners
point of view, or there should be a "batteries included" implementation delivering
all those "standard" functionalities out of the box like python does.

Well; you said "here is the file I have" and you got the answer: use READ (or READ-FROM-STRING). If you want something "like in Python" you should point to the Python functionalities you are using and then they can either be reproduced, or, more likely, found out there...

Cheers

muuh-gnu wrote:Such simple tasks shouldnt be so non-obvious to do in Lisp, from a beginners point of view, or there should be a "batteries included" implementation delivering all those "standard" functionalities out of the box like python does.

Reading, identifying and interpreting floating-point numbers is not a "simple task" [see the examples below], and I'm in severe doubt that Python can do this, at least not with the "batteries included" libraries.

The main "problem" in this case is that Common Lisp is defined as a hardware independent programming language and the data type "double" is strongly hardware dependent (and software dependent, too).

So the question must be: What is wrong with "doubles"?

The trouble begins with the fact that there are several specifications for data types called "double":

• A "double" can be an 16-bit (double byte) or 32-bit (double 16-bit register) or 64-bit (double 32-bit register) integer. All these formats have been used in the past on various hardware platforms. Probably somewhere in the future there will be double 64-bit, double 128-bit, and so on integers.

• The IEEE_754 defines a floating-point data type named "double" that is from the math point of view not even a number, instead it is an inexact approximation of a number because of several hardware and software limitations.

Summary: the data type "double" virtually tells nothing about the data. It neither tells if the number is an integer or a floating-point number, nor tells it if the data is a number at all. The IEEE_754 defines e.g. "NaN" as "not a number".

For the further discussion I assume that we are talking about IEEE_754 "double" floating-point numbers.

The problem with IEEE_754 floating-point numbers is that only the memory format is defined but not the exaxt printed representation, what has to the consequence that there exist several locale dependent print-formats of IEEE_754 floating-point numbers:

100.0 = "one hundred dot zero" in english/american notation
100,0 = "one hundred comma zero" often used e.g. in Europe

In America the comma is often used as a "thousands" separator, while in Europe the dot is used for this purpose. This leads to an even worse "one million" mess:

1,000,000.0 = "one million dot zero" using commas as "thousands" separator
1.000.000,0 = "one million comma zero" using dots as "thousands" separator

And there are also various "e" formats for "standard", "scientific" and whatever exponentiation.

Now the contest: Whoever can tell me a library in any programming language that can reliably read, identify and interpret all the myriads of "double" printed integer and floating-point formats wins a big piece of pie.

I myself don't know a single library in any programming language that can do this.

The "rft" in "edgar-rft" is the german abbreviation of "Radio/Television Broadcast Technician". My job is to work with all sort of hardware and software measurement equipment every day and I can tell you that printed measurement data in floating-point "double" format can be considered as completely useless as long as nothing is known about the exact print-format of the "doubles" and the floating-point errors of the machine that has produced the data.

To me the question is: what do you want to achive with measurement data represented by an inreliable print-format?

- edgar

P.S: I will try to write a Common Lisp function to read your data into an array if you can tell more about the "doubles" print-format.

edgar-rft wrote:

muuh-gnu wrote:Such simple tasks shouldnt be so non-obvious to do in Lisp, from a beginners point of view, or there should be a "batteries included" implementation delivering all those "standard" functionalities out of the box like python does.

Reading, identifying and interpreting floating-point numbers is not a "simple task" [see the examples below], and I'm in severe doubt that Python can do this, at least not with the "batteries included" libraries.

The main "problem" in this case is that Common Lisp is defined as a hardware independent programming language and the data type "double" is strongly hardware dependent (and software dependent, too).

The OP said scanf(3) can do it, so he's talking about ASCII decimal representations, not binary format. In which case, READ will read them.

• The IEEE_754 defines a floating-point data type named "double" that is from the math point of view not even a number, instead it is an inexact approximation of a number because of several hardware and software limitations.

Can you give an example of an "inexact approximation of a number"?

Guys... we (should) all have read Goldberg's paper, but that has little import to the OP question...

Cheers

Paul wrote:The OP said scanf(3) can do it, so he's talking about ASCII decimal representations, not binary format. In which case, READ will read them (just set *read-default-float-format* to 'double).

'double is no valid Common Lisp type specifier, I assume you meant 'double-float, and here is what happens:

CL-USER> (setf *read-default-float-format* 'double-float)
DOUBLE-FLOAT

CL-USER> 100,0
error: comma is not inside a backquote

CL-USER> 1,000,000.0
error: comma is not inside a backquote

CL-USER> 1.000.000,0
error: The variable |1.000.000| is unbound.

Yes, its true, C's scanf also can't read these numbers, too, but all are valid ASCII representations of floating-point numbers. And this means nothing else than that C's scanf is incapable of reading floating-point numbers, too. This is a reality that happens to me every day.

Paul wrote:C requires that a "double" have at least 37 bits of mantissa equivalent, so there are no 16- or 32-bit "double"s.

DSP machines (producing measurement data) usually work with assembly code, not with C and there are still many mesurement machines using 8-bit and 16-bit architectures, and in a 8-bit architecture a "double" has 16-bit.

Paul wrote:Can you give an example of an "inexact approximation of a number"?

What I meant was of course:

CL-USER> (float 1/3)
0.33333334f0

Okay, this is an approximation of the value of a number, and not an approximation of the number itself.

marcoxa wrote:Guys... we (should) all have read Goldberg's paper, but that has little import to the OP question...

That's true - [David Goldberg: What Every Computer Scientist Should Know About Floating-Point Arithmetic]

But that unfortunately doesn't change the fact that the question can't be answered as long as nothing is known about the floating-point print format. C's scanf cannot serve as a reference because it can't read floating-point numbers correctly.

The main topic of my rant was (and still is):

I have to deal every day with floating-point measurement data in printed form, either in ASCII or Unicode data-files or printed on paper, where nothing is known about the tolerances involved from hardware limitations like unknown quantisation depth or software limitations introduced by sample frequency interpolation or floating-point rounding errors, so the data is virtually useless. This is even worsened by the fact that most english/american C-programs cannot even interpret printed floating-point data correctly.

C's scanf has only very limited functionality and in most cases can only read number printed by C's printf but not by other programming languages. Same to say about Lisp's READ, that only can read numbers in Lisp format. [Same to say about the scan or read functions of any other programming language].

So the original question still remains: Is there any solution known how to read floating-point numbers in any print-format correctly?

- edgar

edgar-rft wrote:
So the original question still remains: Is there any solution known how to read floating-point numbers in any print-format correctly?

- edgar

Yes. Even in Common Lisp, provided that you actually define the "floating point" syntax. Common Lisp defines one for you. I am sure that www.kli.org has another idea of what constitutes a valid typographical representation of a floating number

Cheers

marcoxa: I am sure that www.kli.org has another idea ... - LOL

In case of doubt: I'm indeed actually writing a parser converting "special"-formatted float-strings into Common Lisp syntax. I'm already halfway through but please let me first find the most wackiest bugs on my own, then I will show you the results...

The original parser, where I have stolen a lot of code, is here: cs.cmu.edu/user/ai/lang/lisp/code/math/atof/atof.cl

Hopefully the Klingons will like it, too.

- edgar

edgar-rft wrote: DSP machines (producing measurement data) usually work with assembly code, not with C and there are still many mesurement machines using 8-bit and 16-bit architectures, and in a 8-bit architecture a "double" has 16-bit.

"Double", in this sense, doesn't mean "two hardware words", it means "more bits than 'float'" (presumably "double", but I suspect there are probably implementations out there where a "double" has fewer than that) -- do you know of a machine with 8-bit floating point hardware? What can you represent with that?

(I've used 8 bit machines with no hardware float support, and 48 bit floating point numbers implemented in software...the one and only float type it had, though, so that's a "float", not a "double").

Paul wrote:Can you give an example of an "inexact approximation of a number"?

What I meant was of course:

CL-USER> (float 1/3)
0.33333334f0

Okay, this is an approximation of the value of a number, and not an approximation of the number itself.

Depends what you think it represents. It's an exact number, it's just not the same number as 1/3.