Help me on Tokinzer

[omarasl]

Hello every body ,
I am very beginner in lisp , I need your help
I am trying to do some thing like dictionary .
I want an example of how I can take sentence like this "this is an example"
then take each word "this" "is" "an" "example"
thank you very much

[Goheeca]

Code: Select all

(defun split (string delimiters &optional (empty-strings nil))
  (loop for start = 0 then (1+ finish)
        for finish = (position-if #'(lambda (elem) (member elem delimiters)) string :start start)
        for token = (subseq string start finish)
        when (or empty-strings (/= 0 (length token))) collecting token
        until (null finish)))

Usage:

Code: Select all

(split "Hello world, people!" '(#\Space #\, #\!))

I've edited/improved a snippet on Rosetta Code. The part with empty-strins is unavailing it could have an utilization only if I would collect tokens with the delimiters which are delimiting those certain tokens.

[stackman]

As far as I know, there is no ready-to-use splitting function in the cl standard.
However you can use the cl-split-sequence library, available through quicklisp.
Assuming quicklisp is installed,

Code: Select all

CL-USER> (ql:quickload :split-sequence)
To load "split-sequence":
  Load 1 ASDF system:
    split-sequence
; Loading "split-sequence"

(:SPLIT-SEQUENCE)
CL-USER> (use-package :split-sequence)

T
CL-USER> (split-sequence #\space "This is an example.")
("This" "is" "an" "example.")
19
CL-USER>


The code for split-sequence can be read online at the following url:
http://common-lisp.net/project/clbuild/ ... uence.lisp