Help me on Tokinzer

Discussion of Common Lisp
Post Reply
omarasl
Posts: 12
Joined: Sun Nov 25, 2012 6:19 am

Help me on Tokinzer

Post by omarasl » Sun Dec 02, 2012 3:54 am

Hello every body ,
I am very beginner in lisp , I need your help
I am trying to do some thing like dictionary .
I want an example of how I can take sentence like this "this is an example"
then take each word "this" "is" "an" "example"
thank you very much

Goheeca
Posts: 271
Joined: Thu May 10, 2012 12:54 pm
Contact:

Re: Help me on Tokinzer

Post by Goheeca » Sun Dec 02, 2012 4:19 am

Code: Select all

(defun split (string delimiters &optional (empty-strings nil))
  (loop for start = 0 then (1+ finish)
        for finish = (position-if #'(lambda (elem) (member elem delimiters)) string :start start)
        for token = (subseq string start finish)
        when (or empty-strings (/= 0 (length token))) collecting token
        until (null finish)))
Usage:

Code: Select all

(split "Hello world, people!" '(#\Space #\, #\!))
I've edited/improved a snippet on Rosetta Code. The part with empty-strins is unavailing it could have an utilization only if I would collect tokens with the delimiters which are delimiting those certain tokens.
cl-2dsyntax is my attempt to create a Python-like reader. My mirror of CLHS (and the dark themed version). Temporary mirrors of aferomentioned: CLHS and a dark version.

stackman
Posts: 28
Joined: Sat Oct 06, 2012 5:44 am

Re: Help me on Tokinzer

Post by stackman » Sun Dec 02, 2012 5:08 am

As far as I know, there is no ready-to-use splitting function in the cl standard.
However you can use the cl-split-sequence library, available through quicklisp.
Assuming quicklisp is installed,

Code: Select all

CL-USER> (ql:quickload :split-sequence)
To load "split-sequence":
  Load 1 ASDF system:
    split-sequence
; Loading "split-sequence"

(:SPLIT-SEQUENCE)
CL-USER> (use-package :split-sequence)

T
CL-USER> (split-sequence #\space "This is an example.")
("This" "is" "an" "example.")
19
CL-USER> 
The code for split-sequence can be read online at the following url:
http://common-lisp.net/project/clbuild/ ... uence.lisp

Post Reply