using sockets to send a file

[Kuro]

Hi,

I'm trying to figure out how to work with sockets in lisp.
I found the following code, which works fine.
https://www.nicklevine.org/lisp-book/co ... chhttp.pdf

But I ran into some problems as I tried sending a file instead of just strings.

I can send the file, but I haven't managed to report back to the client that the file was received.

Those are the functions I have changed:

Code: Select all

(defun handle-request (stream )
  (with-open-file (file "newpicture.jpg" :direction :output :element-type :default :if-exists :supersede)
    (loop for c = (read-byte stream nil)
       while c do (write-byte c file)))
  (force-output stream))

(defun simple-test (port filepathname)
  (let* ((socket (usocket:socket-connect #(127 0 0 1) port :element-type :default))
         (stream (usocket:socket-stream socket)))
    (with-open-file (file filepathname :element-type 'unsigned-byte)
      (loop for c = (read-byte file nil)
	 while c do (write-byte c stream)))
    (force-output stream)
    (close stream)
    (usocket:socket-close socket)))

This works, but when I try to talk back to the client, the test function just never returns

Code: Select all

(defun handle-file-request (stream )
  (with-open-file (file "newpicture.jpg" :direction :output :element-type :default :if-exists :supersede)
    (loop for c = (read-byte stream nil)
       while c do (write-byte c file)))
  (print "file received" stream)  ;; new
  (force-output stream))

(defun simple-file-test (port filepathname)
  (let* ((socket (usocket:socket-connect #(127 0 0 1) port :element-type :default))
         (stream (usocket:socket-stream socket)))
    (with-open-file (file filepathname :element-type 'unsigned-byte)
      (loop for c = (read-byte file nil)
	       while c do (write-byte c stream)))
    (force-output stream)
    (let ((result (read-line stream))) ;; new
      (close stream)
      (usocket:socket-close socket))
       result))

Any advice?
I'm using SBCL.

Thank you!

[David Mullen]

What effect does the :ELEMENT-TYPE argument have? I'm only familiar with CCL's stream sockets, which are bivalent (taking both characters and bytes).

[pjstirling]

Since you haven't provided all of the code that you are using I have to try and make an educated guess:

First most likely cause of your hang is that you aren't actually writing a newline to your socket, and so READ-LINE on the other end will hang indefintely. Don't use PRINT when you care about the formatting of the output (unless you REALLY know what you're doing). You should use something like:

Code: Select all

(format stream "file received~%")

Second most likely cause is that you are writing bytes on one end of the stream, and characters at the other. Depending on your implementation and choice of EXTERNAL-FORMAT for the socket this may not work as you might hope. FLEXI-STREAMS from quicklisp can help you if that is indeed the problem.

If it's neither of those then I don't know what else to suggest without your full code to play with.

As an aside, writing a file to a socket one byte at a time is AWFUL, you should be using READ-SEQUENCE and WRITE-SEQUENCE.

[Kuro]

What effect does the :ELEMENT-TYPE argument have? I'm only familiar with CCL's stream sockets, which are bivalent (taking both characters and bytes).

you can specify whether you want to send character, unsigned-byte or both.
:element-type :default should be creating a bivalent stream.

Since you haven't provided all of the code that you are using I have to try and make an educated guess:

the full code is provided in the link i posted. maybe i should have made that more clear, sorry. I'll post it here, too.

Code: Select all

(ql:quickload "usocket")
(ql:quickload "bordeaux-threads")

(defun try-make-thread (name function)
  #+bordeaux-threads (bt:make-thread function :name name)
  #-bordeaux-threads (funcall function))

(defvar *server* nil)

(defun start-server (port)
  (let ((socket (usocket:socket-listen usocket:*wildcard-host*
                                       port
                                       :reuse-address t
				       :element-type :default))) 
    (setf *server*
          (try-make-thread (format nil "Port ~a server" port)
                           (lambda ()
                             (unwind-protect
                                  (run-server socket)
                               (usocket:socket-close socket)))))))

#+bordeaux-threads
(defun stop-server ()
  (let ((server (shiftf *server* nil)))
    (when server
      (bt:destroy-thread server))))

(defun run-server (socket)
  (loop
     (usocket:wait-for-input socket)
     (let ((stream (usocket:socket-stream (usocket:socket-accept socket))))
       (try-make-thread (format nil "Request handler for ~s" stream)
                        (lambda ()
			  (with-open-stream (stream stream)
			           (handle-request stream)))))))

(defun handle-request (stream )
  (with-open-file (file "newpicture2.jpg"
			:direction :output
			:element-type :default
			:if-exists :supersede)
    (loop for c = (read-byte stream nil)
       while c do (write-byte c file)))
  (print "file received" stream)
  (force-output stream))

(defun simple-file-test (port filepathname)
  (let* ((socket (usocket:socket-connect #(127 0 0 1) port :element-type :default))
         (stream (usocket:socket-stream socket)))
    (with-open-file (file filepathname :element-type 'unsigned-byte)
      (loop for c = (read-byte file nil)
          while c do (write-byte c stream)))
    (force-output stream)
    (let ((result (read-line stream))) ;; new
      (close stream)
      (usocket:socket-close socket))
       result))

(start-server 4567)
(simple-file-test 4567 "file")

First most likely cause of your hang is that you aren't actually writing a newline to your socket, and so READ-LINE on the other end will hang indefintely. Don't use PRINT when you care about the formatting of the output (unless you REALLY know what you're doing)

i tried that, but without success

Second most likely cause is that you are writing bytes on one end of the stream, and characters at the other. Depending on your implementation and choice of EXTERNAL-FORMAT for the socket this may not work as you might hope. FLEXI-STREAMS from quicklisp can help you if that is indeed the problem.

I played around with that a little bit and that doesn't seem to be the problem either.

one thing that i noticed though, is that the file is only received completly when/after the stream is closed. why is that?

Code: Select all

(defun simple-file-test (port filepathname)
  (let* ((socket (usocket:socket-connect #(127 0 0 1) port :element-type :default))
         (stream (usocket:socket-stream socket)))
    (with-open-file (file filepathname :element-type 'unsigned-byte)
      (loop for c = (read-byte file nil)
          while c do (write-byte c stream)))
    (force-output stream)
    (let ((result (read-line stream))) 
      (close stream)                                     ;; here
      (usocket:socket-close socket))
       result))

As an aside, writing a file to a socket one byte at a time is AWFUL, you should be using READ-SEQUENCE and WRITE-SEQUENCE.

i'll change that, thank you.

[David Mullen]

one thing that i noticed though, is that the file is only received completly when/after the stream is closed. why is that?

Is there a difference between force-output and finish-output with this API?

[Kuro]

Is there a difference between force-output and finish-output with this API?

That's a good question that I don't know the answer to.

But I kinda figured out what the original problem was and everything works now.
read-byte just keeps waiting for input when the stream is empty. So when all the bytes of the file are read it just hangs there.
I changed the code as follows and it works now:

Code: Select all

(defun handle-request (stream )
  (with-open-file (file "newpicture.jpg"
			:direction :output
			:element-type :default
			:if-exists :supersede)
    (loop for c = (if (listen stream) (read-byte stream nil) NIL)
       while c
       do (write-byte c file))) 
  (print "file received" stream)
  (force-output stream))

[David Mullen]

Is there a difference between force-output and finish-output with this API?

That's a good question that I don't know the answer to.

Looking at the SBCL sockets code, the documentation on sb-bsd-sockets:socket-make-stream notes: "Acceptable values for BUFFERING are :FULL, :LINE and :NONE, default is :FULL, i.e. output is buffered till it is explicitly flushed using CLOSE or FINISH-OUTPUT. (FORCE-OUTPUT forces some output to be flushed: to ensure all buffered output is flushed use FINISH-OUTPUT.)"

But I kinda figured out what the original problem was and everything works now.

Yeah, it makes sense that this works. It just seems conceptually hairy to use LISTEN in a bivalent way, when the standard implies it's a character-oriented operation. Like, I wrote this:

Code: Select all

(defun read-sequence-no-hang (sequence stream &key (start 0) end)
  (unless (listen stream) (return-from read-sequence-no-hang start))
  ;; Need some kind of (implementation-specific) non-blocking option.
  (read-sequence sequence stream :start start :end end))

Here, the element being listening for will depend (strictly speaking) on both the type of the sequence and the format of the stream. In the case of multi-byte character encodings, the input element could be a byte or a character-code sequence—so it would make sense to specify which. And this is what Allegro CL does: LISTEN takes an optional argument to know what to listen for—either a character or an octet. An old post on comp.lang.lisp touches on this:

Most fundamentally, CL was developed (note I did not say standardized) at a time before internationalization was popular, when "characters" were the fundamental building block, and bytes were still potentially variable. The description of cl:listen implies this. The time line is of course slushy, and there are those that will argue (incorrectly) that the byte has always been 8 bits, or that characters have always been un-tied to the octet (8-bit-byte). But though this certainly has been true in some pockets of locality, it has only been recent (perhaps the last 10 to 20 years) that the tide has turned toward the proliferation of this reversal, where the octet (8-bit-byte) has become the fundamental data unit of transfer, and the character has achieved some variability.

[David Mullen]

Come to think of it, the more concrete pitfall here is latency—where LISTEN will return false if (some of) the data hasn't arrived yet. In that event, you'd miss out on the rest of the file. (Not a problem with localhost, I guess.)

[pjstirling]

The sensible thing is to send the length of the file before the file data, of course