Page 1 of 1
SBCL - BSD Socket
Posted: Wed Jun 06, 2012 3:24 pm
by plu9in
Hi all,
In my lisp code, I open a socket to download websites but I have a problem with the socket charset. When I open my socket, I have to give it a charset. But, surfing the web, my socket has to deal with others charsets.
Before downloading the page, how can my code know the charset requested ? Is it possible to change the charset of an existing socket during its life ?
Thank you for your help.
Re: SBCL - BSD Socket
Posted: Thu Jun 07, 2012 3:45 am
by pjstirling
The short answer is that this is a very ugly problem
HTTP speaks an 8 bit encoding compatible with ASCII, but the HTML inside the response can be any encoding, including multi-byte encodings like UTF-16 UTF-32 or Shift-JIS.
You need to open the socket with a raw 8 bit encoding, speak to the server in HTTP to make the request, and then download the body of the response.
Once you have the body you can:
- check for the 8, 16, or 32 bit versions of the unicode byte-order mark (BOM), in which case it's that encoding,
- look for an xml charset processing directive,
- scan the html HEAD element for a charset directive, and
- finally: guess based on byte frequencies.
Re: SBCL - BSD Socket
Posted: Fri Jun 08, 2012 1:44 am
by wvxvw
To add one more option to how the page may specify the character encoding - it's through HTTP header, namely "Content-Type: text/html; charset=utf-8" specifies utf-8, obviously. You also need to look for Transfer-Encoding header, because while the content may be encoded in some text coding technique, it may be later compressed using gzip, for example.
Way much-much-much worse than all above is that web was 90% crafted by people who didn't have a clue about how these things work. So very often the information you will receive in headers, meta-equivalent and other "less visible" attributes of the page will be wrong! Effectively, browsers usually just guess the encoding based on character frequencies.
Re: SBCL - BSD Socket
Posted: Sun Jun 10, 2012 7:12 am
by Kompottkin