SBCL - BSD Socket

[plu9in]

Hi all,

In my lisp code, I open a socket to download websites but I have a problem with the socket charset. When I open my socket, I have to give it a charset. But, surfing the web, my socket has to deal with others charsets.

Before downloading the page, how can my code know the charset requested ? Is it possible to change the charset of an existing socket during its life ?

Thank you for your help.

[pjstirling]

The short answer is that this is a very ugly problem

HTTP speaks an 8 bit encoding compatible with ASCII, but the HTML inside the response can be any encoding, including multi-byte encodings like UTF-16 UTF-32 or Shift-JIS.

You need to open the socket with a raw 8 bit encoding, speak to the server in HTTP to make the request, and then download the body of the response.

Once you have the body you can:

• check for the 8, 16, or 32 bit versions of the unicode byte-order mark (BOM), in which case it's that encoding,
• look for an xml charset processing directive,
• scan the html HEAD element for a charset directive, and
• finally: guess based on byte frequencies.

[wvxvw]

To add one more option to how the page may specify the character encoding - it's through HTTP header, namely "Content-Type: text/html; charset=utf-8" specifies utf-8, obviously. You also need to look for Transfer-Encoding header, because while the content may be encoded in some text coding technique, it may be later compressed using gzip, for example.

Way much-much-much worse than all above is that web was 90% crafted by people who didn't have a clue about how these things work. So very often the information you will receive in headers, meta-equivalent and other "less visible" attributes of the page will be wrong! Effectively, browsers usually just guess the encoding based on character frequencies.

[Kompottkin]

Or use Drakma.