UTF-8 support

[Bruno Haible]

Alain Bench wrote:
>     BTW how is a Win32 console app supposed to use libcharset? I mean
> libcharset uses GetACP() only, getting graphic mode default charset
> (typically 1252), while console apps use a usually different text mode
> default charset (typically 850, given by GetConsoleOutputCP()).

The application needs to know where it intends to send a certain
string of text. As a rule of thumb, when writing to a GUI or to a file,
GetACP() is appropriate, and when writing to a console, GetOEMCP()
or GetConsoleOutputCP() (I don't know the difference) is appropriate.

There will always be cases where this heuristic doesn't work, such as
when writing to a pipe whose output ends on the console.

If in the application, you are not able to know in advance whether
a string will end up in the GUI or on a console, a possible implementation
is to work with strings in GetACP() encoding, and recode the output on
the fly in your output routines, immediately before calling WriteFile
or WriteConsole.

> ... on Win32 uses directly GetConsoleOutputCP(),
> unless it fails then GetACP(), then canonicalizes names (28591 ==>
> ISO-8859-1) with the same table as libcharset.

Sounds reasonable.

> There are still cases
> where forcing charset with --charset option becomes necessary.

Such cases will always remain, but one can try to minimize them.

Bruno