Wide-issue and WT_SESSION

[gvanem]

I've made a little test-program for UTF-8 encoded strings. The gist of it:

/**
 * Check the UTF-8 output of 2 airports.
 *
 * "KEF" should translate to "Reykjavik" with an acute 'i'; U+00ED, as hex "C3 AD"
 * https://www.compart.com/en/unicode/U+00ED
 *
 * "ENFL" should translate to "Floro" with stoke across the 'o'; U+00F8, as hex "C3 B8"
 * https://www.compart.com/en/unicode/U+00F8
 */
static void test_utf8 (void)
{
  const char    *KEF1 =  "Reykjav\xC3\xADk";  /* UTF-8 encoding */
  const wchar_t *KEF2 = L"Reykjav\xEDk";      /* UTF-16 encoding */
  const char    *FLO1 =  "Flor\xC3\xB8";      /* UTF-8 encoding */
  const wchar_t *FLO2 = L"Flor\xF8";          /* UTF-16 encoding */
  int   y;

  if (Modes.tui_interface == TUI_CURSES)
       y = getcury (stdscr);
  else y = con_info_out.dwCursorPosition.Y;

  (*api->print_format) (0, y++, "Using TUI=%s, locale: %s",
                        Modes.tui_interface == TUI_CURSES ? "Curses" : "WinCon",
                        setlocale (LC_ALL, NULL));

  (*api->print_format) (0, y++, "KEF1: '%s'\n"
                                "FLO1: '%s'", KEF1, FLO1);
  y++;

  (*api->print_wformat) (0, y++, L"KEF2: '%s'\n"
                                  "FLO2: '%s'", KEF2, FLO2);
  y += 2;

  (*api->print_wformat) (0, y++, L"Wide:  %-20.20s: %S", KEF2, hex_dump(KEF2, 2*wcslen(KEF2)));
  (*api->print_wformat) (0, y++, L"Wide:  %-20.20s: %S", FLO2, hex_dump(FLO2, 2*wcslen(FLO2)));

  y++;
  (*api->print_format) (0, y++, "ASCII: %-20.20s: %s", KEF1, hex_dump(KEF1, strlen(KEF1)));
  (*api->print_format) (0, y++, "ASCII: %-20.20s: %s", FLO1, hex_dump(FLO1, strlen(FLO1)));
}

The *api->print_format and *api->print_wformat are just func-pointers set to either plain WinCon or PDCurses-mod
functions at runtime.

With a set WT_SESSION=1 before running, it display just fine:

But w/o a set WT_SESSION=1, it looks like this:

I've tried with setlocale (LC_ALL, ".utf8"); and/or SetConsoleOutputCP (CP_UTF8); w/o much difference. Only
this strange set WT_SESSION=1 fixes it.

So what is this and pdc_wt all about?

And BTW, I'm on Win-10 using TakeCommand as my shell.
No Leagacy Console and UTF-8 set in some Windows setting.

[Bill-Gray]

Hmmm... with the warning that this part of the code is not something I know much about, and have therefore refrained from messing about with, and I don't have a way to test this, so I'm just looking at the source code :

pdc_wt has a variety of effects on colors (doesn't seem relevant here as best I can tell). But setting it to TRUE causes pdc_ansi to also be TRUE.

That, in turn, causes us (in pdcdisp.c) to call _show_run_of_ansi_characters(), which displays the actual text using (in wide-character mode) WriteConsoleW().

If you don't have WT_SESSION=1, we instead call (logically enough) _show_run_of_nonansi_characters(). That, in turn, does the actual text display using WriteConsoleOutput().

It would appear that your setup doesn't support WriteConsoleOutput(). The above link suggests that it's a deprecated function.

At present, you can set pdc_ansi to TRUE either by setting WT_SESSION=1 (i.e., the way you did it) or by setting ConEmuANSI=ON. Either way, I think your program will show the characters correctly (the widetest demo would be another good test to apply). It looks to me as if the main difference between the two methods is in how they handle colors. I think if you run the testcurs demo, and run its Color Test, you'll see that one method has a different result (probably more or fewer colors available).

[gvanem]

It would appear that your setup doesn't support WriteConsoleOutput(). The above link suggests that it's a deprecated function.

No, it's still a supported function. Only that Microsoft wants us all to use VT-sequences.

(the widetest demo would be another good test to apply)

With set WT_SESSION=1 (or set ConEmuANSI=ON), the combining stuff gets broken:

With set WT_SESSION=0, it looks OK:

This encoding/UTF stuff is really confusing.

[Bill-Gray]

No, it's still a supported function.

Supported, yes, but deprecated (looks as if I messed up the relevant link previously...) so yes, it should work and Microsoft says it will "support existing usages for the indefinite future".

From your examples, it looks as if the precomposed characters are shown correctly in both cases ("correct" diacritical marks over the top line). The second line should render a duplicate of that line, but using combining characters. In both cases, the combining-character version fails, once by showing blanks, and once by omitting the diacritical marks.

I don't think I've ever seen a Windows terminal that displayed combining characters correctly. They are shown correctly in WinGUI, so the concept isn't totally foreign to our Microsoft colleagues. But as you say, the encoding/UTF stuff is really confusing in Windows; there could easily be some bit of magic that I've overlooked. (It's been about ten years since I worked on a frequent basis on Windows software; that doesn't help!)

[Bill-Gray]

We may have a solution : I see that William McBrine has implemented a revised check for Windows Terminal in 'upstream' PDCurses (a one-line change). Apparently, he also found that WT_SESSION is not necessarily set when it's supposed to be.

If you make that modification, do you find that things suddenly Just Work? If so, I'll cherry-pick the fix and close this issue.