Nice-to-have/future Basic Unicode support in parts of game

[davidjoffe]

[This stuff relates to the new 'unstable/dev' stuff that's not yet enabled in the core stable codebase]
These two issues relate to one another: #178 #148

Rough off-the-cuff design thoughts: Just thinking out loud here - Now that we're on SDL2 this may be slightly easier (been mulling in the back of my mind for a long time but on SDL1 it would have been more difficult) but would be nice to start having some Unicode/internationalization support in some parts of the game, perhaps starting with:

• High score names (as this firstly has a very specific user interface area so can maybe just use a different font there without negatively affecting the current 'look and feel' of the game) and also obviously for players with names requiring Unicode, it would be cool to be able to enter their own names. Also we can probably just start treating the high score file format as utf8. Also it can be a test area to get basic helpers working
• Then hmm maybe some main and help screens to help users that don't understand English BUT:
  Obviously the game's main "8x8 font" with its gradient is part of the 'retro vibe' and 'look and feel' and overall '90s/Duke Nukem1 feel' so I feel a bit attached to it - and even though some might find the 8x8 gradient sprite font stuff gross I kinda feel that should stay to start with and not just have everything switched to "modern" font rendering (though will think about ideas to handle that) as it may lose some of that retro vibe. Maybe though for the main menu we could at least start supporting other languages that are largely Latin-alphabet-based but again will think about that.

Will think about if/how this might all be more customizable by porters etc.

This should perhaps later be broken up into separate issues.

{{Unicode stuff --- Possibly for "Version 2"? Or another 1.something release - I guess whether this is part of a new version number depends on whether the core game in terms of gameplay is still just the same or if that changes e.g. levels/graphics etc.}}

[davidjoffe]

FYI created a new branch called "feature-unicode" (to start with will be 'unstable dev branch') to maybe start playing around with this sometime but it opens various cans of worms (eg managing dependencies, possible need for extra fonts .. possibly sdl2-ttf addition to dependencies and its dependent libs) .. so may want to keep this a bit beta for a while and not just throw into the main repo just yet but will think about it. May need to later break it up into additional separate issues ... this is really something for the long term probably not in near future

[davidjoffe]

(For feature-unicode branch beta stuff)

git checkout feature-unicode

In Makefile manually enable:
djUSE_UNICODE=y

BETA Unicode dependencies (experimental may change)

So far just libsdl2-ttf and libutf8proc:

Ubuntu / Debian:

sudo apt install libsdl2-ttf-dev

sudo apt install libutf8proc-dev

Macports:

sudo port install libsdl2_ttf

sudo port install libutf8proc

[davidjoffe]

Very basic still (and some bugs e.g. direction of right to left languages wrong) but this is the first Unicode and TTF support in Dave Gnukem ... you get the idea: :)

NOT to be included in real releases yet until it's more tested and stable (and we add some new font files to data)

Don't want to go crazy and replace every bit of text rendering everywhere (the main 8x8 font is part of the whole 'retro vibe') but I think certain areas could benefit, e.g. maybe start with high score list.

(Would be important to select a decent set of basic fonts for various languages that NB are compatible copyright-wise and license-wise)

It's thanks to Matteo Bini's SDL2 update we can now seriously start doing more interesting things like this

[davidjoffe]

Just to help with Unicode testing here's a quick crude hiscores.dat I made to help with testing .. will update/refine/improve this later .. but just want to try a number of common cases here in this test file .. not all will work perfectly yet but can improve the support over time.

Also FYI for Visual Studio builds and for e.g. languages like Arabic do:
vcpkg install --recurse sdl2-ttf[harfbuzz]
For most the languages the other core deps are:

vcpkg install sdl2-ttf
vcpkg install utf8proc

(I might reconsider utf8proc or maybe remove it as it's only used for one tiny thing so far but not sure yet) 

9
 العربية
1000000
dsf s=דגכעעךלדגכעחי ךדגכ
100000
שלום
69750
试验
42600
asv 试验 よって忘れている世よって忘れている世
34120
דגכよって忘れている世 גכ
31000
Géorgḗ̱̰́̀.
30600
כְּתִיב מָלֵא
3498
والد جون كان يعمل كصحفي تلفزيوني
3000

[dev-testing screenshot] Should look roughly like this (though some issues to fix - here I've correct the direction issues in earlier screenshot but the Arabic requires 'harfbuzz extension' support):