utf-8 for RPL source code

[Claudio L.]

(04-21-2015 02:06 PM)Claudio L. Wrote:  Now it's time to tackle the fourth property: decompositions.
...
Time to analyze the patterns to see how to encode this for faster access without increasing the 20k too much.
I'll report back with my findings.

Back on topic, similar analysis by ranges revealed that the singletons can be compacted in 4948 bytes, while the doubles need approx. 18 kbytes to be processed with reasonable speed.
While these tables are efficient for decomposition, they won't help with the inverse process, so an additional table is needed for composition. The additional table is expected to require other 12 kbytes.

This brings the total to about 35 kbytes in tables, plus the space taken by the code.
The penalty in speed is not negligible by any means, but we'll have to live with it.
NFC normalization is only intended to be used when importing Unicode text from external sources (SD card/Serial IO). After that, all text will be assumed to be NFC normalized.

End-of-topic.