I found that I was unable to let this lie, feeling that what I had suggested was viable, and took your advice, did some research.
I've spent a bit of time thinking about it, and while I am sleep-deprived and not as knowledgeable on the topic in general as you, I
can read.
From Wikipedia, ever a somewhat-reliable source:
"A UTF-8 file that contains only ASCII characters is identical to an ASCII file."I proposed that rather than sending the characters which supposedly would be represented as a single byte within Torque, but as two bytes in UTF-8 (your server), we replaced the characters which required two or more bytes in UTF-8 with the escape sequences which represent them, which consist only of single-byte characters (backslash, x, 0-9 A-F) that Torque can collapse. Sending this to the server using only characters UTF-8 represents with a single byte is valid, as there are no characters which should be represented with multiple bytes and are not. Correct?
To do this, assumedly you would have a fixed resource for the Torque client (
Clockturn@connect.returntoblockland.com/Torque, for example) and using this resource indicates to send the data to your client in its expanded form - single byte characters only, and also to interpret your output this way.
Using any other resource (
Clockturn@connect.returntoblockland.com/Mobile) it would send the collapsed form - the escape sequences collapsed into multi-byte characters as applicable.
So if a Torque client sends a message to another user on a Torque client, the server will leave the input unchanged, and the receiver will display this as it was intended.
If a Torque client sends a message to an outside client, or vice versa, it will need to be translated one way or the other to be displayed as intended.
This would not violate UTF-8 encoding - you are merely sending only certain characters to certain clients to support them, and receiving only certain characters from these clients and translating it as appropriate for clients which can handle proper UTF-8 encoding.
To test this, I created a UTF-8 text file, and wrote "öäadafaöáï¥ë" to it. Reading the UTF-8 encoded file back in Blockland resulted in "öäadafaöáï¥ë".
However, when I wrote "öäadafaöáï¥ë" escaped to the file, and read it back with collapseescape() applied, no characters were lost. Escaped, it reads "\xF6\xE4adafa\xF6\xE1\xC3\xAF\xA5\xEB". It survived being sent through UTF-8; does this not prove my solution?
And if there is a form of these escapes which Java supports, presumedly you have the Torque clients expand and collapse the escapes to and from the Java-compatible format to reduce the strain on the server, so that potentially hundreds of computers are performing these manipulations on single messages, than a single computer performing them on hundreds of messages.
While before I did not quite comprehend the issue and certain things involved in it - and I apologize for that, I am likely still missing something pertinent - I believe I have acquired more clarity on this now, and I still propose the exact same solution, simply expressed far better.
And while I can see how perhaps simplicity might be your priority here, this does not seem even potentially nonviable in any way to me; if there is still something flawed in this, I will leave it.
