Oracle Text In cards.txt Files

[Chris H.]

here is why. examining the card in that rev, a hex editor shows the byte 0x9e is present where the "LATIN SMALL LETTER U WITH CIRCUMFLEX" belongs. this is neither UTF-8 nor ISO-Latin-1. it is Macintosh specific: http://kb.iu.edu/data/aeqb.html

edit 1: 0x9e isn't even defined by ISO-Latin-1. but in the Windows-1252 encoding, you get the accented Z. http://en.wikipedia.org/wiki/Windows-1252 (column E, row 9)

`
Wow.

UTF-8 and ISO-8859-1 and now Mac OS Roman plus Windows-1252.

It has been an interesting and educational day.

I reverted my card name change.

[Snacko]

whole solution is to convert utf-8 to unicode as this is what Java uses for strings sample code that fixes all the issues with names showing wrong characters.

Code: Select all

try {
               t = new String(t.getBytes(), "UTF-8");
            } catch (UnsupportedEncodingException e) {
               // TODO Auto-generated catch block
               e.printStackTrace();
            }


[Braids]

whole solution is to convert utf-8 to unicode as this is what Java uses for strings

yes, but the best place to do this is where the file is being loaded. having UTF-8 encoded Strings running around is unnecessarily confusing and error prone. the UTF-8 must be decoded as part of the process of creating the String in the first place. that is why we must make a decision about the character encoding of the file itself.

here is how to read a UTF-8 encoded file into Java:

Code: Select all

FileInputStream inStream;
InputStreamReader streamReader;
BufferedReader bufReader;
try {
    inStream = new FileInputStream(file);
    streamReader = new InputStreamReader(inStream, "UTF-8");
    bufReader = new BufferedReader(streamReader);

    String line = bufReader.getLine();
    // ...
}
// TODO catch Exceptions here, like UnsupportedEncodingException
finally {
    try { bufReader.close(); } catch (Exception ignored) { ; }
    try { streamReader.close(); } catch (Exception ignored) { ; }
    try { inStream.close(); } catch (Exception ignored) { ; }
}


also, we have to make sure that the values placed into those files have the correct encoding as well.

for the short term, i think ascii is still the best plan. we have bigger fish to fry.

[Braids]

. . . the best place to do this is where the file is being loaded.

in a fit of irony, i found myself editing the very file that loads card data from disk. this is in my current effort to modify CardFactory to load cards on demand instead of all at once. of course, there are parts of the code that want all the cards loaded all at once anyway. i'm not fixing those yet.

Code: Select all

public class ReadCard implements Runnable, NewConstants {
    public static final String DEFAULT_CHARSET_NAME = "US-ASCII";
    //...
}


it hasn't been committed yet. but change that constant and you change everything.

apologies in advance to Rob. i am trying to limit the number of files i change in this effort, but it is more than i originally anticipated. hmm, there are some files where i only added comments. i can commit those in the next couple of hours. it's a plan.

[friarsol]

Alright, so back to the original point of this thread. Should I be doing anything to the Oracle text of the cards with non-ASCII characters? I'll hold off committing the updated script, I just was gone most of the weekend and didn't get to read up on anything conclusive.

[Braids]

Alright, so back to the original point of this thread. Should I be doing anything to the Oracle text of the cards with non-ASCII characters? I'll hold off committing the updated script, I just was gone most of the weekend and didn't get to read up on anything conclusive.

despite the change i made, i'm still voting ascii, so that gives ascii a narrow margin of one vote.

i guess i can unilaterally declare the voting over and recommend total conversion to ascii.

[friarsol]

i guess i can unilaterally declare the voting over and recommend total conversion to ascii.

You can?

[Rob Cashwalker]

ascii... I was testing the sealed mode missing land issue, and kept getting exceptions because Dandan can't be found by CardFactory.

edit - look, the forum recognizes Dandan without the funny looking "a".

[Braids]

• ascii: Braids, friarsol, Rob Cashwalker
• utf-8: Hellfish
• abstain: Chris H., Jaedayr

ascii wins 3:1.

[Braids]

look, the forum recognizes Dandan without the funny looking "a".

imp mode: yes, but its oracle text includes the circumflex over the a.

i'm still voting ascii in the card text.

[Rob Cashwalker]

Braids - don't be afraid to include more than one response in a post... use the edit button if necessary... I know you're new and trying to bump up your post count....

Yes, I did notice that it uses the weird a in the text...

Why exactly did all this come up anyway? Just because the Oracle text friarsol is going to use may include weird characters that don't match our card names? The Oracle text part is just for human reference... I think our brains can skip to the punchline, and ignore weird symbols. As long as Card.getName is consistent with existing data, then Forge is functional. The printed text of the card can be anything, it doesn't have to match a deck file...

[Braids]

Braids - don't be afraid to include more than one response in a post... use the edit button if necessary... I know you're new and trying to bump up your post count....

actually, i wasn't considering my post count at all. does the subscription system notify its subscribers if i edit a post? i have been assuming it didn't.

Why exactly did all this come up anyway? Just because the Oracle text friarsol is going to use may include weird characters that don't match our card names? The Oracle text part is just for human reference... I think our brains can skip to the punchline, and ignore weird symbols.

yes, but if those weird symbols are a math radical symbol following by a backwards Russian R, when it's supposed to be a LATIN SMALL LETTER A WITH MACRON, it not only looks really really bad, it is beyond the realm of sensible legibility. those are the kinds of behavior we were getting.

the oracle text has to undergo transformation for mana and tap symbols anyway; we may as well asciify it to make it legible. also, function is not everything.

[Hellfish]

It's like election night with all the vote tracking
I'll edit this post when things are back to normal.
EDIT:Done as of r10097.

[Rob Cashwalker]

Braids, just a little gentle hazing... no harm. To be honest, I don't know much about the subscription mode....

[ZzzzSleep]

Even though all I've done is script some cards, my vote (if I get one) is to use the ascii version of the text. Otherwise, somebody could have the situation of a cursed scroll in play and no idea how to type 'Ærathi Berserker'.