Author Topic: Binary build save format test.  (Read 1619 times)

Zip with support_binary and server_binarysave.
To save try saveBricks(%path)
To load try loadBricks(%path)
http://www.filedropper.com/twobinaryaddons
It doesn't save anything but basic brick data at the moment as its a test.
So no prints or events or anything QUIET yet.



Alright, so I'm going to be releasing and using a script that enables the writing of binary data to files. Unfortunately you can't use this to write most existing file types, however you can use it to make large files smaller and/or to obfuscate data. I'm going to be releasing a binary save format test shortly after this.

There's a few things to the file format to keep in mind, first is a few sequences:
  • 0xFF,0xFF is the character 0xFF, or the escape character
  • 0xFF,0xFE is the character 0x0A, or the new line character
  • 0xFF,0xFD is the character 0x00, or the null string terminator
  • 0xFF,0xFC is the character 0x0D, or the carriage return
  • 0xFF,0xFB signifies the end of a file.

On the lowest level of reading and writing data these escape sequences are accounted for, if you're using this you don't even need to know this is a thing.

The script also supports the saving of complex data types:
  • Type 3 is a signed three byte integer, that is a number from roughly -8.3 million to positive 8.3 million
  • Type 4 is a signed one byte character, that is a number from -127 to 127
  • Type 5 is an unsigned one byte character, that is a number from 0 to 255
  • Type 6 is an unsigned 4 byte integer, that is a number from about 0 to 4.2 billion
  • Type 7 is a string, which is sort of null terminated, with the escape sequence 0xFF,0xFD instead of an actual 0x00
  • There are some array data types in the works too
When using a writeNumber function, the functions will automatically choose the most efficient data type for the number. So any numbers from 0 to 255 for example, won't even use more than two bytes, where as numbers in the millions will. readNumber functions automatically detect datatypes.

API:
  • openBinaryFileForRead(%path) --- Returns the binaryFile if successful, 0 on error
  • openBinaryFileForWrite(%path) --- Returns the binaryFile if successful, 0 on error
  • binaryFile.read() --- Low level function for advanced use, reads next byte. Does translate escape sequences, does not look for objects. Can't be used with readObject on the same file.
  • binaryFile.write(%data) --- Low level function for advanced use, writes a byte. Does translate escape sequences, will not make an object. Limited from 0 to 255. This function may not be used with writeArray, writeString, or writeNumber on the same file.
  • binaryFile.writeNumber(%number) --- High level function for common use. Writes a number to the file.
  • binaryFile.writeString(%string) --- High level function for common use. Writes a string to the file.
  • binaryfile.writeArray(%arrayName,%start,%end) --- High level function for common use. Writes an array to a file. The first argument is the name of the variable before the brackets. Start and end are the starting an ending positions. Example: %file.writeArray("$array",0,10) would write the values for $array[0] to $array[10]
  • binaryFile.readObject() --- Detects and returns the next object in the file.
  • binaryFile.close() --- Writes the buffer if its a write file, closes the file, then deletes the objects. This MUST be done for any write-file you have, or else the last buffer's worth of data won't write to the file. Also works with read-files, but not as required.
  • binaryFile.isEOF() --- Returns if the file has been completely read.

Tell me what you think, I'm going to post this probably tomorrow once I've bug tested.
Soon after I'm going to release a binary save format. It should have smaller files and even fix some of the serious problems with the current format.
« Last Edit: August 25, 2014, 10:29:51 PM by DrenDran »

0xFF,0xFD is the character 0x00, or the null string terminator

This is the important one. What do you mean?

1) Passing 00 to the api writes FFFD to the file
2) Passing FFFD to the api writes 00 to the file

This is the important one. What do you mean?

1) Passing 00 to the api writes FFFD to the file
2) Passing FFFD to the api writes 00 to the file
The first lol.

You can't write 0 to the file because torque can't handle 0 at all. (I'm sure you know this, btw.) You can't write 10 or 13 because torque reads files by line and that'd just be interpreted as ending the line. You can't write 255 because that's the escape code I choose to help convey the idea of 0,10, and 13.

The first lol.

You can't write 0 to the file because torque can't handle 0 at all. (I'm sure you know this, btw.)

Yeah, I was just hoping you stumbled across something incredible. :(

Good luck on the project's actual goal then!

Yeah, I was just hoping you stumbled across something incredible. :(
Yeah, I knew I was dashing some hopes there.
Good luck on the project's actual goal then!
Thanks mang.


Now to figure out how to save decimals.
I was thinking of just writing two or three bytes for the whole number part, and one byte for the decimal point.
Can you or anyone else think of times in blockland where numbers need to be more precise than the nearest ~0.004 interval?

Wait... I thought 0x0A was the carriage return. If it's not that, then what is it?
OT: This looks extremely useful, when do you think you could release it?

Test script with test file:
http://www.filedropper.com/supportbinary

Try this code:
Code: [Select]
$testFile = openBinaryFileForRead("add-ons/support_binary/test.bin");
while(!$testFile.eof)echo($testFile.readNextObject());
$testfile.close();

Results should be:
Code: [Select]
0
1
5
-1
-5
0.5
-0.5
-300
300
8e+006
-8e+006
-50.75
50.25
This is a test string!

If 0xfffd is replaced with 0x00 then how does one write 0xfffd to the file without it being automatically converted to 0x00?

If 0xfffd is replaced with 0x00 then how does one write 0xfffd to the file without it being automatically converted to 0x00?
0xfffffd lol
0xff is one of the characters that can't be written directly.
255 is represented by 0xffff not 0xff.

Alright fair enough

Another concern: Torque doesn't have full 4-byte unsigned integers. It's all signed integers. They'll be auto-converted to floats and you'll loose 9 bits of precision.

Alright fair enough

Another concern: Torque doesn't have full 4-byte unsigned integers. It's all signed integers. They'll be auto-converted to floats and you'll loose 9 bits of precision.
>they'll be auto converted to floats
What will be?

Torque not having unsigned integers but only signed integers only affects people who want numbers from 2.1 to 4.2 billion.
This mod as of yet doesn't support anything close to that.
It supports signed 24bit integers from about -8.3 million to 8.3 million, that's the biggest datatype save for the decimal point Integer.

>they'll be auto converted to floats
What will be?

Torque not having unsigned integers but only signed integers only affects people who want numbers from 2.1 to 4.2 billion.
This mod as of yet doesn't support anything close to that.
It supports signed 24bit integers from about -8.3 million to 8.3 million, that's the biggest datatype save for the decimal point Integer.
I'm saying that Type 6 shouldn't exist because torque doesn't support unsigned integers of that size.

I'm saying that Type 6 shouldn't exist because torque doesn't support unsigned integers of that size.
Yeah, type 6 was the only one I haden't implemented yet, besides string arrays.
The actual updated reference is here:
Code: [Select]
//By DrenDran BL_ID 22749
//
//Reference
//
//Bytes that can't be directly written to file:
//10 -- New line                    Written as 255,254
//0 -- Null string terminator       Written as 255,253
//255 -- Escape code                Written as 255,255
//13 -- Carriage Return     Written as 255,252
// Sequence 255,251 is EOF
//
//Data types and ids:
//Signed three byte Integer               3
//Signed one byte char   4
//Unsigned one byte char                  5
//Decimal Point Number  (not a float)     6
//String                                  7
//Array of 6   8
//
//Numbers guarenteed percise to 0.011 up to +/-8.3 million

Edit: second test here:
http://www.filedropper.com/supportbinary_1
« Last Edit: August 25, 2014, 04:54:20 PM by DrenDran »