Author Topic: Binary save format re-release. (Added beta version with issues fixed.) (Read 2603 times)

DrenDran · March 27, 2016, 09:13:31 PM

This is a reupload of this:
http://forum.blockland.us/index.php?topic=265798.0
The links died many years ago, but I found the files again. I tried them out and it turns out they work... for the most part.
Anyway, if you want to know why you should try this add-on out, here's why:

In fact, I found an old log that said I could get up to 14x speeds here:
http://pastebin.com/e21mbtVF

Anyway, your save files will be as small as half the size they were before, and upload up to 10x faster to an online server.
The only downside right now is that if you load bricks and the owner of the bricks is not online the events won't load. You'll need to wait until they are online to load their bricks if you want events, or you can just save without ownership.

Downloads:
The support add-on that lets other add-ons make binary files.
The actual add-on advertised, which requires the above add-on.
(updated at 11:17 PM 3/27/2016)

New beta version:
http://blockland.org/newversion/system_binarysave.zip
http://blockland.org/newversion/support_binary.zip
(You need to replace both add-ons, and if you saved files using the previous version, you can't load them with this version)
This version should be able to load events even when the owner of the bricks are offline.
(Updated at 1:45 PM 3/29/2016)

Commands:
/loadBinarySave <filename> --- Loads a save that is saved on the server.
/uploadBinarySave <filename> --- Uploads a save from your client to the server.
/saveBinarySave <filename> --- Saves a save to the server.
/binarySaveCancel --- Cancels the loading of a save.
/autosave --- Turns on autosaver.

Ad Bot · Advertisement

Office-Pirate · March 27, 2016, 09:16:05 PM

hodot · March 27, 2016, 09:19:10 PM

once this is able to load events, i'm definitely using it. i'm assuming it accounts for custom bricks too, and prints?

DrenDran · March 27, 2016, 09:20:21 PM

Quote from: hodot on March 27, 2016, 09:19:10 PM

once this is able to load events, i'm definitely using it. i'm assuming it accounts for custom bricks too, and prints?

Yeah, it should support everything the default format does besides some events. If it doesn't, let me know.
Hell, it even used to work with events, but like I said, something broke that functionality.

Zeblote · March 27, 2016, 09:34:47 PM

You should be able to get a lot more out of this, for example by combining all the flags into a single byte and actually using all available bandwidth for uploading. Chunk the file in 400 byte segments (no more or you risk breaking netevent system!) and send those with commandToServer.

Me and port made some experiments with this a while ago, uploading the compressed file was 40 times faster than the normal upload

DrenDran · March 27, 2016, 09:37:52 PM

Quote from: Zeblote on March 27, 2016, 09:34:47 PM

You should be able to get a lot more out of this, for example by combining all the flags into a single byte and actually using all available bandwidth for uploading.

Code: [Select]

		%stuff = $binaryLoadingFile.readNextObject();

		%rendering = false;
		%raycasting = false;
		%colliding = false;
		%light = false;
		%emitter = false;
		%item = false;
		%print = false;
		%other = false;

		if((1 & %stuff) != 0)
			%rendering = true;
		if((2 & %stuff) != 0)
			%raycasting = true;
		if((4 & %stuff) != 0)
			%colliding = true;
		if((8 & %stuff) != 0)
			%light = true;
		if((16 & %stuff) != 0)
			%emitter = true;
		if((32 & %stuff) != 0)
			%item = true;
		if((64 & %stuff) != 0)
			%print = true;
		if((128 & %stuff) != 0)
			%other = true;

And while I'm pointing out things I've done, yes I already put all the datablocks in a string table at the beginning of the file.

Zeblote · March 27, 2016, 09:41:07 PM

I haven't read the script, but time difference in your post seems quite low compared to the test (it took 18 seconds to upload golden gate bridge vs the normal 12 minutes)

DataProxy · March 27, 2016, 09:51:23 PM

I think I may have stuff bricks reading the description.

DrenDran · March 27, 2016, 10:24:23 PM

Alright, now events will load fine so long as the owner of the bricks is online. If not, they won't load, but you can just clear them and try loading them again when they *do* come online.

portify · March 28, 2016, 04:20:18 AM

Quote from: Zeblote on March 27, 2016, 09:34:47 PM

You should be able to get a lot more out of this

Yes, you can fit rendering, raycasting, collision, angle ID, color ID, color FX ID, shape FX ID into just 2 bytes.
Right now you're writing just angle ID, color ID, color FX ID, shape FX ID as 8 bytes (|writeNumber| < 128 form).

DrenDran · March 28, 2016, 01:08:32 PM

Quote from: portify on March 28, 2016, 04:20:18 AM

Yes, you can fit rendering, raycasting, collision, angle ID, color ID, color FX ID, shape FX ID into just 2 bytes.
Right now you're writing just angle ID, color ID, color FX ID, shape FX ID as 8 bytes (|writeNumber| < 128 form).

I wrote both add-ons like three years ago, but I have been looking through the code and already found a few things I could improve.
The problem is if I release new versions everyone will need to upgrade.
Also, raycasting, collision, and rendering are all stored in the same byte, as shown in the example above.

I may completely rewrite support_binary to allow you to write single bits to a file, which would allow decrease filesize significantly.

ZER� · March 28, 2016, 02:47:38 PM

Is there a shimmering chance that once you rewrite it might replace our default save system?

Also I'm curious, is it possible to have it only save bricks that have your BLID? That would be revolutionary.

DrenDran · March 28, 2016, 03:16:14 PM

Quote from: ZER� on March 28, 2016, 02:47:38 PM

Is there a shimmering chance that once you rewrite it might replace our default save system?

I mean you'd have to talk to Badspot about that one. I doubt he makes user add-ons default often.

Quote from: ZER� on March 28, 2016, 02:47:38 PM

Also I'm curious, is it possible to have it only save bricks that have your BLID? That would be revolutionary.

That seems pretty simple, yeah.

Edit:

I'm working on rewriting support_binary, the new specification will look like this:

11 bit file type header
3 bit version number
3 bit variable type
000: Signed 24 bit integer (-8.3mill to 8.3mill)
001: Signed 31 bit decimal (-8.3 mill to 8.3 mill accurate to 0.01)
010: Signed 16 bit integer (-32768 to 32768)
011: Signed 23 bit decimal (-32768 to 32768 accurate to 0.01)
100: Signed 9 bit integer (-256 to 256)
101: Signed 16 bit decimal (-256 to 256 accurate to 0.01)
110: 6bit/char null terminated string (Stores upper/lower case letters, underscore and numbers)
111: 8bit/char null terminated string (Stores any string)

Conan · March 28, 2016, 04:11:47 PM

i doubt he would replace the default format with this, simply because its less clear whats going on with this format. atm its pretty easy to read and fix (or modify) saves due to its readability.

nevertheless there are obvious benefits to this compressed format - maybe instead actually saving with this format, have it convert save files on the fly and upload it that way? would keep the best of both worlds at the cost of processing the file into the upload format every time before load

DrenDran · March 28, 2016, 08:51:28 PM

I finished the support_binary rewrite. It's not uploaded yet but using it with the current version of system_binarysave seems to create saves that are 15% smaller.