Author Topic: The Chronology of the Blockland Forums (Read 229297 times)

bloody jumper · October 26, 2019, 07:15:16 AM

Quote from: Masterlegodude on October 26, 2019, 05:42:08 AM

Archive.org has a fairly expansive history of the BLF

Yeah, that's true. But I want something client-side that doesn't rely on a server. Maybe something I can save on a hard drive?

Ad Bot · Advertisement

K3k0m@n · October 26, 2019, 10:22:26 AM

Quote from: bloody jumper on October 26, 2019, 07:15:16 AM

Yeah, that's true. But I want something client-side that doesn't rely on a server. Maybe something I can save on a hard drive?

at the very start of 2019 i ran a custom downloader on the blf to download every blf topic (along with almost every linked image and attached files). right now its not in a browsable state however, as its just raw html files not yet localised or put in a proper (searchable!) database.
the total size of the archive is 145.1gb, though just the html files is only 25.8gb. once parsed into a database, this would reduce the filesize quite a bit i imagine

bloody jumper · October 26, 2019, 01:34:28 PM

Quote from: K3k0m@n on October 26, 2019, 10:22:26 AM

at the very start of 2019 i ran a custom downloader on the blf to download every blf topic (along with almost every linked image and attached files). right now its not in a browsable state however, as its just raw html files not yet localised or put in a proper (searchable!) database.
the total size of the archive is 145.1gb, though just the html files is only 25.8gb. once parsed into a database, this would reduce the filesize quite a bit i imagine

Holler at me when you get it into something usable, would love to have the whole forums as a keepsake.

Pecon · October 26, 2019, 10:14:54 PM

Quote from: K3k0m@n on October 26, 2019, 10:22:26 AM

at the very start of 2019 i ran a custom downloader on the blf to download every blf topic (along with almost every linked image and attached files). right now its not in a browsable state however, as its just raw html files not yet localised or put in a proper (searchable!) database.
the total size of the archive is 145.1gb, though just the html files is only 25.8gb. once parsed into a database, this would reduce the filesize quite a bit i imagine

Did you use Heritrix? I'm actually running an archive job right now.

K3k0m@n · October 27, 2019, 08:53:12 AM

Quote from: Pecon on October 26, 2019, 10:14:54 PM

Did you use Heritrix? I'm actually running an archive job right now.

no, i wasnt even aware this program existed. i wrote a (probably not very good) program in nodejs utilising wget which runs through every topic id and downloads them (along with detecting for extra pages and following them too), with functions for downloading attachments and getting source photobucket images. doing all this with my login cookies to bypass the forum word filter.
i have yet to download profiles though

Masterlegodude · December 02, 2019, 08:03:44 AM

Conan · December 02, 2019, 05:18:14 PM

day discussion thread dies

should also add the night night discussion thread died as well :(

Masterlegodude · December 02, 2019, 08:09:47 PM

They can still be posted in, they're just not gonna be updated anymore

I think it's best to wait until they can't be posted in before declaring them dead