| 01:08:13 | <@kiska> | brad: Take a gander towards https://tracker.archiveteam.org/googleplus/ :D |
| 02:00:01 | | Dallas quits [Client Quit] |
| 02:03:41 | | Dallas (Dallas) joins |
| 02:35:40 | <brad> | @kiska: I definitely plan on running my own warrior. Maybe even a team of warriors. We’ll have to see how many I can afford to run. I know a bit about running containers in ECS on AWS, but I gather that would probably be rather expensive for running warriors. So, I might learn about the process there, and then try to figure out how to run containers somewhere else that is less expensive. |
| 02:36:35 | <brad> | But regardless of where they are running, I would also like to contribute to the process of automating their updates and coordination of tasks, whether through a traditional CI/CD process, or something else. |
| 02:42:50 | <brad> | As I understand it, the archive bot is for smaller projects, and the warriors are for larger ones? What is the break over, and how do you determine how large a crawl will be? |
| 02:45:50 | <brad> | I’ve also been involved in other projects like Mailman and NTP, since 2003. Would it be appropriate for me to request that they also get explicitly mirrored? |
| 03:44:20 | | etnguyen03 (etnguyen03) joins |
| 04:40:46 | | qw3rty__ joins |
| 04:44:23 | | qw3rty_ quits [Ping timeout: 252 seconds] |
| 04:51:53 | <OrIdow6> | brad: So I will first mention that the Warrior proper (the Virtualbox image) has been broken for ~2/3 of a year(?), basically due to its copy Ubuntu being too old for the zstd dependency of newer versions of ArchiveTeam's wget fork (wget-lua/wget-at), and there are several different parts, with several people responsible for them, that would have to be fit together to get it working again |
| 04:52:13 | <OrIdow6> | *and due to there being |
| 04:52:52 | <OrIdow6> | There is an intended new version that uses Docker that will hopefully come out soon - but I don't know the details |
| 04:53:41 | <OrIdow6> | In the meantime, most people run the scripts (which I will call "warrior projects", despite this being somewhat of a misnomer at present) in Docker anyway |
| 04:55:37 | <OrIdow6> | A warrior project essentially divides the site into pieces (items), sends each item out to a worker (warriors/Docker containers/raw scripts), and then sends the resultant warc files back to be concatenated together and sent to the Internet Archive |
| 04:56:48 | <OrIdow6> | These require writing a script for the workers to run (which controls that wget fork) and a lot of other preparation (such as making a list of items to be sent) |
| 04:59:03 | <OrIdow6> | ArchiveBot is more or less a regular recursive web crawl that only recurses through pages under a single URL prefix (which must be a host, at minimum) and to first-degree outlinks (as well as getting page requisites etc.) |
| 05:00:35 | <OrIdow6> | Which requires much less preparation, and has a much more limited degree of control (adding blacklist regexes and adjusting the inter-request delay and concurrency) that is exercised after the job is started |
| 05:01:47 | | etnguyen03 quits [Remote host closed the connection] |
| 05:02:43 | <OrIdow6> | Which means it takes much less labor, but is not as scalable, to site size (I think I've heard the figure of a few million URLs thrown around for when it starts getting slow) or to speed (there is a maximum of 9? threads, and those all have to be on the same machine) |
| 05:03:26 | <OrIdow6> | And of course there's no opportunity to write a script for the site, which means there's a lot less flexibility |
| 05:05:40 | <OrIdow6> | Occasionally rarer tools are used, and as with most things there are caveats to both these descriptions |
| 05:06:27 | | Ryz quits [Remote host closed the connection] |
| 05:07:14 | | Ryz (Ryz) joins |
| 05:07:59 | <OrIdow6> | I think there is a fair bit of automation with regard to building the Docker containers etc.? Not closely involved in that area |
| 05:10:06 | <OrIdow6> | And you can at least ask for those things to be saved in #archivebot , but the bigger it is, the less likely it is to get saved if it's simply proactive |
| 05:14:39 | <OrIdow6> | --------------------------- |
| 05:15:45 | <OrIdow6> | Something that shows just how much SmackJeeves has been abandoned: the Forum link that appears fairly prominently on every non-reader page has been broken since July |
| 05:19:49 | <OrIdow6> | " I can't think of a single corporate acquisition like this that went well for the site's users ;_;" - you said it "eishiya" (https://web.archive.org/web/20191205204431/https://forum.smackjeeves.com/viewtopic.php?f=4&t=19772) |
| 06:25:42 | | godane (godane) joins |
| 06:57:24 | | tzt quits [Ping timeout: 240 seconds] |
| 07:16:25 | | Arcorann (Arcorann) joins |
| 07:27:22 | | Doranwen joins |
| 07:27:34 | | Doranwen is now authenticated as Doranwen |
| 07:46:20 | | Mateon1 quits [Remote host closed the connection] |
| 08:19:39 | | Mateon1 joins |
| 09:28:08 | | Ryz quits [Remote host closed the connection] |
| 09:28:57 | | Ryz (Ryz) joins |
| 11:29:56 | | VADemon joins |
| 12:38:15 | | AlsoHP_Archivist joins |
| 12:41:04 | | HP_Archivist quits [Ping timeout: 240 seconds] |
| 12:45:44 | | AlsoHP_Archivist quits [Ping timeout: 240 seconds] |
| 12:46:00 | | AlsoHP_Archivist joins |
| 13:25:46 | | VADemon quits [Client Quit] |
| 13:45:28 | | AlsoHP_Archivist quits [Client Quit] |
| 13:45:44 | | HP_Archivist (HP_Archivist) joins |
| 13:56:44 | | Arcorann quits [Ping timeout: 240 seconds] |
| 14:28:32 | | etnguyen03 (etnguyen03) joins |
| 16:16:35 | | hooway joins |
| 17:14:32 | | hooway quits [Client Quit] |
| 17:21:36 | | hooway joins |
| 17:40:07 | | tzt joins |
| 20:13:30 | | Doranwen quits [Ping timeout: 252 seconds] |
| 20:15:19 | | Doranwen joins |
| 20:23:59 | <themadpro> | Random question from someone who has no idea how .zip compression works: |
| 20:24:16 | <themadpro> | Is it possible to (in theory) crawl the contents of a file without uncompressing? |
| 20:24:32 | <themadpro> | Assuming the .zip archive is homogenous (all text files or all .html files) |
| 20:25:46 | <OrIdow6> | What do you mean, "crawl the contents"? |
| 20:28:45 | <Sanqui> | it's possible to "stream" a zip file without having to decompress on disk yes. |
| 20:30:05 | <@JAA> | Any decompression tool basically reads the data, decompresses in memory, then writes the decompressed version to disk. You can always get rid of the last step with sufficient motivation and just do something with the decompressed data in memory. |
| 20:32:15 | <themadpro> | Orldow6: Sorry for the abrupt afk, basically the Save Yahoo Groups/Yahoo Geddan crew decided to try hunting for Mediafire links (#mediaonfire) and out of curiosity someone asked if it might be possible to search for URL strings without unzipping a zip archive |
| 20:33:24 | <themadpro> | So that's a regular expression that goes a little something like r`http://mediafire.com/file/[0-9a-z]+` |
| 20:33:54 | <themadpro> | The streaming idea seems interesting, how would that work? |
| 20:34:23 | <themadpro> | If we wanted to search for matches while "streaming" a zip? |
| 20:34:33 | <OrIdow6> | -p |
| 20:35:44 | <OrIdow6> | On the "unzip" that gets bundled with (I assume to be) most Linux distros |
| 20:41:10 | <OrIdow6> | By the way, I think Pyxia in #mediaonfire gave the format of some sort of short MF link - might want to include that as well |
| 20:42:43 | <thuban> | mfi.re |
| 21:02:43 | <atphoenix> | themadpro, is that conversation on the SYG discord? |
| 21:27:15 | <themadpro> | yeah |
| 21:27:23 | <themadpro> | feel free to take it there! |
| 21:27:30 | <themadpro> | (I'm pretty busy rn) |
| 21:43:04 | | tzt quits [Ping timeout: 240 seconds] |
| 22:59:45 | | godane quits [Client Quit] |
| 23:32:54 | | Doranwen is now authenticated as Doranwen |
| 23:57:21 | | Connection closed. |