| 00:01:49 | <betamax> | OrIdow6: I've just put the full url list of all sploder games into archivebot (~4 million URLs) |
| 00:02:47 | <betamax> | I've also put the first round of comments in (saving the last page of comments for every game - I'll then need to go through the output, see which games have multiple pages of comments and archive those in a separate job) |
| 00:04:00 | <betamax> | I'm off to bed now (it's midnight here) but those jobs will take a while to start as they each need to download a ~300MB list of URLs first - if anyone notices them misbehaving, feel free to kill them |
| 00:05:10 | <@Kaz> | I'm more impressed you managed to get 300MB onto that transfersh instance |
| 00:08:07 | <betamax> | it took around 30 mins, but seemed to go OK :) |
| 00:08:54 | <betamax> | from experience with the Yahoo groups /info pages grab from a week ago, archivebot takes quite a while to download the url list |
| 00:09:12 | <@JAA> | Yeah, the transfer instance being in Sydney does that. |
| 00:10:26 | <@Kaz> | why does that still exist |
| 00:11:00 | <@Kaz> | though actually, I guess I don't care. I'm less willing to host a public file upload service than others, clearly |
| 00:11:08 | <@Kaz> | if it being in Sydney is the tradeoff for that, then sure |
| 00:11:25 | <betamax> | how can I see what pipelines the jobs are running on? Ideally I'd want to have them on different ones, to avoid hammering sploder twice from the same IP |
| 00:11:59 | <@Kaz> | |
| 00:12:03 | <@Kaz> | http://dashboard.at.ninjawedding.org/logs/recent |
| 00:12:06 | <@JAA> | I don't think there's a particular reason for it being in Sydney. That's just what kiska clicked on in the AWS interface when creating the server. |
| 00:13:16 | <betamax> | excellent, both jobs are on separate pipelines |
| 00:55:39 | | fieugh quits [Client Quit] |
| 01:01:10 | <OrIdow6> | Thanks betamax |
| 01:03:17 | <OrIdow6> | arkiver: Are you working on SmackJeeves, or should I? |
| 01:04:12 | <OrIdow6> | Also, do any of you FlashPoint people know if they've done anything on LaCartoonerie? It's an animation site w/o games, so I'd guess not, but I want to check first |
| 01:04:42 | | godane1 joins |
| 01:07:08 | | godane quits [Ping timeout: 252 seconds] |
| 01:12:19 | <@arkiver> | OrIdow6: just cloned your repo, we'll start tomorrow |
| 01:12:49 | <OrIdow6> | arkiver: What repo? I don't think I published any code I wrote for SJ on GH |
| 01:12:57 | <@arkiver> | the eu one |
| 01:13:00 | <OrIdow6> | Oh |
| 01:13:35 | <@arkiver> | I can get something going to smackjeeves, but feel free to start working on something |
| 01:13:40 | <OrIdow6> | Ok |
| 01:13:51 | <@arkiver> | from the looks of it, it's luckily not too difficult to get done |
| 01:14:21 | <OrIdow6> | Any ideas on scraping the .eu sites themselves (as opposed to the whois)? I talk about this at "So my current approach"... |
| 01:14:33 | <OrIdow6> | Yeah, SmackJeeves looks like it has a simple structure |
| 01:15:05 | <OrIdow6> | *grabbing |
| 01:16:49 | <@arkiver> | why not just go up to a certain depth for each .eu domain? |
| 01:17:20 | <@JAA> | ++ |
| 01:18:21 | <@JAA> | I was thinking of throwing the .eu lists into AB as !a < and aborting after a while (since there isn't a way to limit recursion depth), but if we can do that more cleanly, even better. |
| 01:18:23 | <@arkiver> | say 4 or 5, that'll get you most of the pages if it's a "regular" (not news, etc., but businesses, etc.) website |
| 01:18:40 | <@arkiver> | JAA: could pretty easily make a project for that |
| 01:19:06 | <@JAA> | Yeah, really easy, no custom code needed, and no runaway danger. |
| 01:19:14 | <@arkiver> | :) |
| 01:19:36 | <@arkiver> | could add a hard stop after x seconds, or a certain number of URLs |
| 01:19:48 | <@arkiver> | 100k URLs? |
| 01:19:49 | <@JAA> | + size perhaps? |
| 01:19:50 | <@arkiver> | or 10k or so |
| 01:19:59 | <@arkiver> | sure, can do size as well |
| 01:21:09 | <@arkiver> | i gotta go now though, and will be minimally available (but available) tomorrow |
| 01:22:10 | <@arkiver> | please do leave thought here, will go over them tomorrow. |
| 01:35:54 | <OrIdow6> | Ok |
| 02:00:01 | | Dallas quits [Client Quit] |
| 02:03:48 | | Dallas (Dallas) joins |
| 03:21:16 | | fuzzy8021 is now authenticated as fuzzy8021 |
| 03:22:54 | | fuzzy8021 quits [Changing host] |
| 03:22:55 | | fuzzy8021 (fuzzy8021) joins |
| 03:50:35 | <flashfire42> | OrIdow6 flashpoint does animations but not as many people focus on then |
| 03:52:43 | <@kiska> | betamax If you want to speed up the download, take the resulting url, replace transfer.notkiska.pw with kiska.b-cdn.net :D |
| 03:54:03 | | paul2520 joins |
| 03:54:27 | | paul2520 quits [Remote host closed the connection] |
| 04:11:49 | | lukash7 quits [Quit: The Lounge - https://thelounge.chat] |
| 04:15:25 | | lukash7 joins |
| 04:40:45 | <thuban> | we had a request on the subreddit for farmville assets. i have a list, do we have archivebot capacity? it's about 400k urls, mostly swf (still or lightly-animated images, i expect) and jpg. |
| 04:42:53 | | qw3rty__ joins |
| 04:43:57 | <Ryz> | ...O: |
| 04:44:06 | <Ryz> | We could go through all of it thuban |
| 04:44:09 | <Ryz> | Assuming that's all of it |
| 04:44:21 | <Ryz> | We still plenty of slots |
| 04:46:24 | | qw3rty_ quits [Ping timeout: 240 seconds] |
| 04:46:46 | <thuban> | ok, sounds good |
| 04:47:09 | <thuban> | https://bluepload.unstable.life/selif/farmvilleassets.txt <- 44M of urls |
| 04:48:52 | <@JAA> | Akamai, nice, let's go nuts. |
| 04:50:20 | <Ryz> | Even though I never played Farmville at all, I love the art goods o: |
| 04:55:01 | <OrIdow6> | flashfire42: Ok |
| 04:55:42 | <@JAA> | thuban: Running now. |
| 04:56:15 | <thuban> | :3 |
| 04:59:09 | <OrIdow6> | From the AB dashboard, looks like there are only about 395K, not 44M, URLs in that list? |
| 04:59:45 | <OrIdow6> | Oh, never mind, that's bytes |
| 05:02:50 | <Ryz> | No, that seems about right, it's not 44 million links? |
| 05:02:59 | <Ryz> | It's just 384k amount of links? |
| 05:04:13 | <OrIdow6> | File is about 43.8 MB |
| 05:08:11 | <Ryz> | Oh, are you talking about the filesize? And not the amount of links? |
| 05:08:59 | <OrIdow6> | That's where I assume "44M of urls" came from |
| 05:09:09 | | lennier1 quits [Quit: Going offline, see ya! (www.adiirc.com)] |
| 05:09:31 | <@JAA> | Yeah, 400k URLs, 44 MB file size. |
| 05:50:37 | | lennier1 (lennier1) joins |
| 06:43:20 | <thuban> | https://transfer.notkiska.pw/ntvIt/farmvilleassets2.txt small supplementary to above |
| 06:56:49 | | HP_Archivist quits [Client Quit] |
| 08:13:06 | | wickedplayer494 quits [Remote host closed the connection] |
| 08:34:16 | | wickedplayer494 joins |
| 08:41:58 | | wickedplayer494 is now authenticated as wickedplayer494 |
| 09:09:32 | <OrIdow6> | arkiver: When I encounter a deleted/nonexistant user/comic, should I abort or succeed with 1 page? |
| 10:17:18 | <jodizzle> | thuban: I threw it in |
| 10:17:28 | <jodizzle> | (already done) |
| 10:18:50 | <@kiska> | OrIdow6: Task failed successfully :D |
| 10:31:42 | <OrIdow6> | Yeah, that's basically what's going on |
| 10:53:26 | | hooway joins |
| 11:02:36 | | lunik1 quits [Ping timeout: 252 seconds] |
| 11:07:59 | | lunik1 joins |
| 13:03:44 | | ragu__ quits [Ping timeout: 240 seconds] |
| 13:05:54 | | ragu joins |
| 13:20:55 | | klg (klg) joins |
| 13:48:03 | | DLoader_ joins |
| 13:51:17 | | DLoader quits [Ping timeout: 252 seconds] |
| 13:51:24 | | DLoader_ is now known as DLoader |
| 14:40:47 | | Arcorann quits [Ping timeout: 252 seconds] |
| 16:43:07 | <@arkiver> | OrIdow6: if you find from the first URL that the ID doesnt exist, you can just stop queuing URLs and let if finish |
| 16:43:28 | <@arkiver> | so that'll be a WARC with a single response record or so |
| 16:58:49 | | pallog joins |
| 18:18:32 | | godane1 quits [Client Quit] |
| 18:31:35 | | HackMii quits [Remote host closed the connection] |
| 18:31:56 | | HackMii (hacktheplanet) joins |
| 18:32:45 | | HackMii quits [Remote host closed the connection] |
| 18:33:04 | | HackMii (hacktheplanet) joins |
| 18:36:18 | | lukash7 quits [Client Quit] |
| 18:38:59 | | lukash7 joins |
| 18:47:09 | | pallog is now authenticated as pallog |
| 18:50:59 | | pallog_ joins |
| 18:53:24 | | pallog quits [Ping timeout: 240 seconds] |
| 19:19:32 | | pallog_ quits [Client Quit] |
| 20:03:57 | | pallog (pallog) joins |
| 20:04:30 | <OrIdow6> | Ok |
| 21:31:27 | | Arcorann (Arcorann) joins |
| 21:32:41 | <OrIdow6> | Almost done with SmackJeeves comics, just ironing out playback |
| 21:57:44 | <OrIdow6> | Great time for my net speed to go down to like 30 KB/s |
| 22:14:01 | <OrIdow6> | This is like trying to run underwater |
| 22:14:06 | <OrIdow6> | Will finish it later |
| 22:32:40 | | tzt quits [Ping timeout: 252 seconds] |
| 22:36:12 | | tzt joins |
| 23:07:24 | | tzt quits [Ping timeout: 240 seconds] |
| 23:16:24 | | pallog quits [Ping timeout: 240 seconds] |
| 23:19:51 | | tzt joins |
| 23:21:03 | | c00k13 quits [Quit: bye~] |
| 23:21:27 | | c00k13 joins |
| 23:26:28 | | c00k13 quits [Client Quit] |
| 23:28:04 | | tzt quits [Ping timeout: 240 seconds] |
| 23:35:32 | | tzt joins |
| 23:41:08 | | c00k13 joins |
| 23:53:53 | | hooway quits [Client Quit] |