| 00:21:34 | <@arkiver> | tech234a: yep! |
| 00:25:00 | <OrIdow6> | Also, on the main .eu crawl: need a way to deal with the shutdown (cache DNS? Stop the grab at midnight UTC?); and maybe it would be possible to convert the AB ignores to whatever syntax wget uses |
| 00:25:32 | <@arkiver> | OrIdow6: yes, will take care of that |
| 00:25:42 | <@arkiver> | getting the comics project started now |
| 00:25:48 | <OrIdow6> | Ok, thanks |
| 00:32:14 | <@arkiver> | OrIdow6: you're in a team now with the smackjeeves repos |
| 00:32:23 | <@arkiver> | well you're invited |
| 00:57:00 | <purplebot> | Voat edited by Wickedplayer494 (+125, Case closed according to Fusl, …) just now -- https://www.archiveteam.org/?diff=45986&oldid=45980 |
| 00:57:05 | <@arkiver> | kiska: HCross: do you have space for a target for smackjeeves? |
| 00:57:30 | <@kiska> | Uh sure |
| 00:57:33 | <@arkiver> | args would be smackjeeves_ archiveteam_smackjeeves_ and Archive Team Smack Jeeves: |
| 00:57:35 | <@kiska> | vars? |
| 00:57:38 | <@kiska> | :D |
| 00:57:40 | <@arkiver> | :) |
| 00:57:45 | <@arkiver> | good timing |
| 00:57:56 | <@kiska> | I have my lb that I can use :D |
| 00:58:00 | <purplebot> | Current Projects edited by Wickedplayer494 (+0, Voat is done for) just now -- https://www.archiveteam.org/?diff=45987&oldid=45967 |
| 00:58:00 | <@kiska> | give me a few seconds |
| 00:58:06 | <@arkiver> | awesome, thanks! |
| 00:58:53 | <wickedplayer494> | while I'm at it is there a smackjeeves channel or are we just gonna use -BS here for it? |
| 00:59:44 | <@kiska> | How big do you expect this? |
| 01:00:01 | <@arkiver> | not big |
| 01:00:10 | <@arkiver> | maybe TB max? |
| 01:00:16 | <@kiska> | ack |
| 01:00:18 | <@arkiver> | wickedplayer494: using -bs for now |
| 01:00:38 | <@arkiver> | but we can do a channel, I believe we had #smackedjeeves before |
| 01:00:38 | <OrIdow6> | arkiver: Thanks, joined the org |
| 01:00:57 | <@arkiver> | OrIdow6: alright, pushing and update and then we'll start |
| 01:01:17 | <wickedplayer494> | ok w/e I'll just list -BS for right now to get things going |
| 01:01:21 | <OrIdow6> | Ok |
| 01:01:28 | <@arkiver> | can't thoroughly test everything, but should be good |
| 01:01:34 | <@arkiver> | wickedplayer494: yep |
| 01:01:48 | <@kiska> | rsync://lb1.kiska.pw/smackjeeves/:downloader/ |
| 01:03:20 | <@arkiver> | going up to ID 350k |
| 01:03:30 | <@arkiver> | kiska: yes |
| 01:03:37 | <@arkiver> | is this the hetzner balancer btw? |
| 01:03:54 | <@arkiver> | added |
| 01:04:00 | <purplebot> | File:SmackJeeves - 12-26-20.png uploaded by Wickedplayer494 (+0) just now -- https://www.archiveteam.org/?diff=45989&oldid=0 |
| 01:04:14 | <@kiska> | Yes :D |
| 01:05:14 | <@arkiver> | nice, i want to try it out some day |
| 01:08:30 | <@kiska> | Is this live? |
| 01:08:37 | <@kiska> | Or coming online soon? |
| 01:08:50 | <@kiska> | I assume I'll need to enable the docker repo? |
| 01:09:01 | <purplebot> | File:SmackJeeves logo.png uploaded by Wickedplayer494 (+0) just now -- https://www.archiveteam.org/?diff=45990&oldid=0 |
| 01:13:31 | <@arkiver> | kiska: yes please! |
| 01:13:36 | <@kiska> | Creating |
| 01:15:37 | <@kiska> | Docker image now available at atdr.meo.ws/ArchiveTeam/smackjeeves-grab:latest |
| 01:16:00 | <purplebot> | SmackJeeves created by Wickedplayer494 (+1369, Looks like this one's up next, …) just now -- https://www.archiveteam.org/?diff=45991&oldid=0 |
| 01:26:01 | <purplebot> | Current Projects edited by Wickedplayer494 (+141, SmackJeeves to upcoming) 24 minutes ago -- https://www.archiveteam.org/?diff=45988&oldid=45987 |
| 01:29:52 | <@arkiver> | kiska: perfect |
| 01:29:57 | <@arkiver> | smackjeeves project is started! |
| 01:30:29 | <@arkiver> | im not logged in on the wiki here, if someone can wants to upload a logo and icon of smackjeeves, we'll set that in the tracker |
| 01:31:41 | <@arkiver> | OrIdow6: fully started! |
| 01:32:45 | <wickedplayer494> | arkiver: just did, scroll up :) |
| 01:33:12 | <wickedplayer494> | File:SmackJeeves logo.png |
| 01:33:27 | <@arkiver> | wickedplayer494: hah, yes! |
| 01:34:13 | <@arkiver> | added! |
| 01:44:09 | <@JAA> | jrwr: The page move of SmackJeeves to Smack Jeeves (https://archiveteam.org/index.php?title=Smack_Jeeves&diff=45995&oldid=45994) got caught in the mod queue even though wickedplayer494 is automodded...? |
| 01:49:31 | <wickedplayer494> | is automod on mediawiki supposed to be inclusive of page moves, though? |
| 01:51:49 | <@JAA> | I would expect so because you can create new pages and edit ones, so you could do a manual move. |
| 01:52:02 | <@JAA> | So there's no reason to block proper moves. |
| 01:52:50 | <wickedplayer494> | oh yeah it is supposed to be granted to users with the automoderated role, at least in its default config https://www.mediawiki.org/wiki/Extension:Moderation#User_rights |
| 01:53:05 | <wickedplayer494> | weird |
| 01:58:00 | <purplebot> | Current Projects edited by Wickedplayer494 (+0, And SmackJeeves is a go) 22 minutes ago -- https://www.archiveteam.org/?diff=45992&oldid=45988 |
| 02:00:01 | | Dallas quits [Client Quit] |
| 02:02:00 | <purplebot> | SmackJeeves edited by Wickedplayer494 (+2, It's started), Wickedplayer494 (+3, Spaces, because the name does have …), moved by Wickedplayer494 (+0, There is indeed a space in its …) 19 minutes ago -- https://www.archiveteam.org/?diff=45995&oldid=45991 |
| 02:03:40 | | Dallas (Dallas) joins |
| 02:36:49 | | JimBilliams joins |
| 02:37:37 | <Craigle> | Just in case anyone else wants to copy/paste, the docker image is at: atdr.meo.ws/archiveteam/smackjeeves-grab:latest |
| 02:38:06 | <Craigle> | Has to be all lowercase, or Docker throws a fit |
| 02:38:57 | | a_spook__ joins |
| 02:46:26 | <a_spook__> | Hi, I don't know if it matters, but there's a tiny error-filled comic on Smack Jeeves I'm aware of: https://www.smackjeeves.com/discover/articleList?titleNo=73484 (old url: http://karecoffee.smackjeeves.com/) |
| 02:46:28 | <a_spook__> | Today and yesterday, only pages 2 and 3 seemed to work. The images for the other pages have been unavailable for years anyways because they were hosted on Imageshack, but at the time the webpages just had broken comic images. Now its seems they are entirely inaccessible with an Internal Server Error message. The comments api still returns comments for these pages though. |
| 03:11:15 | <@kiska> | arkiver: We are rate limiting? |
| 03:30:26 | <OrIdow6> | Nice to see you haven't broken everything yet |
| 03:32:17 | <OrIdow6> | a_spook__: That item should just fail after exponential backoff on the 500 runs out |
| 03:33:29 | <OrIdow6> | Looks like you or someone else has SPNd it anyhow |
| 03:34:46 | <a_spook__> | OrIdow6: Ok, I just wanted to bring it up in case it mattered at all in the grand scheme of things :p and yes that was likely me, haha |
| 03:37:58 | <OrIdow6> | If you know of any other abnormalities, feel free to bring them up here too |
| 03:38:00 | | JimBilliams quits [Remote host closed the connection] |
| 03:42:14 | | etnguyen03 (etnguyen03) joins |
| 03:46:56 | <@kiska> | 478=403 https://images.smackjeeves.com/article/743/110743/7/110743_7_1_1574114628354.jpg |
| 03:46:57 | <@kiska> | Oh? |
| 03:49:12 | <OrIdow6> | What item is that? |
| 03:51:41 | <OrIdow6> | kiska |
| 03:55:08 | <@kiska> | comic:110743 |
| 03:59:12 | <OrIdow6> | Thanks |
| 04:10:30 | <OrIdow6> | So last and early this month, according to the CDX server, someone has SPNd *a bunch* of pages that give 403s on the images; and those same URLs give 403s now, too |
| 04:11:23 | <OrIdow6> | Only 128 200s, but 950 403s, for images.smackjeeves.com, in the WBM, apparently |
| 04:18:03 | <OrIdow6> | As far as I can tell, the only commonalities between those SPN'd 403s are that: |
| 04:18:22 | <OrIdow6> | - They are all (or all the ones I checked are) from comics related to either Pokemon or Kirby |
| 04:18:52 | <OrIdow6> | - All use a non-"legacy" image URL, whereas most comics I've seen use the "legacy" one |
| 04:20:41 | <OrIdow6> | I'll note that the comic kis ka found appears to be a "Sonic the Hedgehog" comic, which is in that same general category |
| 04:21:32 | <OrIdow6> | Anyhow, if nothing changes here (i.e. if a static sprinkling of image URLs continues to give 403s), I'll add something to skip them |
| 04:22:34 | <OrIdow6> | But one possibility for how the SPNer "selected" comics that give errors is that this is some sort of strange rate-limiting that blocked SPN |
| 04:23:08 | <OrIdow6> | In which case we should see those increase in frequency |
| 04:35:11 | <OrIdow6> | Only see 2 200s in the WBM on those URLs, both with an "expiration" of a Unix timestamp near capture (early 2020) and a "token" that is a 64-bit hex string |
| 04:35:15 | <OrIdow6> | *64-char |
| 04:35:30 | <OrIdow6> | (So 32 bytes) |
| 04:40:46 | | qw3rty__ joins |
| 04:44:23 | | qw3rty_ quits [Ping timeout: 252 seconds] |
| 05:01:47 | | etnguyen03 quits [Remote host closed the connection] |
| 05:06:27 | | Ryz quits [Remote host closed the connection] |
| 05:07:12 | | Ryz (Ryz) joins |
| 06:25:42 | | godane (godane) joins |
| 06:48:18 | <Jake> | Aborted on this page for the "internal server error" https://www.smackjeeves.com/discover/detail?titleNo=111799&articleNo=137 |
| 06:57:24 | | tzt quits [Ping timeout: 240 seconds] |
| 06:58:11 | | a_spook__ quits [Client Quit] |
| 07:16:25 | | Arcorann (Arcorann) joins |
| 07:27:22 | | Doranwen joins |
| 07:27:24 | <OrIdow6> | Looks similar to the previous 500 |
| 07:27:34 | | Doranwen is now authenticated as Doranwen |
| 07:29:00 | <OrIdow6> | Oh, yeah, SmackJeeves doesn't actually send anything but a 200 |
| 07:34:43 | <OrIdow6> | Glad ti aborts anyhow |
| 07:36:38 | <OrIdow6> | (Lines 238-242) |
| 07:38:14 | <OrIdow6> | I think there is enough time left that we can retry these ones that give errors in a few days, then add exceptions (or, in this case, remove the detection) if that doesn't work |
| 07:46:20 | | Mateon1 quits [Remote host closed the connection] |
| 08:19:39 | | Mateon1 joins |
| 09:28:08 | | Ryz quits [Remote host closed the connection] |
| 09:28:55 | | Ryz (Ryz) joins |
| 10:01:48 | <Arcorann> | Is there a priority list of Docker-based projects somewhere? I want to test my setup |
| 10:24:03 | <@HCross> | are there limits on smackjeeves |
| 10:24:08 | <@HCross> | or can i go full harry mode on them |
| 10:26:59 | <@kiska> | There seems to be a rate limit in place |
| 10:27:22 | <@kiska> | Arcorann do smackjeeves or url grab |
| 10:29:29 | <Arcorann> | How do I find the image address |
| 10:35:29 | <jodizzle> | Arcorann: It's atdr.meo.ws/archiveteam/{grabname} |
| 10:35:42 | <Arcorann> | Thanks |
| 10:35:46 | <jodizzle> | For for the URLs grab, it's atdr.meo.ws/archiveteam/urls-grab |
| 10:36:34 | <jodizzle> | grabnames are based on the repo names on the AT GitHub |
| 11:19:43 | | ragu joins |
| 11:20:02 | | ragu is now authenticated as ragu |
| 11:29:56 | | VADemon joins |
| 12:38:15 | | AlsoHP_Archivist joins |
| 12:41:04 | | HP_Archivist quits [Ping timeout: 240 seconds] |
| 12:45:44 | | AlsoHP_Archivist quits [Ping timeout: 240 seconds] |
| 12:46:00 | | AlsoHP_Archivist joins |
| 12:47:30 | <EggplantN_> | so ~ 2 days for smackjeeves at current concurrency not bad |
| 12:50:26 | <EggplantN_> | i'd say yolo go faster but thats fine for now |
| 12:50:29 | <EggplantN_> | but faster is better |
| 12:55:35 | <atphoenix> | I'm seeing a lot of tracker rate limiting |
| 13:00:50 | <Arcorann> | As am I |
| 13:04:34 | <atphoenix> | I'm also seeing a lot of EggplantN_ items roll by, so at least someone is making progress :) |
| 13:05:17 | <EggplantN_> | I got a server that's going away in like 4 days time so I just made it do this can't do GH due to lack of IPs |
| 13:14:17 | <atphoenix> | I don't mind being low on tracker rankings. I am somewhat bandwidth limited. I have added a worker to assist, especially when every IP counts, and to be available as a reserve capacity in case rate limits are lifted to where every last worker is needed. |
| 13:25:46 | | VADemon quits [Client Quit] |
| 13:31:16 | <Matthww> | EggplantN_ hey share some work to other :P |
| 13:31:22 | <Matthww> | others |
| 13:31:44 | <EggplantN_> | By all means defeat me. :P |
| 13:32:18 | <Matthww> | I'm trying but I'm hitting a tracker rate limit :) |
| 13:34:41 | <mgrandi> | Did we see that voat is shutting down? |
| 13:35:59 | <mgrandi> | Oh I guess it's already dead, but there is https://searchvoat.co/forum/ |
| 13:37:21 | <themadpro> | #scrapevoat |
| 13:37:39 | <themadpro> | I think a sizable chunk of it was saved |
| 13:38:09 | <mgrandi> | From what I'm seeing, it seems that forum was an archive itself |
| 13:45:28 | | AlsoHP_Archivist quits [Client Quit] |
| 13:45:44 | | HP_Archivist (HP_Archivist) joins |
| 13:56:44 | | Arcorann quits [Ping timeout: 240 seconds] |
| 14:00:42 | <@arkiver> | yeah we got a good percentage of voat |
| 14:15:02 | | paul2520 (paul2520) joins |
| 14:17:37 | <paul2520> | I know it's on the deathwatch list, so let me know if I should check back in a couple of weeks (the site goes down Feb 1). But any update on https://community.fantasyflightgames.com/? I've been running a wget that's at 62 GB and counting and just curious if this sounds plausible or if something wonky might be happening. |
| 14:18:45 | | etnguyen03 (etnguyen03) joins |
| 14:18:56 | <paul2520> | it's 10s of thousands of posts at least over at least a decade... but I'd still think mostly text. But maybe there are lots of photos too? I was going to dig deeper into what I've pulled but TBH haven't rrally ever debugged wget output, other than a sanity check |
| 14:19:06 | <paul2520> | *really |
| 14:22:41 | <mgrandi> | Is it wget-lua and downloading a war? |
| 14:22:53 | <mgrandi> | Warc* |
| 14:23:20 | <paul2520> | no. I'm just pulling the files for myself, and potentially to compress and upload to IA |
| 14:25:21 | <paul2520> | I take it you're recommending the wget-lua/WARC approach? |
| 14:26:12 | <mgrandi> | Or wpull, but yeah, those are probably better for uploading. If you are using regular lua then you probably just have a folder attempting to mirror the website |
| 14:27:05 | <paul2520> | valid. but still is 60+ GB unsurprising for a forum site like that? |
| 14:28:02 | <mgrandi> | In my experience, it depends on the contents. Modern website love to have the dumb javascript scripts so you might be downloading a ton of copies of that which is inflating the size |
| 14:29:00 | <paul2520> | thanks, that's at least affirming. I'll probably let it run longer, but look at the approach you suggested. I'll probably be back with questions. |
| 14:31:17 | <mgrandi> | A few regexes of stuff to ignore could probably fix it, but then normal wget doesn't save it's place so it will have to restart, better just leave it for now |
| 14:32:13 | <paul2520> | sure. I figured archivebot will probably crawl it anyway as the date gets closer & will handle some of that better than me |
| 14:41:10 | <@arkiver> | repo domain-grab will hold a general project to archive domains up to a certain depth |
| 14:41:11 | <@arkiver> | we'll use that for .eu domains |
| 14:41:48 | <paul2520> | 👍🏻 |
| 14:42:54 | <@arkiver> | paul2520: nope those didnt show up for me |
| 14:43:15 | <paul2520> | just a thumbs up |
| 14:58:45 | | paul2520 quits [Remote host closed the connection] |
| 16:16:35 | | hooway joins |
| 17:14:32 | | hooway quits [Client Quit] |
| 17:21:36 | | hooway joins |
| 17:40:03 | | tzt joins |
| 18:18:05 | <EggplantN_> | Matthww I see you climbing >_> |
| 18:20:34 | <Matthww> | EggplantN_ not fast enough |
| 18:20:42 | <EggplantN_> | I believe |
| 18:26:12 | <atphoenix> | The question is who has more workers, and thus can increase their chance of grabbing an available job from the tracker sooner. But certainly don't setup more workers than your infra can reasonably handle such that they are competing with each other inside your own infra. |
| 18:32:42 | <Matthww> | The tracker is just favoring Eggplant :( |
| 19:09:17 | <EggplantN_> | Trust me it doesnt favour me |
| 19:12:46 | <Matthww> | your just hitting it with atleast 2k+ workers aren't you? |
| 19:33:40 | <atphoenix> | I'm up to the 5 minute tracker retry interval |
| 19:52:30 | <OrIdow6> | Once the .eu grab proper starts there will be plenty of jobs for your workers |
| 19:57:50 | <@HCross> | is there an eta on that? |
| 19:58:55 | <OrIdow6> | If there is, I don't know it - I only worked on the discovery (i.e. WHOIS scrape) half |
| 20:01:20 | <Matthww> | URLS is also here with alot of jobs |
| 20:09:11 | <brad> | paul2520: I can confirm there a multiple threads on the site https://community.fantasyflightgames.com/ with lots of pictures. That’s one of the reasons we want to save it. And one of the reasons I’d like to do whatever I can to help. I’m working on setting up my own warrior, but I don’t think I’ll be allowed to contribute to the archivebot project for a while. |
| 20:10:07 | <thuban> | he's not on at the moment |
| 20:10:48 | <thuban> | fwiw it's already been run through archivebot http://archive.fart.website/archivebot/viewer/job/5l4qk |
| 20:12:15 | <@JAA> | s/been run/running/ |
| 20:12:42 | <thuban> | oh shit, so it is. not sure why i thought it was done |
| 20:13:11 | <@JAA> | It got blocked with 403s several times already, so it's completely unclear whether that will get anywhere overall. |
| 20:13:27 | <thuban> | yeah, ouch |
| 20:13:30 | | Doranwen quits [Ping timeout: 252 seconds] |
| 20:15:08 | <thuban> | warrior candidate? normal crawl for discovery and then send out topics as items? |
| 20:15:19 | | Doranwen joins |
| 20:15:47 | <@JAA> | Yeah, possibly. But let's postpone that to January since there are other things shutting down in the next few days. |
| 20:15:53 | <thuban> | yeah |
| 20:27:46 | | dunnousername joins |
| 20:39:37 | | dunnousername quits [Client Quit] |
| 21:43:04 | | tzt quits [Ping timeout: 240 seconds] |
| 22:26:38 | <@arkiver> | yeah be prepared for .eu project, will be a ton of work for all of you |
| 22:59:45 | | godane quits [Client Quit] |
| 23:32:54 | | Doranwen is now authenticated as Doranwen |
| 23:57:21 | | Connection closed. |