HomeSearchPrevious dayNext day

00:00:31ThreeHeadedMonkey quits [Ping timeout: 265 seconds]
00:00:57<@JAA>yawkat: Yeah, I can't really recommend using my PR though (but if you do, let me know how well it works please). I'd suggest either using wpull 1.2.3 or setting the concurrency from a plugin. 1.2.3 is very stable in my experience.
00:02:02ThreeHeadedMonkey (ThreeHeadedMonkey) joins
00:04:41<@JAA>And by that I mean: my PR should probably work, but it's not finalised yet, more changes are needed, there's a performance penalty, and the version isn't bumped, so if you upload that anywhere, please make it absolutely clear which code you were using.
00:05:50OrIdow6 quits [Ping timeout: 265 seconds]
00:19:37isthisthedarkweb quits [Remote host closed the connection]
00:33:34sliccricc (sliccricc) joins
00:37:50hooway quits [Client Quit]
00:38:58onetruth joins
00:44:58<@arkiver>WEBS PROJECT COMING UP #webbed
00:46:49<@arkiver>THANKS TO tech234a ^
00:57:01<sliccricc>damn, they gave their employees free lunch every day?
00:57:51<@arkiver>guess they gave them free restroom visits as well
01:08:06<thuban>jodizzle, revi: archivebot job for the catalog entries running now
01:08:51<thuban>(to be clear, all of those urls are going to 302, but i understand that ab handles that properly)
01:16:56<thuban>jodizzle: provided there are no token problems, i think what i'd like to do is do a similar processing/!ao pass to get the xml, then the page images (again for provenance), and then process the issues into pdfs and upload them to ia as their own items (for browsability). thoughts?
01:21:49OrIdow6 (OrIdow6) joins
01:22:32OrIdow6 quits [Remote host closed the connection]
01:22:59OrIdow6 (OrIdow6) joins
01:40:31BlueMaxima quits [Remote host closed the connection]
02:07:38godane quits [Ping timeout: 265 seconds]
02:23:49lunik1 quits [Read error: Connection reset by peer]
02:24:04lunik1 joins
02:55:52<jodizzle>thuban: That all sounds fine to me.
02:56:47<jodizzle>Having PDFs on IA sounds especially nice
02:57:41<thuban>jodizzle: good to know. unfortunately ab didn't actually save anything from the list i passed it, even though i've confirmed the urls are working--do you know whether there's some problem with redirects, maybe in combination with archive-only?
03:02:26<jodizzle>Hm, I think it depends on how it redirects. I always forget the rules on !ao and redirects. Might need to ask Ryz or JAA.
03:04:04<jodizzle>The case I have from scanning some logs is that if the redirect is like https://example.org/foo/something -> https://example.org/bar/something, only the first is grabbed
03:04:13<thuban>Ryz spotted the issue and said they weren't sure; haven't heard from JAA
03:04:53<jodizzle>So if the redirect is to a different directory on the same domain, it redirect target is not grabbed.
03:04:59<jodizzle>But I'm not totally sure of the rules
03:05:07<thuban>(i tried reading the source, but nothing leapt out at me and there are a lot of layers of indirection there)
03:05:40<thuban>huh. that seems odd; i'm curious what the rationale was
03:10:22<jodizzle>If true, would that explain what was happening with your job?
03:11:03<thuban>yes, those urls are redirects to a different directory under the same domain.
03:13:20<Ryz>I think the logic is like this, being "!ao <" or "!ao", if the link goes to a redirect and goes to something that doesn't match any of the links, it gets treated as an offsite link
03:13:28<Ryz>"!ao" jobs don't grab offsite goods
03:14:26<thuban>i guess i can just curl -I them all
03:17:43ThreeHeadedMonkey quits [Ping timeout: 265 seconds]
03:19:08<Ryz>There could be another way but I rather wait for JAA to respond
03:19:25ThreeHeadedMonkey (ThreeHeadedMonkey) joins
03:19:29<jodizzle>Ryz: That could make sense, except I'm also seeing from the logs that redirect targets _are_ grabbed if the target is offsite, e.g., https://example.com/ -> https://example2.com/ would grab both. But again, not totally sure.
03:20:39<jodizzle>thuban: You could also pull the redirect targets from the logs I'm pretty sure, once they're uploaded. But `curl -I` might be faster.
03:26:49<thuban>i'm running it now, in case; will still be a while
03:39:41DogsRNice quits [Read error: Connection reset by peer]
03:40:10thuban quits [Client Quit]
03:41:54BlueMaxima joins
03:57:19godane joins
04:01:01daalekz (daalekz) joins
04:04:49qw3rty_ joins
04:08:34qw3rty__ quits [Ping timeout: 244 seconds]
04:27:42OrIdow6 quits [Ping timeout: 264 seconds]
04:28:09godane1 joins
04:30:42godane quits [Ping timeout: 265 seconds]
04:37:46daalekz1 (daalekz) joins
04:39:24daalekz quits [Ping timeout: 265 seconds]
05:13:59etnguyen03 quits [Client Quit]
05:33:52OrIdow6 (OrIdow6) joins
05:55:51daalekz1 quits [Read error: Connection reset by peer]
06:31:13<flashfire42>https://nopaystation.com/ at some point I think we should archive this. Its piracy but downloads directly from the Playstation CDN could be useful to grab for whenever Sony decides to yank it down
06:32:52<flashfire42>I would call it a non urgent suggestion for archival 😅
06:37:16OrIdow6 quits [Ping timeout: 252 seconds]
06:37:52Wayward (wayward) joins
07:15:51Hackerpcs quits [Quit: Hackerpcs]
07:17:37Hackerpcs (Hackerpcs) joins
07:20:14thuban joins
09:06:41Eighty quits [Ping timeout: 244 seconds]
09:13:35Eighty (Eighty) joins
09:40:24Mineroboter joins
09:40:57Mineroboter quits [Client Quit]
09:58:05hooway joins
10:03:27spirit joins
10:15:43BlueMaxima quits [Client Quit]
11:04:40daalekz (daalekz) joins
11:06:51<avoozl>flashfire42: wow that is very friendly for downloading.. the TSV has the full urls in there
11:09:32<avoozl>flashfire42: 18.7TB just adding the filesizes on the PS3 games part
11:09:45<avoozl>(if I'm reading this correctly)
11:23:07OrIdow6 (OrIdow6) joins
11:30:43rsn_ joins
11:32:54rsn quits [Ping timeout: 244 seconds]
11:45:38daalekz quits [Read error: Connection reset by peer]
13:53:41ragu__ joins
13:57:03ragu_ quits [Ping timeout: 244 seconds]
14:10:22ragu_ joins
14:11:03etnguyen03 (etnguyen03) joins
14:11:40Arcorann quits [Ping timeout: 265 seconds]
14:11:42ragu__ quits [Read error: Connection reset by peer]
14:11:56ragu__ joins
14:15:25ragu_ quits [Ping timeout: 252 seconds]
14:16:33ragu_ joins
14:19:49ragu__ quits [Ping timeout: 252 seconds]
14:29:21@EggplantN is now known as @EggplantN2
14:30:19EggplantN joins
14:30:27EggplantN quits [Changing host]
14:30:27EggplantN (EggplantN) joins
14:30:27@ChanServ sets mode: +o EggplantN
14:37:04@EggplantN quits [Client Quit]
14:37:17EggplantN joins
14:37:59EggplantN quits [Client Quit]
14:38:14EggplantN (EggplantN) joins
14:38:14@ChanServ sets mode: +o EggplantN
14:44:08@EggplantN quits [Client Quit]
14:44:26EggplantN (EggplantN) joins
14:44:26@ChanServ sets mode: +o EggplantN
14:51:17@EggplantN quits [Client Quit]
14:51:37EggplantN (EggplantN) joins
14:51:37@ChanServ sets mode: +o EggplantN
15:24:03ragu__ joins
15:27:33ragu_ quits [Ping timeout: 265 seconds]
15:27:48ragu_ joins
15:29:58ragu__ quits [Ping timeout: 265 seconds]
16:07:00<purplebot>Deathwatch edited by Squidboy (+142, /* Other endangered species and …) just now -- https://www.archiveteam.org/?diff=46287&oldid=46285
16:08:00<purplebot>FTP/List edited by Squidboy (+34, /* Available FTPs */ adding ftp.fidonet.io) just now -- https://www.archiveteam.org/?diff=46288&oldid=45890
16:08:00<purplebot>Alive... OR ARE THEY edited by VDD (+81, /* Alarm */) just now -- https://www.archiveteam.org/?diff=46289&oldid=46202
16:08:00<purplebot>Webs edited by Tglass (+795, Added info from tech234a) just now -- https://www.archiveteam.org/?diff=46290&oldid=46088
16:08:26godane joins
16:09:36godane1 quits [Ping timeout: 265 seconds]
16:09:49godane1 joins
16:13:00<purplebot>Flash edited by Tglass (-5) just now -- https://www.archiveteam.org/?diff=46291&oldid=46021
16:13:00<purplebot>.eu domains edited by Tglass (-5) just now -- https://www.archiveteam.org/?diff=46292&oldid=46211
16:13:00<purplebot>Chrome Web Store edited by Tglass (-3) just now -- https://www.archiveteam.org/?diff=46293&oldid=45950
16:13:07godane quits [Ping timeout: 252 seconds]
16:33:00<purplebot>Network.hu edited by Bzc6p (+295, first stage completed) just now -- https://www.archiveteam.org/?diff=46294&oldid=45815
16:34:27vitzli (vitzli) joins
16:34:43vitzli quits [Remote host closed the connection]
17:11:28<@JAA>jodizzle, thuban, Ryz: If a redirect on the initial URL (whether that's an individual URL on !a or !ao or one from a list for !ao < or !a <) has different parent path components, the redirect isn't followed and only the 3xx response is archived. This is due to the --no-parent option and has nothing to do with offsite links. Redirects to other hosts *are* followed. Which yeah, is pretty unintuitive.
17:13:52<@JAA>Actually, this kind of applies to all URLs, not just initial ones. But redirects *within* the same parent path are followed. For example, if you !a https://example.org/foo/ and it comes across /foo/bar/baz.html redirecting to /foo/baz.html, that is followed. If that redirect goes to /baz.html though, it isn't.
17:14:36<@JAA>So redirects to other hosts are followed, but redirects within the same domain are only followed if they are in the same path as the initial URL.
18:15:37onetruth quits [Read error: Connection reset by peer]
19:32:46fionera (Fionera) joins
19:34:13<@Fusl>https://quake.ingame.de/forum/ closing down in 3 days, someone queue everything please?
19:38:35Barto quits [Ping timeout: 244 seconds]
19:38:53<jodizzle>Fusl: queued in AB
19:39:30Barto (Barto) joins
19:51:56Barto quits [Ping timeout: 265 seconds]
19:55:56DogsRNice (Webuser299) joins
19:56:01Barto (Barto) joins
20:00:35nerdguy1138 (nerdguy1138) joins
20:09:09<spirit>Fusl: oh noes, the seksual :ugly:
20:33:17BlueMaxima joins
20:33:50Kenshin quits [Quit: ZNC - http://znc.in]
20:46:45atphoenix is now known as Alexa
20:47:23Alexa is now known as atphoenix
20:51:16Kenshin joins
21:41:05benjins quits [Remote host closed the connection]
21:41:53benjins joins
21:43:14<@HCross>I grabbed all the BT Community Web Kit sites off the index from the Deathwatch page, it's beginning the slow march into the IA
21:56:08<billy549>Awesome c:
21:59:26<billy549>HCross: should it be marked somewhere that you've done that?
21:59:30<billy549>might be a good idea to point out on the Deathwatch that you've sent it along to IA, so we dont capture it again later
21:59:56<@HCross>when I flipped through the sites to check them, some of them were even being updated _THIS WEEK_
22:00:07<@HCross>so I want to go back again closer to the shutoff of May 24th and grab them again
22:00:42<billy549>fair c:
22:01:07<billy549>also "September 30: Khan Academy will remove their MCAT preparation material[IA•Wcite•.today•MemWeb].[5] " - we could get that now surely?
22:01:25<billy549>theyre not updating it, so if we threw it into archivebot now and then checked it closer to the time, it'd be finished?
22:02:41<billy549>same for the Tom Lehrer 2024 stuff
22:22:47benjinsmith joins
22:24:40benjins quits [Ping timeout: 265 seconds]
23:21:29<thuban>billy549: lehrer has been done https://archive.fart.website/archivebot/viewer/job/4vxsv
23:24:04jut quits [Quit: WeeChat 3.0]
23:26:18<thuban>(a lot of stuff on the deathwatch page that gets handled by archivebot doesn't get updated to say so--i plan to fix that at some point, but per some discussion with JAA i'm waiting to see the new viewer-indexer)
23:27:20<thuban>JAA: thanks, i get it now.
23:27:27<thuban>redirected list is done so i'll just pop it in
23:29:30jut (jut) joins
23:31:42<@JAA>thuban: The Deathwatch page isn't supposed to say whether something was archived.
23:32:20<@JAA>If we were to add that to every entry, it'd get even more unwieldy than it already is.
23:33:33<@JAA>Noteworthy sites have their own pages to list the details, and smaller sites, well, not sure if there's a good way to document that.
23:33:38<thuban>we talked about this
23:34:00<thuban>i don't think a (truncated?) job id that links to the viewer would be excessive
23:34:26<@JAA>It might work in footnotes, but not in the main text.
23:34:38<@JAA>And yeah, I remember now that we discussed that before.
23:35:50Arcorann (Arcorann) joins
23:36:00<thuban>special collapsible footnotes section, with visual distinction in text from other references?
23:36:20<thuban>would need some mediawiki magic, but
23:46:44<@JAA>Yes. No magic needed, that's already supported and used on some pages (can't remember any examples right now though).
23:48:25<thuban>oh good to know, i'll go poking around.
23:48:53<@JAA>https://www.mediawiki.org/wiki/Help:Cite#Grouped_references
23:52:51<billy549>thuban: ty for let me know
23:54:11<thuban>iirc khan academy is both video-based and js-heavy, so not a good candidate for archivebot
23:54:34Barto quits [Ping timeout: 252 seconds]

HomeSearchPrevious dayNext day