HomeSearchPrevious dayNext day

00:03:35etnguyen03 (etnguyen03) joins
00:24:28wessel1512 joins
00:35:38BlueMaxima joins
00:52:29etnguyen03 quits [Ping timeout: 252 seconds]
01:06:07etnguyen03 (etnguyen03) joins
01:17:35<project10>so I went looking at my 135G zowa warc on IA. Found it at https://archive.org/download/archiveteam_zowa_20230923012400_df2de1d0 but also at https://archive.org/download/archiveteam_zowa_20230924040422_7fbffef8. Why would there be two copies, uploaded on different days with different filenames/timestamps?
01:18:10<@JAA>Probably the item was reclaimed and completed twice (or more times).
01:18:40cascode quits [Ping timeout: 265 seconds]
01:18:59<project10>oh, interesting. I assume IA won't dedupe/reap these and they will show on the WBM as captures on different days?
01:19:10<@JAA>Yes
01:20:07cascode joins
01:20:14<project10>ok, good to know the total size displayed on the tracker is not necessarily indicative of the amount shipped to IA
01:27:08etnguyen03 quits [Ping timeout: 252 seconds]
01:33:48haha joins
01:34:38haha quits [Remote host closed the connection]
01:35:52etnguyen03 (etnguyen03) joins
02:21:17cascode quits [Read error: Connection reset by peer]
02:21:40cascode joins
02:36:14Wohlstand quits [Client Quit]
02:36:40Wohlstand (Wohlstand) joins
02:40:20Wohlstand quits [Client Quit]
02:43:35etnguyen03 quits [Ping timeout: 252 seconds]
02:49:12etnguyen03 (etnguyen03) joins
03:06:04<anarcat>so this debian developer died https://abrahamraji.in/
03:06:39<anarcat>i'm going to crawl that site and https://wiki.abrahamraji.in/
03:06:50<anarcat>there's also https://www.youtube.com/@abrahamraji3699/ i'm not sure what to do with
03:07:34<anarcat>there's also https://gitlab.com/avron https://aana.site/@avronr - same
03:08:45<anarcat>oh looks like pabs already did it
03:08:45mindstrut1 quits [Read error: Connection reset by peer]
03:09:04<pabs>anarcat: yeah, well covered
03:09:07mindstrut1 joins
03:09:21<pabs>anarcat: did the youtube in #down-the-tube
03:09:38<anarcat>thanks
03:09:41<anarcat>so sad
03:10:02<pabs>the mastodon I don't think can be saved, too much JS and AT doesn't save fediverse I thought
03:19:13<anarcat>ack
03:20:03<pabs>if we wanted to, this could be repurposed for that https://github.com/jwilk/zygolophodon
03:23:22dumbgoy quits [Ping timeout: 265 seconds]
04:09:55dumbgoy joins
04:10:06Exorcism quits [Remote host closed the connection]
04:10:44Exorcism (exorcism) joins
04:17:26Exorcism quits [Remote host closed the connection]
04:17:54Exorcism (exorcism) joins
04:24:45DogsRNice quits [Read error: Connection reset by peer]
04:35:28Exorcism quits [Remote host closed the connection]
04:36:09Exorcism (exorcism) joins
04:41:54icedice quits [Read error: Connection reset by peer]
04:42:17icedice (icedice) joins
04:46:20Exorcism quits [Remote host closed the connection]
04:47:10Exorcism (exorcism) joins
04:50:44etnguyen03 quits [Client Quit]
04:54:20appledash quits [Remote host closed the connection]
04:54:54cascode quits [Read error: Connection reset by peer]
04:55:05cascode joins
05:08:01Island quits [Read error: Connection reset by peer]
05:46:35Earendil7 quits [Quit: Leaving]
05:48:00Earendil7 (Earendil7) joins
05:48:15magmaus3 quits [Client Quit]
05:50:05magmaus3 (magmaus3) joins
05:51:11decky_e_ joins
05:54:26decky quits [Ping timeout: 252 seconds]
05:56:33BlueMaxima quits [Read error: Connection reset by peer]
06:08:40cascode quits [Ping timeout: 265 seconds]
06:09:05nepeat (nepeat) joins
06:09:50cascode joins
06:10:29tzui joins
06:11:57thunder_steak joins
06:18:45tzui quits [Remote host closed the connection]
06:40:57Dango360 quits [Read error: Connection reset by peer]
06:42:35cascode quits [Read error: Connection reset by peer]
06:42:55Arcorann (Arcorann) joins
06:43:11cascode joins
06:46:08Naruyoko quits [Ping timeout: 252 seconds]
06:48:03Naruyoko joins
07:00:08nfriedly quits [Remote host closed the connection]
07:06:36Unholy23613166180851599 (Unholy2361) joins
07:10:49ffff joins
07:17:09decky joins
07:20:41decky_e_ quits [Ping timeout: 265 seconds]
07:40:13VerifiedJ quits [Quit: Ping timeout (120 seconds)]
07:40:22lunik173 quits [Quit: Ping timeout (120 seconds)]
07:40:26VerifiedJ (VerifiedJ) joins
07:40:34lunik173 joins
08:10:10<flashfire42>I dunno what happened but I am seeing a lot more movement across the warrior projects
08:15:19lukash91 joins
08:16:20lukash9 quits [Ping timeout: 252 seconds]
08:20:08lukash91 quits [Ping timeout: 265 seconds]
08:56:29lun4 quits [Ping timeout: 252 seconds]
08:56:29ave quits [Ping timeout: 252 seconds]
08:58:13lukash9 joins
09:06:44ave (ave) joins
09:06:50lun4 (lun4) joins
09:16:58icedice quits [Client Quit]
09:24:45ymgve_ joins
09:27:17ymgve quits [Ping timeout: 252 seconds]
09:32:44Exdetransitioner (exdetransitioner) joins
09:35:02<Exdetransitioner>does there anybody has an access to genspect's chatroom?
09:35:06<Exdetransitioner>https://www.dailydot.com/debug/genspect/
09:35:28<Exdetransitioner>they claim to run a semi-secret forum where they discuss anti-trans extermist talking points
09:46:21IRC2DC joins
09:49:17IRC2DC quits [Remote host closed the connection]
09:52:27parfait_ quits [Ping timeout: 265 seconds]
09:54:47tsyesika quits [Ping timeout: 252 seconds]
10:00:01railen63 quits [Remote host closed the connection]
10:00:20railen63 joins
10:11:40Exdetransitioner quits [Client Quit]
10:15:39imer quits [Ping timeout: 265 seconds]
10:16:08nfriedly joins
10:29:40<thunder_steak>how is decided how often a website will be crawled/snapshotted? e.g. http://zwisler.de/
10:42:46icedice (icedice) joins
10:50:46beario_ joins
10:53:38beario quits [Ping timeout: 252 seconds]
11:10:37Peroniko joins
11:12:20RetiredTurtle quits [Ping timeout: 252 seconds]
11:31:18shreyasminocha quits [Remote host closed the connection]
11:31:18thehedgeh0g quits [Remote host closed the connection]
11:31:18evan quits [Remote host closed the connection]
11:31:22evan joins
11:31:25shreyasminocha (shreyasminocha) joins
11:31:25thehedgeh0g (mrHedgehog0) joins
11:38:33icedice quits [Remote host closed the connection]
11:38:57icedice (icedice) joins
11:58:02JohnnyJ quits [Quit: Ping timeout (120 seconds)]
11:58:24JohnnyJ joins
12:00:23JohnnyJ quits [Client Quit]
12:17:54qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins
12:23:51imer (imer) joins
13:04:20AmAnd0A quits [Ping timeout: 265 seconds]
13:05:07AmAnd0A joins
13:12:37RetiredTurtle joins
13:12:54railen64 joins
13:13:37kiryu quits [Remote host closed the connection]
13:14:49kiryu (kiryu) joins
13:15:27Peroniko quits [Ping timeout: 265 seconds]
13:16:25railen63 quits [Ping timeout: 265 seconds]
13:16:51lflare quits [Read error: Connection reset by peer]
13:19:38lflare (lflare) joins
13:21:14etnguyen03 (etnguyen03) joins
13:33:07imer quits [Killed (NickServ (GHOST command used by imer7))]
13:33:14imer (imer) joins
13:40:37imer quits [Killed (NickServ (GHOST command used by imer7))]
13:40:44imer (imer) joins
13:42:41<pabs>thunder_steak: in what context? for ArchiveBot, usually when the site is closing or there is another reason for doing it
13:43:28imer quits [Killed (NickServ (GHOST command used by imer0))]
13:43:35imer (imer) joins
13:47:40railen64 quits [Remote host closed the connection]
13:47:56railen64 joins
13:51:13Arcorann quits [Ping timeout: 265 seconds]
13:51:22Wohlstand (Wohlstand) joins
13:53:21toss (toss) joins
14:14:55vukky quits [Quit: @ERROR: max connections (-1) reached -- try again later]
14:15:17vukky (vukky) joins
14:17:24<thunder_steak>pabs e.g. http://zwisler.de/ has been snapshotted multiple times but with no constant frequency
14:19:08vukky quits [Client Quit]
14:19:25vukky (vukky) joins
14:24:25<pabs>I guess you mean in web.archive.org. if you click the "About this capture" thing on the top right, you can get some idea
14:25:00<pabs>as you can see here, zero of those were ArchiveTeam ArchiveBot snapshots: https://archive.fart.website/archivebot/viewer/?q=zwisler.de
14:39:45<@JAA>My Canucks forums topic page qwarc grab finished earlier today without any obvious issues.
14:40:47etnguyen03 quits [Ping timeout: 252 seconds]
14:41:41Exorcism quits [Read error: Connection reset by peer]
14:43:20<@JAA>196068 We could not find that topic.
14:43:20<@JAA>21026 You do not have permission to view this topic.
14:43:20<@JAA>122653 There are no posts to show
14:43:28Exorcism (exorcism) joins
14:43:39<@JAA>The rest of the 409104 topic IDs were retrieved.
14:46:18DogsRNice joins
14:49:19railen69 joins
14:53:05railen64 quits [Ping timeout: 265 seconds]
14:59:13thunder_steak quits [Remote host closed the connection]
15:01:35etnguyen03 (etnguyen03) joins
15:07:50Island joins
15:24:30RetiredTurtle quits [Ping timeout: 265 seconds]
15:30:01guest9234 joins
15:31:33icedice quits [Client Quit]
15:31:52<@JAA>I got approximately 6007327 posts, which matches the homepage. :-)
15:33:13Exorcism5 (exorcism) joins
15:34:22Exorcism quits [Read error: Connection reset by peer]
15:34:22Exorcism5 is now known as Exorcism
15:34:27<@JAA>I might try to grab new posts as they're being made until the shutdown if I have time to set that up.
15:35:01<@JAA>Although the post URLs require a topic ID, it doesn't have to be correct; you can do something like https://forum.canucks.com/topic/0-x/?do=findComment&comment=16942183 instead.
15:36:05Exorcism quits [Remote host closed the connection]
15:36:46Exorcism5 (exorcism) joins
15:46:42BigBrain (bigbrain) joins
15:51:18toss quits [Client Quit]
16:04:56HP_Archivist quits [Ping timeout: 252 seconds]
16:05:46albertlarsan68 quits [Quit: The Lounge - https://thelounge.chat]
16:10:21Exorcism5 is now known as Exorcism
16:42:04Dango360 (Dango360) joins
16:58:27HP_Archivist (HP_Archivist) joins
17:01:34IRC2DC joins
17:03:06guest9234 quits [Ping timeout: 265 seconds]
17:03:14etnguyen03 quits [Ping timeout: 252 seconds]
17:12:19Wohlstand quits [Client Quit]
17:12:42nahimgood joins
17:14:03nahimgood quits [Remote host closed the connection]
17:14:25nahimnotgood joins
17:14:26nahimnotgood quits [Remote host closed the connection]
17:14:43aaaa1 joins
17:24:15webuser9995 joins
17:24:23webuser9995 leaves
17:32:27etnguyen03 (etnguyen03) joins
17:49:37IRC2DC quits [Remote host closed the connection]
17:49:45IRC2DC joins
17:53:36IRC2DC quits [Remote host closed the connection]
17:53:48IRC2DC joins
17:54:56etnguyen03 quits [Ping timeout: 252 seconds]
18:25:16HP_Archivist quits [Ping timeout: 265 seconds]
18:28:41parfait_ joins
18:30:34mindstrut1 quits [Read error: Connection reset by peer]
18:32:07<@JAA>FOIAonline completion rate has slowed down due to larger items, now at about a third done and an estimated 3 TiB total. ETA is still on time but only just (a bit over 4 days).
18:32:21<@JAA>(That's based on the rate of the past 6 hours.)
18:34:32<@JAA>Actually, probably closer to 4 TiB.
18:35:27<thuban>hm, rough--chronological ordering suggests sizes will continue to increase
18:37:45qw3rty_ joins
18:38:19qw3rty quits [Ping timeout: 265 seconds]
18:41:50Exorcism|tor (exorcism) joins
18:41:59<@JAA>Yeah
18:43:09mindstrut joins
18:43:16Peroniko joins
18:44:03<@JAA>I can try throwing more concurrency at it. My machine is nowhere near its limits.
18:44:21<@JAA>And I haven't seen any rate limiting or blocks whatsoever so far, just some random timeouts.
18:46:54<thuban>seems wise, especially if you can adjust on the fly. what tooling are you using?
18:49:59<@JAA>qwarc
18:50:24<@JAA>I can't adjust the concurrency of running processes, but I can add more processes. :-)
18:50:48<thuban>>:?
18:50:52<@JAA>(I'd have to stop them, ideally gracefully, for the former.)
18:51:46<@JAA>I originally had one process at 25 concurrency, but that was far from ideal because it got blocked sometimes by large downloads.
18:51:52<@JAA>So now it's 5 processes with 5 concurrency each.
18:56:53<thuban>ah, i forgot qwarc runs off a database and everything. it's sufficiently self-organizing that you can just tell new processes to jump in, then?
18:57:57<@JAA>Yep, each process just takes items from the DB, processes them, and writes the new status back (plus any new items it might've discovered, not relevant in this case).
18:58:24<thuban>neat
19:00:22Exorcism|tor quits [Client Quit]
19:01:04<@JAA>It really is pretty much like a local tracker in that respect. That's what I modelled it after conceptually, anyway.
19:02:36<@JAA>Also, some of the timeouts I'm seeing are actually due to large downloads taking time to process, similar to the problems in wpull.
19:03:14<@JAA>Eventually™, I'll refactor that so the actual HTTP stuff happens in a separate thread.
19:15:39AmAnd0A quits [Read error: Connection reset by peer]
19:15:54AmAnd0A joins
19:21:12IRC2DC quits [Remote host closed the connection]
19:21:29aaaa1 quits [Remote host closed the connection]
19:23:29qw3rty_ quits [Ping timeout: 252 seconds]
19:23:30qw3rty joins
19:28:14imer quits [Client Quit]
19:28:44imer (imer) joins
19:31:11qw3rty quits [Ping timeout: 252 seconds]
19:31:14qw3rty_ joins
19:32:27AmAnd0A quits [Ping timeout: 265 seconds]
19:32:43AmAnd0A joins
19:34:13Rootliam joins
19:36:50<Rootliam>I got a response from Jason Scott about yahoo video with "all I can say is all the data is up there, one way or another. There'sno other stores out there."
19:37:18AmAnd0A quits [Read error: Connection reset by peer]
19:37:20<Rootliam>I'm not really sure if that means it could have been mixed up with something else or if it wasn't uploaded then it's gone forever
19:37:38AmAnd0A joins
19:37:54<thuban>Rootliam: did you ever open that github issue?
19:38:06<Rootliam>No but I guess I should do that soon
19:42:44programmerq quits [Read error: Connection reset by peer]
19:42:51programmerq (programmerq) joins
19:51:47cascode quits [Ping timeout: 265 seconds]
19:52:30cascode joins
20:05:48Doomaholic quits [Ping timeout: 265 seconds]
20:08:25Doomaholic (Doomaholic) joins
20:09:35programmerq quits [Client Quit]
20:09:57programmerq (programmerq) joins
20:15:21icedice (icedice) joins
20:17:55cascode quits [Read error: Connection reset by peer]
20:18:12cascode joins
20:25:19programmerq quits [Client Quit]
20:52:05<flashfire42>Wait we are completely clogged? Like completely?
21:00:22ThetaDev quits [Client Quit]
21:00:30ThetaDev joins
21:04:41etnguyen03 (etnguyen03) joins
21:18:47BearFortress quits [Ping timeout: 265 seconds]
21:26:08girst quits [Ping timeout: 252 seconds]
21:31:22ffff quits [Remote host closed the connection]
21:39:49Peroniko quits [Read error: Connection reset by peer]
21:41:09Peroniko joins
21:49:39Exorcism quits [Remote host closed the connection]
21:50:18Exorcism (exorcism) joins
22:00:21Rootliam quits [Ping timeout: 265 seconds]
22:00:21Perk quits [Read error: Connection reset by peer]
22:08:24Perk joins
22:08:25Perk7 joins
22:08:35Perk7 quits [Remote host closed the connection]
22:28:33ffff joins
22:33:14AmAnd0A quits [Ping timeout: 252 seconds]
22:33:17AmAnd0A joins
22:43:36flashfire42 quits [Client Quit]
22:43:36mindstrut quits [Read error: Connection reset by peer]
22:43:36kiska quits [Client Quit]
22:43:36Ryz263 quits [Client Quit]
22:43:36s-crypt2 quits [Client Quit]
22:43:50mindstrut joins
22:45:03Ryz263 (Ryz) joins
22:45:03s-crypt2 (s-crypt) joins
22:45:11flashfire42 joins
22:46:47kiska (kiska) joins
22:55:09girst (girst) joins
22:59:56benjins quits [Read error: Connection reset by peer]
23:07:55AmAnd0A quits [Read error: Connection reset by peer]
23:09:05AmAnd0A joins
23:13:13HP_Archivist (HP_Archivist) joins
23:17:28BlueMaxima joins
23:22:45benjins joins
23:29:37<flashfire42>So is it Optane9 again rewby or is it the transferring stuck?
23:30:57benjinsm joins
23:32:19<flashfire42>Ok looks like its optane9 that needs a kick if you have access to it JAA I did a test and Mediafire uses a seperate target and one of them went through fine
23:32:29<@JAA>flashfire42: Please stop.
23:33:04<@JAA>Targets are doing target things as well as they can. The situation isn't great, and everyone's aware of it.
23:33:44benjins quits [Ping timeout: 252 seconds]
23:41:24BearFortress joins
23:41:38Exorcism quits [Remote host closed the connection]
23:42:21Exorcism (exorcism) joins
23:44:11RetiredTurtle joins
23:46:56Peroniko quits [Ping timeout: 252 seconds]
23:47:07benjinsmi joins
23:49:39AmAnd0A quits [Read error: Connection reset by peer]
23:50:14benjinsm quits [Ping timeout: 252 seconds]
23:52:13AmAnd0A joins
23:56:50Chris5010 quits [Ping timeout: 252 seconds]

HomeSearchPrevious dayNext day