HomeSearchPrevious dayNext day

00:05:46etnguyen03 quits [Ping timeout: 252 seconds]
00:09:59etnguyen03 (etnguyen03) joins
00:17:32AffinityFan joins
00:18:40h|ca2 (h) joins
00:20:10<AffinityFan>Ok, so I've got a list of all the urls of both the topics and the forums/sub forums. How do I get it added as a project for Warrior?
00:21:50<pokechu22>Regarding the number of posts, "posts" includes replies, not just topics. E.g. https://forum.affinity.serif.com/index.php?/forum/5-desktop-questions-macos-and-windows/ says https://forum.affinity.serif.com/index.php?/forum/136-pre-v2-archive-of-desktop-questions-macos-and-windows/ has 406.5k posts but it has only 2534 pages where each page lists 25 topics, so 63350 topics.
00:21:52<pokechu22>That just means that the average thread has 406500/63350=6.4 topics
00:22:24AffinityFan quits [Client Quit]
00:23:39AffinityFan joins
00:24:33<AffinityFan>I'm thinking 6.4 sounds about right, so that is reassuring.
00:24:48beardicus quits [Quit: Ping timeout (120 seconds)]
00:25:52beardicus (beardicus) joins
00:29:32<pokechu22>For a forum of that site I don't think we can put it into a warrior project (or at least not into #//) since it'd probably overload the site. Archivebot (which runs jobs from a single machine) is more suitable for forums like that, though it looks like it took about a month to run last time
00:30:00<pokechu22>(but I assume the first run had archivebot grabbing offsite links as well; if we don't do that, it probably would finish faster)
00:31:36<pokechu22>Did you end up discovering additional URLs that weren't in the sitemap?
00:34:12<AffinityFan>Just the subforum pages for proper navigation, though I suspect I'm missing some(didn't recurse so I don't have subforums of subforums, which I didn't realize the forum had until after running. Should be able to modify and rerun fairly easily for that.
00:35:42<pokechu22>If that's the case it's probably easiest for me to just start a new archivebot job rather than doing anything with a URL list - archivebot crawls recursively so it'll find stuff like https://forum.affinity.serif.com/index.php?/forum/4-news-and-information/page/3/ naturally
00:36:02<AffinityFan>Oh neat.
00:37:00<AffinityFan>I will say it's probably important to keep attachments, I'll check to see if those are stored on the same domain.
00:38:46AffinityFan quits [Client Quit]
00:40:32AffinityFan joins
00:41:39<pokechu22>We can get offsite links after the job finishes if needed
00:42:24<@JAA>They are on the same domain under https://forum.affinity.serif.com/applications/core/interface/file/attachment.php?... URLs.
00:44:40<pokechu22>AffinityFan: AB job started; you can watch progress at http://archivebot.com/?initialFilter=forum.affinity.serif.com
00:44:48<AffinityFan>Nice! Thanks so much.
00:46:25AffinityFan quits [Client Quit]
00:48:21AffinityFan joins
00:52:06<AffinityFan>Really need to get a proper IRC client... Anyway, is there anything I can do to help from here? I see that new people can't create pip
00:52:43<AffinityFan>Can't create pipelines, which I understand to mean I can't lend my bandwidth?
00:53:43<pokechu22>Yeah, for archivebot specifically, you can't lend bandwidth. But you can contribute to other warrior projects :)
00:54:20<pokechu22>If you see the archivebot job doing something weird on the dashboard, let us know. I've re-added ignores that were present on the 2024 run so it seems unlikely though
00:54:20AffinityFan0 joins
00:58:43AffinityFan quits [Client Quit]
00:58:43AffinityFan0 quits [Client Quit]
00:59:35emhpaticist joins
00:59:58emphatic quits [Read error: Connection reset by peer]
01:00:27AffinityFan joins
01:01:27AffinityFan quits [Client Quit]
01:04:00Hackerpcs quits [Remote host closed the connection]
01:04:54Hackerpcs (Hackerpcs) joins
01:06:40Webuser374223 joins
01:06:57Webuser374223 quits [Client Quit]
01:07:31AffinityFan joins
01:08:27<AffinityFan>Looks good so far, I see it's doing profiles which is something I hadn't even thought about-perks of using a proper crawl, I guess.
01:10:13<AffinityFan>Also, I'm having some issues with https://irclogs.archivete.am/archiveteam-bs/2026-07-28 and I'd originally assumed it was on my end, and it *probably* is, but I do see there is a current job on a separate subdomain, might want to make sure we haven't overwhelmed the service(s).
01:10:56<AffinityFan>Looks like the job I was referencing is gone
01:11:02AffinityFan quits [Client Quit]
01:12:46AffinityFan joins
01:13:48<AffinityFan>And, possibly coincidentally, the log is loading again for me. And I can actually access the page the job was running against.
01:15:19AffinityFan quits [Client Quit]
01:20:52Jake330 quits [Quit: Leaving for a bit!]
01:23:29Jake330 (Jake) joins
01:47:41AffinityFan joins
01:51:22nine quits [Ping timeout: 252 seconds]
01:52:43nine joins
01:56:40AffinityFan quits [Client Quit]
02:00:13AffinityFan joins
02:01:04Starchives_ (Starchives) joins
02:03:50Starchives__ quits [Ping timeout: 252 seconds]
02:07:08grill quits [Ping timeout: 252 seconds]
02:08:03grill (grill) joins
02:10:35useretail quits [Remote host closed the connection]
02:11:30<pabs>https://text.npr.org/2026/07/23/nx-s1-5900901/bike-lanes-dot-websites
02:15:20<pokechu22>Already reran highways.dot.gov and also https://transfer.archivete.am/DhXHg/highways.dot.gov_removed_urls_from_sitemap_2026-07-23_comparing_2026-02-01.txt (#UncleSamsArchive)
02:15:21<eggdrop>inline (for browser viewing): https://transfer.archivete.am/inline/DhXHg/highways.dot.gov_removed_urls_from_sitemap_2026-07-23_comparing_2026-02-01.txt
02:22:09AffinityFan quits [Client Quit]
02:26:21etnguyen03 quits [Client Quit]
02:26:34etnguyen03 (etnguyen03) joins
02:36:42ericgallager joins
02:57:15etnguyen03 quits [Ping timeout: 268 seconds]
02:58:29ScenarioPlanet quits [Ping timeout: 268 seconds]
02:59:06TheTechRobo quits [Ping timeout: 268 seconds]
02:59:06Pedrosso quits [Ping timeout: 268 seconds]
03:00:51etnguyen03 (etnguyen03) joins
03:03:21etnguyen03 quits [Remote host closed the connection]
03:24:32entrox joins
03:28:26cyanbox joins
04:09:24ericgallager quits [Ping timeout: 268 seconds]
04:09:25ericgallager joins
04:18:42chunkynutz60181694 quits [Quit: The Lounge - https://thelounge.chat]
04:18:56chunkynutz60181694 joins
04:21:27cyanbox quits [Read error: Connection reset by peer]
04:23:03cyanbox joins
04:42:46Island quits [Read error: Connection reset by peer]
05:12:18beastbg8__ quits [Ping timeout: 268 seconds]
05:23:23nexussfan quits [Remote host closed the connection]
05:26:31mparramont joins
05:26:31<mparramont>Agent: Miguel owns legacy Ustream account mparramont / channel acampadabcn (ID 8299412). He is looking for uncatalogued WARCs, private rescue sets, torrents, or contributor-held downloads for 28 recordings from May-July 2011. Full approved request and recording IDs: https://transfer.archivete.am/nB6bo/acampadabcn-ustream-recovery-request.txt Contact: mparramont@gmail.com
05:26:33<eggdrop>inline (for browser viewing): https://transfer.archivete.am/inline/nB6bo/acampadabcn-ustream-recovery-request.txt
05:26:34mparramont quits [Client Quit]
05:36:39<nicolas17>"Agent"
05:36:41<nicolas17>is this some AI shit
05:38:56beardicus3 (beardicus) joins
05:41:41beardicus7 (beardicus) joins
05:41:46<Dango360>007
05:42:22beardicus quits [Ping timeout: 252 seconds]
05:42:22beardicus7 is now known as beardicus
05:45:18beardicus3 quits [Ping timeout: 252 seconds]
05:45:24nine quits [Quit: See ya!]
05:45:36nine joins
06:01:09nathang218438 joins
06:03:20kirb quits [Quit: oh no]
06:03:20missaustraliana quits [Quit: ZNC - https://znc.in]
06:03:38kirb joins
06:05:48kirb quits [Client Quit]
06:08:35missaustraliana joins
06:10:08kirb joins
06:14:10McAfee leaves [Disconnected: Hibernating too long]
06:24:27ericgallager quits [Ping timeout: 268 seconds]
06:37:14skankhunt42 quits [Quit: Ping timeout (120 seconds)]
06:37:31skankhunt42 (skankhunt42) joins
06:49:31hexa- quits [Quit: Disconnected]
06:50:27hexa- (hexa-) joins
07:04:34McAfee joins
07:10:49Umbire quits [Quit: Umbire zaps a wand of digging!]
07:16:55nine quits [Client Quit]
07:17:07nine joins
07:20:23McAfee leaves
07:30:44amphitryon0 joins
07:30:45amphitryon_ quits [Read error: Connection reset by peer]
08:00:47McAfee joins
08:01:54McAfee leaves
08:38:11<klea>Indeed...
09:09:03<h2ibot>KleaBot edited Main Page/Current Warrior Project (+117, Default projects are now telegram (weight 1),…): https://wiki.archiveteam.org/?diff=63308&oldid=63298
09:17:40McAfee joins
09:20:35McAfee leaves
09:23:24HP_Archivist (HP_Archivist) joins
09:24:28McAfee joins
09:27:07systwo (systwi) joins
09:31:08McAfee leaves
09:31:18systwi quits [Ping timeout: 268 seconds]
09:44:52BornOn420 quits [Ping timeout: 268 seconds]
10:02:59McAfee joins
10:09:52McAfee leaves
10:10:13McAfee joins
10:17:07thewinwin88 joins
10:21:15thewinwin8 quits [Ping timeout: 268 seconds]
10:21:18thewinwin88 is now known as thewinwin8
10:28:16beastbg8 joins
10:38:15Karlett (Karlett) joins
10:40:59Kayla quits [Ping timeout: 268 seconds]
11:00:19Bleo18260072271962345522201107 quits [Quit: The Lounge - https://thelounge.chat]
11:03:05Bleo18260072271962345522201107 joins
11:30:57nine quits [Quit: See ya!]
11:31:09nine joins
11:34:26Webuser567315 joins
12:23:46ericgallager joins
13:04:04stepney141 (stepney141) joins
13:08:58stepney141 quits [Ping timeout: 252 seconds]
13:31:08TheTechRobo (TheTechRobo) joins
13:41:14TheTechRobo quits [Ping timeout: 252 seconds]
13:45:59TheTechRobo (TheTechRobo) joins
13:48:10Mateon2 joins
13:49:04Mateon1 quits [Ping timeout: 268 seconds]
13:49:04Mateon2 is now known as Mateon1
13:58:40HP_Archivist quits [Quit: Leaving]
14:00:37McAfee leaves [Disconnected: Hibernating too long]
14:05:57McAfee joins
14:19:08Webuser673136 joins
14:24:10nexussfan (nexussfan) joins
14:34:27<Webuser673136>I've got a growing collection of warcs in an archive I've been maintaining. I'm at a point (unemployed and out of harddrive space) where I need to start to prune away some unrelated, unneeded, and duplicative urls/files in those archives (First order, removing a lot of linked videos that are not important for the archives focus, etc.) Does anyone
14:34:27<Webuser673136>have scripts they like to use for bulk exploration across large numbers of warcs or for cleaning out things that should have been ignored during the grab-site phase? I'm currently cobbling some things together for myself using metawarc as a starting place. But, I imagine this is not a new or unique workflow.
14:42:50fetcher quits [Ping timeout: 252 seconds]
14:44:45fetcher joins
15:06:18fetcher quits [Ping timeout: 252 seconds]
15:14:06fetcher joins
15:20:10<@arkiver>Webuser673136: how big is this archive?
15:20:53<@arkiver>if it's not too big, perhaps it could be uploaded to IA. it'll not go into the Wayback Machine though, but it'll sit there
15:21:10rohvani joins
15:23:13<Webuser673136>It's only like 5-6 TB at the moment (but growing quickly now that I have time to work on it.) While IA is a good approach for backing it up. I think it would be worth optimizing the archived content to focus on relevant content as I grow it either way. A curated archive of content will be far easier to pass along to others and save than one that is
15:23:13<Webuser673136>all over the place.
15:33:10<@JAA>It's not a unique situation, but I'm not aware of any tooling for such WARC processing/filtering. And it's easy to accidentally mangle the data since there are several sharp edges in the WARC spec, so I wouldn't recommend trying to implement it yourself.
15:34:00Island joins
15:34:11Kabaya quits [Read error: Connection reset by peer]
15:34:20Kabaya joins
15:39:02nine quits [Quit: See ya!]
15:39:14nine joins
15:45:07<h2ibot>Brad edited Alive... OR ARE THEY (+324, Added De technoloog): https://wiki.archiveteam.org/?diff=63309&oldid=62933
15:51:02chunkynutz601816946 joins
15:51:55chunkynutz60181694 quits [Read error: Connection reset by peer]
15:51:55chunkynutz601816946 is now known as chunkynutz60181694
15:54:18unknownsrc quits [Quit: Ping timeout (120 seconds)]
15:54:30unknownsrc (unknownsrc) joins
15:59:49Webuser865601 joins
16:00:04Webuser865601 quits [Client Quit]
16:20:36McAfee leaves
16:21:53McAfee joins
16:39:16@rewby quits [Ping timeout: 268 seconds]
16:44:15<h2ibot>Manu edited Mailman/2 (-68, Queued https://lists.tetaneutral.net/): https://wiki.archiveteam.org/?diff=63310&oldid=63296
16:45:31Webuser673136 quits [Client Quit]
16:48:16<h2ibot>Manu edited Mailman/2 (+4, Queued http://lists.vtigercrm.com/): https://wiki.archiveteam.org/?diff=63311&oldid=63310
16:49:02rohvani quits [Read error: Connection reset by peer]
16:50:16<h2ibot>Manu edited Mailman/2 (+4, Queued https://lists.smc.org.in/): https://wiki.archiveteam.org/?diff=63312&oldid=63311
16:54:17<h2ibot>Manu edited Mailman/2 (+22, Queued https://lists.svlug.org/): https://wiki.archiveteam.org/?diff=63313&oldid=63312
17:01:47Webuser979189 joins
17:02:49Webuser979189 quits [Client Quit]
17:06:19<h2ibot>Manu edited Mailman/2 (+43, https://mail.openjdk.org/ switched to Mailman 3): https://wiki.archiveteam.org/?diff=63314&oldid=63313
17:06:20<h2ibot>Manu edited Mailing Lists (+26, https://mail.openjdk.org/ switched to Mailman 3): https://wiki.archiveteam.org/?diff=63315&oldid=63289
17:07:38^ quits [Ping timeout: 268 seconds]
17:09:19<h2ibot>Manu edited Mailman/2 (+18, Lost https://lists.rpath.com/): https://wiki.archiveteam.org/?diff=63316&oldid=63314
17:10:20<h2ibot>Manu edited Mailman/2 (-79, https://lists.ibiblio.org/ switched to Sympa): https://wiki.archiveteam.org/?diff=63317&oldid=63316
17:11:20<h2ibot>Manu edited Mailing Lists (+27, https://lists.ibiblio.org/ switched to Sympa): https://wiki.archiveteam.org/?diff=63318&oldid=63315
17:12:20<h2ibot>Manu edited Mailman/2 (+17, Lost http://lists.ingate.com/): https://wiki.archiveteam.org/?diff=63319&oldid=63317
17:12:43^ (^) joins
17:18:21<h2ibot>Manu edited Mailman/2 (+108, Lost http://lists.matroska.org/): https://wiki.archiveteam.org/?diff=63320&oldid=63319
17:18:52systwo quits [Read error: Connection reset by peer]
17:22:21<h2ibot>Manu edited Mailman/2 (+51, Lost http://lists.lm-sensors.org/): https://wiki.archiveteam.org/?diff=63321&oldid=63320
17:27:22<h2ibot>Manu edited Mailman/2 (+46, Queued https://lists.jifty.org/): https://wiki.archiveteam.org/?diff=63322&oldid=63321
17:37:23<h2ibot>Manu edited Mailman/2 (-2, Queued https://lists.siam.org/): https://wiki.archiveteam.org/?diff=63323&oldid=63322
17:42:24<h2ibot>Manu edited Mailman/2 (+33, https://lists.siam.org/ not possible with AB): https://wiki.archiveteam.org/?diff=63324&oldid=63323
17:56:25VerifiedJ quits [Remote host closed the connection]
17:57:04VerifiedJ (VerifiedJ) joins
18:01:50thewinwin86 joins
18:05:36thewinwin8 quits [Ping timeout: 268 seconds]
18:05:40thewinwin86 is now known as thewinwin8
18:08:03key is now known as xkey
18:08:08xkey is now known as key
18:33:30nexussfan quits [Remote host closed the connection]
18:34:26nexussfan (nexussfan) joins
19:02:21nexussfan quits [Read error: Connection reset by peer]
19:26:07nexussfan (nexussfan) joins
19:30:23systwo (systwi) joins
19:41:10rewby (rewby) joins
19:41:10@ChanServ sets mode: +o rewby
19:43:58Wohlstand (Wohlstand) joins
19:50:34Kayla (Karlett) joins
19:51:10Wohlstand quits [Client Quit]
19:53:31Karlett quits [Ping timeout: 268 seconds]
19:58:52Goofybally quits [Killed (NickServ (GHOST command used by Goofybally7!~Goofyball@145.82.3.135))]
19:58:57Goofybally joins
20:06:43Mateon2 joins
20:08:56Mateon1 quits [Ping timeout: 268 seconds]
20:08:56Mateon2 is now known as Mateon1
20:29:42APOLLO03 quits [Ping timeout: 252 seconds]
20:34:06APOLLO03 joins
20:34:32APOLLO03 quits [Client Quit]
20:36:35APOLLO03 joins
21:04:09Goofybally quits [Killed (NickServ (GHOST command used by Goofybally5!~Goofyball@145.82.8.77))]
21:04:14Goofybally joins
21:42:56thewinwin88 joins
21:43:18etnguyen03 (etnguyen03) joins
21:43:37thewinwin8 quits [Read error: Connection reset by peer]
21:43:40thewinwin88 is now known as thewinwin8
21:55:08etnguyen03 quits [Client Quit]
22:13:30driib975 quits [Ping timeout: 268 seconds]
22:14:58driib975 (driib) joins
22:23:18etnguyen03 (etnguyen03) joins
23:03:31ptricky joins
23:31:21Mist8kenGAS joins
23:33:00Mist8kenGAS quits [Client Quit]
23:45:23Sokar quits [Ping timeout: 268 seconds]
23:50:29Sokar joins

HomeSearchPrevious dayNext day