HomeSearchPrevious dayNext day

00:00:00etnguyen03 quits [Client Quit]
00:02:34Guest58 quits [Read error: Connection reset by peer]
00:03:52Shard765 (Shard) joins
00:04:32Shard76 quits [Ping timeout: 248 seconds]
00:04:32Shard765 is now known as Shard76
00:10:26<h2ibot>Manu edited Webring (+53, Found…): https://wiki.archiveteam.org/?diff=62852&oldid=62643
00:12:38abirkill quits [Quit: Let us prepare to grapple with the ineffable itself, and see if we may not eff it after all.]
00:17:04abirkill (abirkill) joins
00:17:53Wohlstand quits [Quit: Wohlstand]
00:20:53abirkill quits [Client Quit]
00:23:28etnguyen03 (etnguyen03) joins
00:24:30<h2ibot>Manu edited IRC/Logs (+35, Queued https://monitor.opensuse.org/heroes/): https://wiki.archiveteam.org/?diff=62853&oldid=62641
00:28:39abirkill (abirkill) joins
00:30:31<h2ibot>Manu edited IRC/Logs (+42, Queued https://www.gtk-gnutella.nl/): https://wiki.archiveteam.org/?diff=62854&oldid=62853
00:31:32<h2ibot>Manu edited IRC/Logs (-22, Queued https://irclogs.alpinelinux.org/): https://wiki.archiveteam.org/?diff=62855&oldid=62854
00:40:39<cruller>klea: Webarchive.io definitely does. Please see https://wiki.archiveteam.org/index.php/Webarchive.io
00:41:00<cruller>ghostarchive.org probably does, because it uses ReplayWeb.page
00:45:29useretail joins
00:46:35<h2ibot>Manu edited IRC/Logs (+36, Queued https://plugwash.raspbian.org/logs/): https://wiki.archiveteam.org/?diff=62856&oldid=62855
00:47:43useretail__ quits [Ping timeout: 248 seconds]
00:55:36<h2ibot>Manu edited Anubis/uncategorized (+40, https://elinux.org/ and https://feddit.org/ use…): https://wiki.archiveteam.org/?diff=62857&oldid=62827
00:57:37<h2ibot>Cruller edited Social network (+13068, /* List of social networks */ Isolated old…): https://wiki.archiveteam.org/?diff=62858&oldid=61266
01:59:35<yzqzss>I have another question regarding zstd WARC spec
01:59:49<yzqzss>According to the specification https://iipc.github.io/warc-specifications/specifications/warc-zstd/, it states that "Each Zstandard frame in a compressed WARC file must include the Frame_Content_Size (FCS) and Content_Checksum fields."
02:00:38<yzqzss>I don't quite understand why the standard mandates the inclusion of Frame_Content_Size (FCS) here. What is the main purpose behind this? While I can see how it might slightly optimize memory allocation for ZSTD readers when parsing WARC zstd frames, it doesn't seem to offer significant benefits to the broader WARC ecosystem.
02:01:33<yzqzss>xD
02:02:10<nicolas17>is FCS the compressed or uncompressed size?
02:02:57<yzqzss>it's uncompressed size of the original data within that frame.
02:05:45<yzqzss>btw, ZSTD uses Last_Block_Flag to indicate whether a data block is the last one in the frame. Frame_Content_Size is more of a size hint rather than an EOF marker.
02:08:01<@JAA>I believe it was to make the consumers' life easier, yes. Also feature parity with gzip, which always includes the uncompressed size (albeit mod 2^32).
02:08:33<yzqzss>if the FCS is set, I believe most zstd reader implementations will also verify after reading that it matches the actual value.
02:09:10<@JAA>I wasn't aware of that, but that's actually nice as an additional verification then.
02:09:42<yzqzss>(yzqzss don't know if those zstd readers do)
02:10:04<@JAA>https://github.com/iipc/warc-specifications/issues/53#issuecomment-706787990
02:34:34<yzqzss>thanks, very useful
02:37:03grill quits [Ping timeout: 248 seconds]
02:37:28<yzqzss>actully, I found that the zstd warcs generated by Zeno do not have the FCS set 😅😅😅😅
02:37:47grill (grill) joins
02:39:49hamouda quits [Quit: Ooops, wrong browser tab.]
02:50:30etnguyen03 quits [Client Quit]
02:55:25etnguyen03 (etnguyen03) joins
02:57:47IHaveAnIdea joins
02:58:09<IHaveAnIdea>I have two very big ideas I feel like I need to tell you about
02:58:41<IHaveAnIdea>Basically, there is the problem that so much of the web is reliant on JavaScript, but specifically on triggering DOM events
02:59:16<h2ibot>Hans5958 edited Main Page/Current Projects (+213): https://wiki.archiveteam.org/?diff=62859&oldid=62512
02:59:17<h2ibot>Hans5958 edited Main Page/Current Projects (-1): https://wiki.archiveteam.org/?diff=62860&oldid=62859
02:59:28<IHaveAnIdea>For example, none of the files on the file hosting website bowlroll.net have been saved on archive.org because they required a click event on a button
03:00:11<IHaveAnIdea>Basically, the idea I have for the first is that if we are using something like mnbot, we could detect all the registered event handlers a trigger them all
03:00:16<h2ibot>Hans5958 edited Main Page/Current Projects (+0, Rec Room is done): https://wiki.archiveteam.org/?diff=62861&oldid=62860
03:00:35<IHaveAnIdea>For the special case of scrolling, we could keep scrolling until either nothing changes, or a limit is reached
03:01:22<IHaveAnIdea>The second idea is an alternative to the heaviness and slowness of headless browsers
03:01:44<IHaveAnIdea>The idea is to use something like https://github.com/cloudflare/stpyv8 combined with a crude imitation of the basic DOM features
03:02:31<IHaveAnIdea>This would only be used for HTML, while other assets and media would be passed onto a separate pipeline that wouldn't even use this and just directly downloads them, although cookies should be passed
03:03:01<IHaveAnIdea>Of course, this will not work for every site feature, but is certainly a lot better than not using a headless browser
03:03:18<h2ibot>Hans5958 created Template:Generic source (+114, Created page with…): https://wiki.archiveteam.org/?oldid=62862
03:03:19<h2ibot>Hans5958 edited Template:Generic source (+8): https://wiki.archiveteam.org/?diff=62863&oldid=62862
03:03:25<IHaveAnIdea>Perhaps, we could detect when specifc unhandled APIs are called and switch to using something like mnbot
03:03:29<IHaveAnIdea>instead
03:03:42IHaveAnIdea quits [Client Quit]
03:07:18<h2ibot>Hans5958 created Template:Archive Team GitHub link (+48, Created page with…): https://wiki.archiveteam.org/?oldid=62864
03:07:19<h2ibot>Hans5958 created Template:Atgh (+47, Redirected page to [[Template:Archive Team…): https://wiki.archiveteam.org/?oldid=62865
03:08:20<h2ibot>Hans5958 edited Template:Generic source (-75): https://wiki.archiveteam.org/?diff=62866&oldid=62863
03:09:20<h2ibot>Hans5958 created Template:Tracker item (+50, Created page with…): https://wiki.archiveteam.org/?oldid=62867
03:10:20<h2ibot>Hans5958 created Template:Tr (+35, Redirected page to [[Template:Tracker item]]): https://wiki.archiveteam.org/?oldid=62868
03:10:21<h2ibot>Hans5958 edited Rec Room (-130): https://wiki.archiveteam.org/?diff=62869&oldid=62813
03:17:24nexussfan (nexussfan) joins
03:19:25<h2ibot>Hans5958 edited Main Page/Current Projects (-257, Remove stale entries): https://wiki.archiveteam.org/?diff=62870&oldid=62861
03:19:29<TheTechRobo>Automatically pressing buttons sounds very much like bad crawler behaviour to me - there's a reason I don't think most crawlers submit forms automatically, for example
03:20:25<h2ibot>Hans5958 edited Wikimedia Etherpad (-67): https://wiki.archiveteam.org/?diff=62871&oldid=61409
03:23:53systwo is now known as systwi
03:25:11<Hans5958>There you go, made {{Archive Team GitHub link|[...]}} (or {{atgh|...}}) and {{Tracker item|[...]}} (or {{tr|...}}) so you don't need to do [https://github.com/ArchiveTeam/[...] [...]] and [https://tracker.archiveteam.org/[...] [...]]
03:29:00<Hans5958>Ideally a bot should replace all the URLs, but let's do that sometime later
03:48:51etnguyen03 quits [Remote host closed the connection]
03:52:47benjins3 quits [Read error: Connection reset by peer]
03:57:56<pabs>TIL there is an anubis solver https://github.com/sleeyax/anubis-solver
03:58:19<pabs>and https://github.com/huzpsb/anubis_solver_pypi
03:59:19nexussfan quits [Client Quit]
04:02:14Cupping128548 joins
04:02:46DogsRNice quits [Read error: Connection reset by peer]
04:03:59Cupping12854 quits [Ping timeout: 248 seconds]
04:06:39Cupping128548 quits [Ping timeout: 248 seconds]
04:12:35nexussfan (nexussfan) joins
04:20:51<nicolas17>pabs: https://github.com/DavidBuchanan314/anubis_offload
05:06:42Ointment8862 (Ointment8862) joins
05:07:59Ointment8862 quits [Client Quit]
05:09:12<cruller>https://webrecorder.net/blog/2025-05-28-create-use-and-automate-actions-with-custom-behaviors-in-browsertrix/#:~:text=Need%20it%20to%20click%20on%20something%20other%20than%20links%2C%20like%20all%20the%20%3Cbutton%3E%20elements%20on%20a%20page%3F%20No%20problem!%20Just%20specify%20a%20different%20CSS%20selector%20for%20autoclick%20directly%20in%20the%20workflow%20editor states this, but they themselves don't do so by default.
05:15:10nexussfan quits [Remote host closed the connection]
05:47:56<h2ibot>Cruller edited Archive.st (+818, Added [[#Vital Signs]] and changed status): https://wiki.archiveteam.org/?diff=62872&oldid=59330
05:51:40IHaveAnIdea joins
05:53:01<IHaveAnIdea>"Automatically pressing buttons sounds very much like bad crawler behaviour to me" We could detect if the element is a form and whether that form has other input elements that are not hidden, invisible, or prefilled. If so, we could choose not to click it
05:53:22<IHaveAnIdea>Besides there are other events that would definitely not pose a problem, like scrolling
05:54:57<h2ibot>Cruller edited Archive.st (+59, Added a description regarding Yandex's snapshots): https://wiki.archiveteam.org/?diff=62873&oldid=62872
05:55:01<IHaveAnIdea>"https://github.com/huzpsb/anubis_solver_pypi" I've used this tool before. One thing you should know about Anubis is that it requires that the requests made to the website and the requests made to solve it come from the same IP, or else it will reject it
05:55:03<klea>If/when someone tells me to, I can write code to try to replace those, tho I might just use JWB.
05:57:30IHaveAnIdea quits [Client Quit]
06:01:02<klea>arkiver: https://tomo.mun-tonsi.net/ Custom JS PoW challenge.
06:01:35xkey (xkey) joins
06:05:28<cruller>The only way to contact Archive.st is/was postal mail... https://web.archive.org/web/20260402151050/https://archive.st/CONTACT/
06:15:19Island quits [Read error: Connection reset by peer]
06:23:48xkey quits [Client Quit]
06:25:24xkey (xkey) joins
06:31:54<BornOn420>curl from the CLI to transfer.archivete.am (as described on https://transfer.archivete.am/#) throws an wrror
06:31:55<eggdrop>inline (for browser viewing): https://transfer.archivete.am/inline/#)
06:32:22<BornOn420>curl: (35) Recv failure: Connection reset by peer # #
06:41:59McAfee joins
06:47:12McAfee leaves
06:55:51McAfee joins
06:59:40McAfee leaves
07:04:07McAfee joins
07:43:42amphitryon_ joins
07:47:42amphitryon quits [Ping timeout: 268 seconds]
08:16:04thelounge69 quits [Ping timeout: 268 seconds]
08:28:17thelounge69 (jonte4) joins
08:35:35<h2ibot>Cruller edited Ghostarchive (+280, Added links to a warc file and [[#Problems]]): https://wiki.archiveteam.org/?diff=62874&oldid=59446
08:36:43<cruller>I found this endpoint by doing a random search, but what is "chimurai4"?
08:45:37<h2ibot>Cruller edited Archive Services (+174, Added similar lists): https://wiki.archiveteam.org/?diff=62875&oldid=62525
08:57:39<h2ibot>Cruller edited Obstacles (+249, /* List of softwares */ Added Tartarus…): https://wiki.archiveteam.org/?diff=62876&oldid=61890
09:12:19Hackerpcs quits [Remote host closed the connection]
09:13:11Hackerpcs (Hackerpcs) joins
09:57:46benjins3 joins
10:05:30cruller|irc quits [Quit: IRCNow and Forever!]
10:14:05cruller|irc (cruller|irc) joins
10:16:04khaoohs joins
10:24:57Wohlstand1 (Wohlstand) joins
10:25:57hyperreal quits [Quit: hyperreal]
10:27:21Wohlstand1 is now known as Wohlstand
10:41:17hyperreal (hyperreal) joins
10:42:55<h2ibot>Cruller edited Microsoft (+124, /* External Links */ Added "Killed by…): https://wiki.archiveteam.org/?diff=62877&oldid=60583
10:43:25Wohlstand quits [Client Quit]
10:48:42Wohlstand (Wohlstand) joins
10:52:06Wohlstand quits [Client Quit]
10:52:15Wohlstand (Wohlstand) joins
10:52:53Wohlstand quits [Client Quit]
10:53:18Wohlstand (Wohlstand) joins
10:55:57<h2ibot>Cruller edited Microsoft (+78, /* External Links */ Added yet another graveyard): https://wiki.archiveteam.org/?diff=62878&oldid=62877
10:59:35simon816 quits [Quit: ZNC 1.10.2 - https://znc.in]
11:00:21Bleo18260072271962345522201107 quits [Quit: The Lounge - https://thelounge.chat]
11:00:44simon816 (simon816) joins
11:00:58<h2ibot>Cruller edited Apple (+154, Added two "Killed by Apple"s as [[#External…): https://wiki.archiveteam.org/?diff=62879&oldid=59265
11:03:02Bleo18260072271962345522201107 joins
11:06:14khaoohs_ joins
11:06:59<h2ibot>Cruller edited Google (+178, /* Other */ Added 3 graveyards): https://wiki.archiveteam.org/?diff=62880&oldid=59333
11:09:21khaoohs quits [Ping timeout: 268 seconds]
11:09:58Goofybally quits [Ping timeout: 268 seconds]
11:09:59<h2ibot>Cruller edited Amazon (+50, /* External links */ Added Amazon Cemetery): https://wiki.archiveteam.org/?diff=62881&oldid=59266
11:15:00<h2ibot>Cruller edited Facebook (+32, /* External links */ Added "Killed by Meta"): https://wiki.archiveteam.org/?diff=62882&oldid=59012
11:35:31Wohlstand quits [Client Quit]
11:40:50Wohlstand (Wohlstand) joins
11:45:08nine quits [Quit: See ya!]
11:45:26nine joins
11:57:51MPThLee quits [Quit: bye]
11:58:19MPThLee (MPThLee) joins
12:15:22T31M quits [Quit: ZNC - https://znc.in]
12:16:14T31M joins
12:18:36<T31M>Hey, haven't been on here for a while but chugging along on the Medica & recroom projects (as you might have seen). Anything else than the default currently that needs support ?
12:32:17Webuser585430 joins
12:32:27Webuser585430 quits [Client Quit]
12:35:58etnguyen03 (etnguyen03) joins
12:37:49BornOn420 quits [Quit: Textual IRC Client: www.textualapp.com]
12:43:13<h2ibot>OrIdow6 edited Roblox (+235, /* Group Walls (Closing down 2025-2026) */ A…): https://wiki.archiveteam.org/?diff=62883&oldid=62421
12:46:10UwU quits [Ping timeout: 268 seconds]
12:47:21etnguyen03 quits [Client Quit]
12:59:02UwU joins
13:00:23<hexagonwin>arkiver is blice going well? we have 50 hours left until shutdown
13:07:07<that_lurker>IDK might know more on that. There are multiple AB jobs for it going
13:08:36thewinwin80 joins
13:12:41thewinwin8 quits [Ping timeout: 268 seconds]
13:12:50thewinwin80 is now known as thewinwin8
13:22:12etnguyen03 (etnguyen03) joins
13:28:06raccoon quits [Ping timeout: 268 seconds]
13:28:56<hexagonwin>i don't think blice can be saved through AB...
13:28:56McAfee leaves [Disconnected: Hibernating too long]
13:30:01<IDK>hexagonwin: me neither, i have no idea what im doing
13:30:38<IDK>but it appears the story itself is hard coded into the html, according to a SPN capture
13:30:57<IDK>but anyways I will try to spin up more jobs later on to get whatever we could
13:31:15<hexagonwin>damn..
13:31:58<IDK>I'm just gonna pray and see tbh
13:32:14<IDK>one thing I know is the novel's pagination is a POST request and cant be saved
13:44:55<IDK>anyways, I always thought there was a blice project upcoming, but nothing came from that
13:46:16<klea>IDK: Are you sure the AB job is at least saving *something* useful?
13:46:36<klea>Although I guess having incomplete stories is better than not having anything.
13:47:02<@arkiver>imer: we got a little emergency project coming up for Blice. do you have some time for setting up a target?
13:47:06<@arkiver>this would be archiveteam_blice_
13:47:07<@arkiver>blice_
13:47:10<@arkiver>Archive Team Blice:
13:47:22<klea>arkiver: Do you have the code ready yet?
13:47:31<@arkiver>klea: nearly
13:47:44<klea>Ack.
13:47:48<klea>arkiver++
13:47:49<eggdrop>[karma] 'arkiver' now has 116 karma!
13:49:39<IDK>arkiver: ah, I thought it was abandoned
13:50:08<IDK>if emergency projects comes up, I could abort the blice jobs
13:50:18<IDK>arkiver++
13:50:18<eggdrop>[karma] 'arkiver' now has 117 karma!
13:50:41<@arkiver>IDK: don't abort anything please!
13:51:34<IDK>arkiver: fyi, im doing the https://blice.co.kr/web/viewer.kt?timesId= without any parameter, but I think the blice viewer doesnt actually use that
13:51:39<h2ibot>Manu edited Discourse/active (+46, Add https://discourse.32bit.cafe/): https://wiki.archiveteam.org/?diff=62884&oldid=62835
13:52:01raccoon (raccoon) joins
13:52:15<IDK>but i will keep the job running then
13:55:27<@arkiver>imer: sorry tracker was not up yet, it's up now
14:02:03etnguyen03 quits [Client Quit]
14:03:59UwU quits [Ping timeout: 248 seconds]
14:05:29nexussfan (nexussfan) joins
14:07:59McAfee joins
14:10:04UwU joins
14:10:44etnguyen03 (etnguyen03) joins
14:19:29<Exorcism>for blice, is there any irc channel ?
14:30:06McAfee leaves
14:32:07<@imer>arkiver: target's up!
14:33:08<@arkiver>Exorcism: no i think we will just keep archiveteam-bs for it
14:33:10<@arkiver>imer: thank you!!
14:33:24<Exorcism>👌
14:36:40<T31M>Please send a ping along to me once the code is up / Docker image available, then I add it to my nodes :)
14:37:16<Exorcism>same for me pls :3
14:49:18Wohlstand quits [Quit: Wohlstand]
14:52:58lunik1 quits [Quit: :x]
14:54:47lunik1 joins
15:00:46lunik1 quits [Client Quit]
15:00:55emily (pseudorizer) joins
15:01:13pseudorizer quits [Ping timeout: 268 seconds]
15:02:38lunik1 joins
15:12:50McAfee joins
15:13:37<@arkiver>we should have some general way to ping those who want to receive them for new projects
15:13:48<@arkiver>klea: do you have ideas about that? ^
15:14:17<klea>Maybe a bot or something?
15:20:17<klea>IIRC kiska's metrics thing hooks to the WarriorHQ projects.json to get new projects, but I know of at least one public project that was not there, on purpose.
15:59:26nexussfan quits [Client Quit]
16:06:36DogsRNice joins
16:12:23<@arkiver>nearly starting
16:23:01Island joins
16:26:16szczot3k quits [Remote host closed the connection]
16:28:43szczot3k (szczot3k) joins
16:30:45Webuser563112 joins
16:32:59etnguyen03 quits [Client Quit]
16:33:14etnguyen03 (etnguyen03) joins
16:40:14Webuser563112 quits [Client Quit]
16:41:46szczot3k_ (szczot3k) joins
16:44:12szczot3k quits [Ping timeout: 268 seconds]
16:44:12szczot3k_ is now known as szczot3k
16:49:48szczot3k quits [Remote host closed the connection]
16:50:35szczot3k (szczot3k) joins
16:54:41PredatorIWD8 quits [Ping timeout: 268 seconds]
16:56:03PredatorIWD8 joins
17:08:52Guest quits [Ping timeout: 268 seconds]
17:13:38etnguyen03 quits [Client Quit]
17:13:59etnguyen03 (etnguyen03) joins
17:14:01etnguyen03 quits [Client Quit]
17:14:20etnguyen03 (etnguyen03) joins
17:21:06Wohlstand (Wohlstand) joins
17:24:17lflare quits [Ping timeout: 268 seconds]
17:24:54PredatorIWD8 quits [Ping timeout: 268 seconds]
17:30:55Cornelius7052865 quits [Quit: Cornelius7052865]
17:33:40Cornelius7052865 (Cornelius) joins
17:34:21etnguyen03 quits [Client Quit]
17:40:58Mateon1 quits [Remote host closed the connection]
17:41:09Guest joins
17:41:09Mateon1 joins
17:47:37dabs joins
17:47:43BearFortress quits [Ping timeout: 268 seconds]
17:48:28dabs quits [Remote host closed the connection]
17:48:47dabs joins
17:49:20MrMcNugg1 (MrMcNuggets) joins
17:52:39MrMcNuggets quits [Ping timeout: 268 seconds]
18:06:05<didyousayboop>Where could I find the 2017 archive on datalounge.com? Just in the Wayback Machine? Or is there a WARC?
18:06:09BearFortress joins
18:07:32<didyousayboop>I searched "datalounge.com" on archive.org hoping I might find a WARC and got no results.
18:08:40<klea>It was made by ArchiveBot, so searching here: https://archive.fart.website/archivebot/viewer/, gives https://archive.fart.website/archivebot/viewer/?q=datalounge.com, and clicking on both entries I found https://archive.fart.website/archivebot/viewer/domain/www.datalounge.com is the one with https://archive.fart.website/archivebot/viewer/job/201712182150292bgzy which has the
18:08:40<klea>links to the WARCs on IA.
18:09:07cyanbox_ joins
18:10:50<didyousayboop>Oh, thank you very much!
18:11:18<didyousayboop>terrible domain name, but a useful website
18:11:59cyanbox quits [Ping timeout: 248 seconds]
18:16:20<klea>You're welcome.
18:18:55oxtyped quits [Ping timeout: 248 seconds]
18:28:38oxtyped joins
18:57:24etnguyen03 (etnguyen03) joins
19:02:38etnguyen03 quits [Client Quit]
19:02:54etnguyen03 (etnguyen03) joins
19:07:09etnguyen03 quits [Client Quit]
19:07:54etnguyen03 (etnguyen03) joins
19:21:22McAfee leaves [Disconnected: Replaced by new connection]
19:21:26McAfee joins
19:29:20BornOn420 (BornOn420) joins
19:29:21etnguyen03 quits [Client Quit]
20:33:25hyperreal quits [Quit: Konversation terminated!]
20:33:44hyperreal (hyperreal) joins
20:58:06hamouda joins
21:48:01etnguyen03 (etnguyen03) joins
21:55:00moth3 quits [Ping timeout: 268 seconds]
21:57:03moth3 joins
22:42:29Kabaya quits [Ping timeout: 268 seconds]
22:48:34Kabaya joins
22:58:03Wohlstand quits [Quit: Wohlstand]
23:30:42Kabaya quits [Client Quit]
23:43:55etnguyen03 quits [Client Quit]
23:44:39Kabaya joins
23:57:51dendory quits [Quit: The Lounge - https://thelounge.chat]
23:59:09dendory (dendory) joins

HomeSearchPrevious dayNext day