HomeSearchPrevious dayNext day

00:09:16Wingy1139793760180 quits [Ping timeout: 265 seconds]
00:40:52Iki quits [Ping timeout: 240 seconds]
01:02:59Arcorann (Arcorann) joins
01:18:32lukash79 joins
01:18:59Wingy1139793760180 (Wingy) joins
01:22:59Mateon1 joins
01:59:35qwertyasdfuiopghjkl quits [Ping timeout: 265 seconds]
02:13:48<pabs>this person died: https://www.archieroach.com/ https://en.wikipedia.org/wiki/Archie_Roach https://www.abc.net.au/news/2022-07-30/archie-roach-aboriginal-musician-songwriter-artist-dead-at-66/101285620
02:38:17HackMii_ quits [Remote host closed the connection]
02:38:48HackMii_ (hacktheplanet) joins
02:57:37Megame quits [Client Quit]
03:38:03tzt (tzt) joins
03:50:03creaturesofthedark (creaturesofthedark) joins
05:06:04Wingy1139793760180 quits [Ping timeout: 240 seconds]
05:11:21thuban quits [Ping timeout: 265 seconds]
05:23:09thuban joins
05:42:43Wingy1139793760180 (Wingy) joins
06:14:16kwahoon joins
06:43:58sonick quits [Client Quit]
06:51:40Wingy1139793760180 quits [Ping timeout: 240 seconds]
07:11:24BlueMaxima quits [Client Quit]
09:34:17IDK (IDK) joins
09:38:16sec^nd quits [Ping timeout: 240 seconds]
09:38:43sec^nd (second) joins
09:53:56Wingy1139793760180 (Wingy) joins
10:20:59Wingy1139793760180 quits [Remote host closed the connection]
10:21:52Wingy1139793760180 (Wingy) joins
10:31:11tech_exorcist (tech_exorcist) joins
11:25:46Wingy1139793760180 quits [Client Quit]
11:26:51Wingy1139793760180 (Wingy) joins
11:58:13sec^nd quits [Remote host closed the connection]
11:58:54sec^nd (second) joins
12:20:39helder7 joins
12:21:00helder7 quits [Remote host closed the connection]
13:03:19Minkafighter quits [Quit: The Lounge - https://thelounge.chat]
13:04:02Minkafighter joins
13:18:36tech_exorcist quits [Client Quit]
13:21:11tech_exorcist (tech_exorcist) joins
13:22:16HackMii_ quits [Ping timeout: 240 seconds]
13:24:24HackMii_ (hacktheplanet) joins
13:34:27qwertyasdfuiopghjkl joins
13:50:36<bleb>is there a good/standard scheme for associating a URL with a piece of data in a filesystem
13:51:08<bleb>currently I hex encode the URL and use that as a name for a directory, within which there are files named after the timestamp, and a log file which records when the file has been fetched, and if it was new / unchanged / changed
13:51:18<bleb>but some filesystems limit filenames to 255 characters, so the scheme is not portable for URLs of 128 characters or more
13:51:33<bleb>I assume most web archives use a database but what would be a good way to store this data in a directory structure?
14:08:46Arcorann quits [Ping timeout: 240 seconds]
14:09:02tech_exorcist quits [Remote host closed the connection]
14:09:16tech_exorcist (tech_exorcist) joins
14:11:06Wingy1139793760180 quits [Remote host closed the connection]
14:11:55Wingy1139793760180 (Wingy) joins
14:12:59Mark joins
14:13:25Mark quits [Remote host closed the connection]
14:15:16sec^nd quits [Ping timeout: 240 seconds]
14:16:37<nimaje1>some filesystems have extended file attributes, but your tools (mostly copying/backup tools) have to be aware of them and they can be size limited too
14:17:43sec^nd (second) joins
14:18:25HP_Archivist (HP_Archivist) joins
14:27:07patrickg joins
14:34:04sec^nd quits [Remote host closed the connection]
14:34:23HP_Archivist quits [Client Quit]
14:34:24sec^nd (second) joins
14:40:01tech_exorcist quits [Remote host closed the connection]
14:42:24<bleb>nimaje1: goal is to have a format which can be zipped or tarballed, and wont depend on any particular filesystem
14:42:46HackMii_ quits [Ping timeout: 240 seconds]
14:43:26nimaje1 is now known as nimaje
14:43:34<bleb>maybe the simplest adaptation is to use sequential IDs for the directory names, and store a "url" file in each per-url directory
14:43:59<bleb>then I can have a separate text file to facilitate url -> ID mappings
14:51:08sec^nd quits [Remote host closed the connection]
14:54:18sec^nd (second) joins
15:40:46katocala quits [Remote host closed the connection]
15:41:23HackMii_ (hacktheplanet) joins
15:46:22Minkafighter quits [Client Quit]
16:04:22Minkafighter joins
16:13:43michaelblob_ quits [Read error: Connection reset by peer]
16:15:10michaelblob (michaelblob) joins
16:15:30michaelblob quits [Read error: Connection reset by peer]
16:16:14michaelblob (michaelblob) joins
16:16:40march_happy (march_happy) joins
16:22:22HackMii_ quits [Remote host closed the connection]
16:22:22sec^nd quits [Remote host closed the connection]
16:23:17sec^nd (second) joins
16:23:21HackMii_ (hacktheplanet) joins
16:28:33Minkafighter quits [Client Quit]
16:29:46Minkafighter joins
16:38:50Wingy1139793760180 quits [Remote host closed the connection]
16:39:41Wingy1139793760180 (Wingy) joins
16:42:23Minkafighter quits [Client Quit]
16:43:37Minkafighter joins
16:44:37Wingy1139793760180 quits [Remote host closed the connection]
16:45:27Wingy1139793760180 (Wingy) joins
16:48:34Minkafighter quits [Client Quit]
16:49:08Wingy1139793760180 quits [Read error: Connection reset by peer]
16:49:39Minkafighter joins
16:50:45Wingy1139793760180 (Wingy) joins
16:53:08Dragnog joins
16:53:50Wingy1139793760180 quits [Remote host closed the connection]
16:54:29Iki joins
16:54:38Wingy1139793760180 (Wingy) joins
16:59:08Wingy1139793760180 quits [Remote host closed the connection]
16:59:58Wingy1139793760180 (Wingy) joins
17:00:44<h2ibot>JAABot edited CurrentWarriorProject (-4): https://wiki.archiveteam.org/?diff=48781&oldid=48488
17:04:24Wingy1139793760180 quits [Remote host closed the connection]
17:07:03Wingy1139793760180 (Wingy) joins
17:08:45T31M quits [Quit: ZNC - https://znc.in]
17:09:42T31M joins
17:12:07Wingy1139793760180 quits [Remote host closed the connection]
17:12:55Wingy1139793760180 (Wingy) joins
17:15:52march_happy quits [Ping timeout: 265 seconds]
17:22:31Wingy1139793760180 quits [Remote host closed the connection]
17:23:20Wingy1139793760180 (Wingy) joins
17:27:33Wingy1139793760180 quits [Remote host closed the connection]
17:31:59systwi_ is now known as systwi
17:32:11Wingy1139793760180 (Wingy) joins
17:32:19Minkafighter quits [Client Quit]
17:33:32Minkafighter joins
17:38:49Wingy1139793760180 quits [Remote host closed the connection]
17:39:38Wingy1139793760180 (Wingy) joins
17:40:02lennier1 quits [Ping timeout: 265 seconds]
17:40:28systwi_ joins
17:41:44lennier1 (lennier1) joins
17:44:08Wingy1139793760180 quits [Remote host closed the connection]
17:44:24Megame (Megame) joins
17:44:56Wingy1139793760180 (Wingy) joins
17:53:02Minkafighter quits [Client Quit]
17:54:27Minkafighter joins
17:57:48Wingy1139793760180 quits [Remote host closed the connection]
18:06:26Wingy1139793760180 (Wingy) joins
18:10:08Wingy1139793760180 quits [Remote host closed the connection]
18:10:58Wingy1139793760180 (Wingy) joins
18:15:54Wingy1139793760180 quits [Read error: Connection reset by peer]
18:16:45Wingy1139793760180 (Wingy) joins
18:23:51Wingy1139793760180 quits [Remote host closed the connection]
18:26:24HiccupJul (HiccupJul) joins
18:26:43<HiccupJul>is warcat a good tool to merge WARCs made with grab-site?
18:27:26<HiccupJul>grab-site's github page suggests that i should use ArchiveWeb.Page to browse the archived data
18:27:44<HiccupJul>but i only browse one of the warcs, then i'd only be browsing an incomplete, i assume
18:40:29Wingy1139793760180 (Wingy) joins
18:45:03Wingy1139793760180 quits [Remote host closed the connection]
18:45:52Wingy1139793760180 (Wingy) joins
18:49:43tech_exorcist (tech_exorcist) joins
18:51:34jacobk quits [Ping timeout: 265 seconds]
18:56:38<@JAA>HiccupJul: A WARC tool that doesn't support consuming multiple WARCs sounds like a broken WARC tool to me. Looks like Ilya's focusing on his WACZ thing instead. https://github.com/webrecorder/replayweb.page/issues/91
18:57:41<@JAA>Maybe pywb would work for your case?
18:58:37<@JAA>Anyway, if you do need to merge WARCs, you can just concatenate them with `cat` or whatever. This works fine with .warc and .warc.gz (but not with .warc.zst, generally).
18:59:35<HiccupJul>ah
18:59:42<HiccupJul>well maybe it will automatically find the second warc
18:59:48<HiccupJul>but if it doesn't, i'll just use cat, thanks
19:13:48bleb quits [Ping timeout: 265 seconds]
19:20:23cm joins
19:38:04cm quits [Ping timeout: 240 seconds]
19:49:35<HiccupJul>seems like you can add multiple warcs to a single "collection" in archiveweb.page
19:50:07<HiccupJul>but yeah pywb would probably work as an alternative playback tool
19:50:32msrn_ quits [Ping timeout: 265 seconds]
19:55:50mikael joins
19:58:03hackbug (hackbug) joins
19:59:29cm joins
20:08:29jacobk joins
20:08:35HiccupJul quits [Client Quit]
20:14:53Megame quits [Client Quit]
20:16:46Wingy1139793760180 quits [Remote host closed the connection]
20:17:36Wingy1139793760180 (Wingy) joins
20:22:56HackMii_ quits [Remote host closed the connection]
20:22:57sec^nd quits [Remote host closed the connection]
20:24:08sec^nd (second) joins
20:26:09HackMii_ (hacktheplanet) joins
20:28:37HackMii_ quits [Remote host closed the connection]
20:28:58HackMii_ (hacktheplanet) joins
20:36:47benjinsmith joins
20:38:23benjins quits [Ping timeout: 265 seconds]
20:40:58HackMii_ quits [Remote host closed the connection]
20:41:21HackMii_ (hacktheplanet) joins
20:52:49benjinsmith is now known as benjins
20:58:12jacobk quits [Ping timeout: 265 seconds]
20:59:26Wingy1139793760180 quits [Remote host closed the connection]
20:59:55jacobk joins
21:00:17Wingy1139793760180 (Wingy) joins
21:02:40sec^nd quits [Remote host closed the connection]
21:02:41HackMii_ quits [Write error: Broken pipe]
21:04:19sec^nd (second) joins
21:04:28thuban quits [Ping timeout: 240 seconds]
21:04:30HackMii_ (hacktheplanet) joins
21:06:09thuban joins
21:10:42thuban quits [Read error: Connection reset by peer]
21:11:06thuban joins
21:15:36jacobk quits [Ping timeout: 265 seconds]
21:18:59thuban quits [Ping timeout: 265 seconds]
21:20:15thuban joins
21:23:59HackMii_ quits [Remote host closed the connection]
21:24:38HackMii_ (hacktheplanet) joins
21:29:02spirit quits [Quit: Leaving]
21:31:16thuban quits [Ping timeout: 240 seconds]
21:35:16thuban joins
21:36:40HackMii_ quits [Remote host closed the connection]
21:37:22HackMii_ (hacktheplanet) joins
21:51:10march_happy (march_happy) joins
22:09:44leo60228 quits [Ping timeout: 265 seconds]
22:10:57leo60228 (leo60228) joins
22:11:04Wingy1139793760180 quits [Read error: Connection reset by peer]
22:11:38tech_exorcist quits [Client Quit]
22:11:57Wingy1139793760180 (Wingy) joins
22:16:54HackMii_ quits [Remote host closed the connection]
22:17:23HackMii_ (hacktheplanet) joins
22:19:57Wingy1139793760180 quits [Remote host closed the connection]
22:20:49Wingy1139793760180 (Wingy) joins
22:36:28cm quits [Ping timeout: 240 seconds]
22:57:05cm joins
23:11:56<pabs>this person died: https://uhura.com/ https://en.wikipedia.org/wiki/Nichelle_Nichols
23:12:58Dragnog quits [Client Quit]
23:15:06Arcorann (Arcorann) joins
23:28:20Discant joins
23:31:20Lord_Nightmare quits [Quit: ZNC - http://znc.in]
23:36:46Lord_Nightmare (Lord_Nightmare) joins
23:41:43sec^nd quits [Remote host closed the connection]
23:41:43HackMii_ quits [Write error: Broken pipe]
23:42:30HackMii_ (hacktheplanet) joins
23:44:25sec^nd (second) joins
23:49:15BlueMaxima joins
23:55:31pabs quits [Remote host closed the connection]
23:56:04tzt quits [Ping timeout: 240 seconds]
23:58:05pabs (pabs) joins

HomeSearchPrevious dayNext day