HomeSearchPrevious dayNext day

00:16:07sec^nd quits [Ping timeout: 252 seconds]
00:24:21sec^nd (second) joins
00:29:12tzt quits [Ping timeout: 265 seconds]
00:30:29tzt (tzt) joins
00:53:02bonga quits [Read error: Connection reset by peer]
00:53:43bonga joins
00:55:33michaelblob quits [Read error: Connection reset by peer]
01:02:36dm4v_ joins
01:03:17dm4v quits [Read error: Connection reset by peer]
01:03:17dm4v_ is now known as dm4v
01:03:17dm4v quits [Changing host]
01:03:17dm4v (dm4v) joins
01:15:44michaelblob (michaelblob) joins
01:23:18michaelblob quits [Read error: Connection reset by peer]
01:26:14Arcorann quits [Ping timeout: 265 seconds]
01:36:45DiscantX joins
02:13:08michaelblob (michaelblob) joins
02:17:02tbc1887 (tbc1887) joins
02:21:04michaelblob quits [Read error: Connection reset by peer]
02:21:28ThreeHM quits [Ping timeout: 265 seconds]
02:22:55michaelblob (michaelblob) joins
02:23:04ThreeHM (ThreeHeadedMonkey) joins
02:28:07michaelblob quits [Read error: Connection reset by peer]
02:29:53michaelblob (michaelblob) joins
02:40:51DiscantX quits [Client Quit]
03:34:18Arcorann (Arcorann) joins
05:56:54benjins quits [Ping timeout: 265 seconds]
06:06:25tbc1887 quits [Read error: Connection reset by peer]
06:08:59thuban quits [Ping timeout: 265 seconds]
06:21:08bonga quits [Remote host closed the connection]
06:22:06bonga joins
06:47:16BlueMaxima quits [Client Quit]
06:56:21gazorpazorp quits [Ping timeout: 265 seconds]
06:57:21gazorpazorp (gazorpazorp) joins
07:14:37thuban joins
08:25:54qwertyasdfuiopghjkl quits [Ping timeout: 265 seconds]
08:42:39michaelblob quits [Ping timeout: 252 seconds]
09:18:33useretail_ joins
09:18:41thetechrobo_ joins
09:19:04immibis quits [Remote host closed the connection]
09:19:04useretail__ quits [Remote host closed the connection]
09:19:04TheTechRobo quits [Remote host closed the connection]
09:19:04gazorpazorp quits [Remote host closed the connection]
09:19:04driib0835367 quits [Client Quit]
09:19:04Ruthalas quits [Quit: Ping timeout (120 seconds)]
09:19:24immibis (immibis) joins
09:19:24Ruthalas (Ruthalas) joins
09:19:26user_ (gazorpazorp) joins
09:19:42immibis quits [Remote host closed the connection]
09:19:50user_ quits [Remote host closed the connection]
09:20:07user_ (gazorpazorp) joins
09:20:10driib0835367 (driib) joins
09:20:54immibis (immibis) joins
09:21:12immibis quits [Remote host closed the connection]
09:22:24immibis (immibis) joins
09:22:42immibis quits [Remote host closed the connection]
09:23:54immibis (immibis) joins
09:24:12immibis quits [Remote host closed the connection]
09:24:39immibis (immibis) joins
09:41:24benjins joins
10:00:41Mateon1 quits [Remote host closed the connection]
10:03:41Mateon1 joins
10:22:13kiska5 quits [Quit: Ping timeout (120 seconds)]
10:22:13Ruthalas quits [Client Quit]
10:22:13immibis quits [Remote host closed the connection]
10:22:13dm4v quits [Client Quit]
10:22:13benjins quits [Remote host closed the connection]
10:22:15kiska5 joins
10:22:18dm4v joins
10:22:19dm4v quits [Changing host]
10:22:19dm4v (dm4v) joins
10:22:24immibis (immibis) joins
10:22:42immibis quits [Remote host closed the connection]
10:22:42Ruthalas0 (Ruthalas) joins
10:23:41benjins joins
10:23:55benjins quits [Remote host closed the connection]
10:25:25immibis (immibis) joins
10:26:11benjins joins
10:26:25benjins quits [Remote host closed the connection]
10:28:09kiska51 joins
10:28:23immibis quits [Remote host closed the connection]
10:28:23kiska5 quits [Client Quit]
10:28:23kiska51 is now known as kiska5
10:28:41benjins joins
10:28:55benjins quits [Remote host closed the connection]
10:31:11benjins joins
10:31:25benjins quits [Remote host closed the connection]
10:33:41benjins joins
10:33:55benjins quits [Remote host closed the connection]
10:36:11benjins joins
10:36:25benjins quits [Remote host closed the connection]
10:38:41benjins joins
10:38:55benjins quits [Remote host closed the connection]
10:40:26benjins joins
11:10:59michaelblob (michaelblob) joins
11:11:23michaelblob quits [Remote host closed the connection]
11:12:30michaelblob (michaelblob) joins
11:12:53michaelblob quits [Remote host closed the connection]
11:14:00michaelblob (michaelblob) joins
11:14:23michaelblob quits [Remote host closed the connection]
11:15:30michaelblob (michaelblob) joins
11:15:53michaelblob quits [Remote host closed the connection]
11:16:16michaelblob (michaelblob) joins
11:44:25michaelblob quits [Ping timeout: 265 seconds]
11:54:58driib08353677 (driib) joins
11:58:27driib0835367 quits [Ping timeout: 252 seconds]
11:58:28driib08353677 is now known as driib0835367
12:16:10<thetechrobo_>Does megawarc automatically replace identical records with revisit records (and undos it when restoring back the original tar)? If not, is there a tool that does this? It doesn't have to be a megawarc tool, even just a tool that can be performed on a megawarc and can be undone would be fine
12:43:25LeGoupil joins
14:17:49qwertyasdfuiopghjkl joins
15:01:19michaelblob (michaelblob) joins
15:01:41michaelblob quits [Remote host closed the connection]
15:02:06Arcorann quits [Ping timeout: 265 seconds]
15:02:48michaelblob (michaelblob) joins
15:03:11michaelblob quits [Remote host closed the connection]
15:04:18michaelblob (michaelblob) joins
15:04:41michaelblob quits [Remote host closed the connection]
15:05:27bonga quits [Ping timeout: 252 seconds]
15:05:48michaelblob (michaelblob) joins
15:06:11michaelblob quits [Remote host closed the connection]
15:06:54bonga joins
15:07:18michaelblob (michaelblob) joins
15:07:41michaelblob quits [Remote host closed the connection]
15:08:48michaelblob (michaelblob) joins
15:09:11michaelblob quits [Remote host closed the connection]
15:10:18michaelblob (michaelblob) joins
15:10:41michaelblob quits [Remote host closed the connection]
15:11:48michaelblob (michaelblob) joins
15:12:11michaelblob quits [Remote host closed the connection]
15:12:33michaelblob (michaelblob) joins
16:13:31HP_Archivist (HP_Archivist) joins
16:15:18bonga quits [Ping timeout: 252 seconds]
16:30:40michaelblob_ (michaelblob) joins
16:33:27michaelblob quits [Ping timeout: 265 seconds]
16:33:59bonga joins
16:38:24bonga quits [Ping timeout: 252 seconds]
16:39:33bonga joins
16:46:06bonga quits [Ping timeout: 252 seconds]
16:46:42bonga joins
17:17:05HP_Archivist quits [Ping timeout: 265 seconds]
18:13:07HP_Archivist (HP_Archivist) joins
18:13:25HP_Archivist quits [Remote host closed the connection]
18:13:51HP_Archivist (HP_Archivist) joins
18:35:03benjinsmith joins
18:35:25benjinsmith quits [Remote host closed the connection]
18:35:29<@JAA>thetechrobo_: No, megawarc doesn't do that, and I'm not aware of any tool that does. At least not without fucking it up. We used warcio before (when we weren't aware of its horrible issues yet), and chromebot got yanked from the WBM in part because its revisit records were wrong.
18:36:21benjins quits [Ping timeout: 265 seconds]
18:37:41benjinsmith joins
18:37:55benjinsmith quits [Remote host closed the connection]
18:39:23AlsoHP_Archivist joins
18:40:11benjinsmith joins
18:40:25benjinsmith quits [Remote host closed the connection]
18:41:56benjinsmith joins
18:44:51AlsoHP_Archivist quits [Client Quit]
19:17:24driib0835367 quits [Client Quit]
19:17:41driib0835367 (driib) joins
19:18:16driib0835367 is now known as driib
19:21:09spirit quits [Client Quit]
19:28:52sonick quits [Quit: Connection closed for inactivity]
19:36:09bonga quits [Ping timeout: 265 seconds]
19:50:21Ruthalas0 quits [Ping timeout: 252 seconds]
19:53:06Ruthalas (Ruthalas) joins
20:00:28thetechrobo_ is now known as TheTechRobo
20:01:15<TheTechRobo>JAA: Oh, ok.
20:01:30benjinsmith is now known as benjins
20:01:35<TheTechRobo>I just have a bunch of WARCs with the exact same URLs being grabbed, but I want each WARC to be usable separately.
20:05:45monoxane4 quits [Ping timeout: 252 seconds]
20:14:15bonga joins
20:30:46bonga quits [Ping timeout: 265 seconds]
21:15:56<@JAA>TheTechRobo: How would revisit records help there? The files would be referencing each other, so they wouldn't be self-contained.
21:16:05<@JAA>Or you just mean revisit records within each WARC?
21:16:35<TheTechRobo><TheTechRobo> [...], even just a tool that can be performed on a megawarc **and can be undone** would be fine
21:16:50<TheTechRobo>Similar to how megawarc is reversible.
21:17:43<@JAA>Ah, missed that part somehow even though you mentioned it twice. *facepalm*
21:17:59<@JAA>I've never heard of anything expanding revisit records again.
21:18:17<TheTechRobo>That's unfortunate. It would be pretty helpful.
21:21:32<TheTechRobo>I suppose something similar could be done with a reversible warc.gz <-> zstd dictionary, not sure whether that'd be more or less complicated than reversible deduping with revisit records
21:23:36<@JAA>If you want bit-identical reproducibility, that's actually quite difficult.
21:24:41<@JAA>At the very least, you'd have to use the exact same compression settings and library versions as you did when the original file was produced, unless you store all of that extra detail in some way.
21:25:45<TheTechRobo>Shouldn't compression softwares be backwards compatible, though, so that you _don't_ need that one specific version?
21:26:50<@JAA>It's compatible as in able to decompress data produced with previous versions. That doesn't mean the compression algorithm doesn't occasionally get tweaked to obtain better compression ratios though.
21:27:54<@JAA>See e.g. here, patch version adjusting compression parameters for better performance: https://github.com/facebook/zstd/releases/tag/v1.5.1
21:28:38<TheTechRobo>Are we talking about the same thing? A warc.gz to zstd compressor wouldn't need to compress the same with an identical input. It would just need to have an equivalent decompressed form, or else it's not good for archival.
21:28:58<TheTechRobo>But it compressing the same way shouldn't matter, no?
21:29:12<@JAA>I thought you meant perfectly reversible conversion between .warc.gz and .warc.zst.
21:29:52<@JAA>For equivalent output, yeah, that doesn't matter.
21:30:45<TheTechRobo>I was more meaning that it doesn't have to compress the exact same between tries, but it has to decompress the same. Obviously the warc.gz to warc.zst doesn't have to be "perfectly" identical to another conversion. But it needs to be equivalent.
21:31:36<@JAA>pywarc will support that kind of conversion.
21:32:02<TheTechRobo>Cool.
21:32:04<@JAA>Not sure about expanding revisit records though, that seems like a pain to implement with very limited use cases.
21:32:37<TheTechRobo>Yeah. I think warc.zst can be used for almost the same use cases as the revisit records, but is significantly easier to impl.
21:34:00<@JAA>I'd suspect you'd get bigger files though, but that would require testing.
21:34:43<TheTechRobo>Do you mean bigger files where the compression is reversed, or bigger dictionarified files?
21:34:49<TheTechRobo>The latter wouldn't make much sense to me.
21:35:39<@JAA>Using revisit records should result in a smaller file size than using dict compression taking advantage of the repetitive data.
21:36:14<@JAA>Although it depends a bit on the details. Is there TE involved? What do the URLs look like? Etc.
21:36:32<@JAA>Anyway, I'm not sure why we're having this conversation in -ot. lol
21:36:34<TheTechRobo>Oh, you meant bigger than the revisit ones.
21:36:45<TheTechRobo>JAA: Would #-bs be better?
21:36:59<TheTechRobo>or #-dev?
21:37:32<@JAA>Probably -bs for what we've been talking about, and -dev for actual implementation stuff. But ¯\_(ツ)_/¯
21:49:12balrog quits [Ping timeout: 265 seconds]
21:51:28balrog (balrog) joins
21:53:20LeGoupil quits [Client Quit]
22:03:21sonick (sonick) joins
22:34:20pnJay___ is now known as pnJay
23:07:27Arcorann (Arcorann) joins
23:08:01mgrytbak quits [Quit: Ping timeout (120 seconds)]
23:09:07mgrytbak joins
23:21:40BlueMaxima joins
23:22:33dxrt joins
23:22:35dxrt quits [Changing host]
23:22:35dxrt (dxrt) joins
23:22:35@ChanServ sets mode: +o dxrt
23:34:55wessel1512 quits [Ping timeout: 265 seconds]
23:36:13wessel1512 joins
23:47:50Ruthalas quits [Client Quit]
23:51:44Ruthalas (Ruthalas) joins

HomeSearchPrevious dayNext day