| 00:39:01 | | BlueMaxima joins |
| 01:00:01 | | dm4v quits [Client Quit] |
| 01:02:56 | | dm4v joins |
| 01:02:58 | | dm4v is now authenticated as dm4v |
| 01:02:58 | | dm4v quits [Changing host] |
| 01:02:58 | | dm4v (dm4v) joins |
| 01:05:04 | | Ruthalas quits [Ping timeout: 252 seconds] |
| 01:05:36 | | Ruthalas (Ruthalas) joins |
| 01:08:01 | | Guest69 quits [Client Quit] |
| 01:54:38 | | Arcorann_ quits [Ping timeout: 258 seconds] |
| 03:36:38 | | @Fusl quits [Excess Flood] |
| 03:36:56 | | Fusl (Fusl) joins |
| 03:36:56 | | @ChanServ sets mode: +o Fusl |
| 03:49:24 | | qwertyasdfuiopghjkl joins |
| 04:04:33 | | qw3rty_ joins |
| 04:08:25 | | qw3rty__ quits [Ping timeout: 265 seconds] |
| 04:58:19 | | sonick quits [Client Quit] |
| 05:26:49 | | paul2520 quits [Remote host closed the connection] |
| 06:05:31 | | Arcorann_ joins |
| 06:07:57 | | sonick (sonick) joins |
| 06:08:01 | | HackMii_ quits [Remote host closed the connection] |
| 06:08:37 | | HackMii_ (hacktheplanet) joins |
| 06:55:33 | | BlueMaxima quits [Read error: Connection reset by peer] |
| 06:56:49 | | HackMii_ quits [Remote host closed the connection] |
| 06:57:07 | | HackMii_ (hacktheplanet) joins |
| 08:19:07 | | Stiletto quits [Ping timeout: 258 seconds] |
| 09:58:48 | | spirit joins |
| 10:06:50 | | shoghicp quits [Ping timeout: 258 seconds] |
| 10:11:37 | | shoghicp (shoghicp) joins |
| 10:28:41 | | HackMii_ quits [Ping timeout: 258 seconds] |
| 10:30:31 | | HackMii_ (hacktheplanet) joins |
| 11:54:37 | | billy549 quits [Ping timeout: 252 seconds] |
| 12:14:14 | | billy549 (Billy549) joins |
| 14:44:23 | | spirit quits [Client Quit] |
| 14:45:47 | | qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds] |
| 14:54:54 | | paul2520 (paul2520) joins |
| 15:22:31 | | Arcorann_ quits [Ping timeout: 252 seconds] |
| 15:22:40 | | spirit joins |
| 16:17:46 | | pabs quits [Ping timeout: 265 seconds] |
| 17:44:15 | | sec^nd quits [Remote host closed the connection] |
| 17:44:35 | | Stiletto joins |
| 17:44:44 | | sec^nd (second) joins |
| 18:19:58 | <tech234a> | IRCCloud just sent me an alert about 4 hours of planned downtime this weekend, so I might not have logs for that time period https://twitter.com/IRCCloud/status/1461703768354050058 |
| 18:57:37 | | Viniter6926 (Viniter) joins |
| 19:01:12 | | Viniter692 quits [Ping timeout: 258 seconds] |
| 19:01:12 | | Viniter6926 is now known as Viniter692 |
| 19:26:58 | | paul2520 quits [Remote host closed the connection] |
| 19:31:55 | | Ruthalas5 (Ruthalas) joins |
| 19:33:52 | | Ruthalas quits [Ping timeout: 252 seconds] |
| 19:33:52 | | Ruthalas5 is now known as Ruthalas |
| 19:59:36 | | Rome joins |
| 20:00:12 | <Rome> | Is this the right channel for asking how to combine warc.gz files into one? |
| 20:27:06 | | qwertyasdfuiopghjkl joins |
| 20:29:33 | <@JAA> | Rome: Yeah, this channel works. If they're valid WARCs, you can simply concatenate them with `cat` or similar. |
| 20:29:44 | <@JAA> | (Note, this is not possible with .warc.zst files.) |
| 20:31:12 | <Rome> | So if I have a dozen or so warc files from the same site and I combine them, does this actually keep duplicate data or does it get deduplicated? I tried warcat but it always crashed somehow. |
| 20:31:27 | <Rome> | (warc.gz files) |
| 20:32:57 | <@JAA> | Well, if you concatenate them, the data stays the same. Deduplication of contents is much trickier. |
| 20:33:20 | <Rome> | :( |
| 20:33:51 | <@JAA> | I've been working on a tool that will make that possible quite easily though. So maybe soon. |
| 20:34:12 | <@JAA> | Still requires a fair bit of work and validation though. |
| 20:34:51 | <Rome> | I just want to combine my 50TB of warc.gz files into something smaller. I have a script that goes thorugh site,checks for new content and then downloads it into a warc.gz, but most of the chnages are only like one new page or so so it would be good to get rid of all the duplkcitae data and combine all of them (up to a certain size). |
| 20:37:28 | <@JAA> | Yeah. WARC has revisit records for exactly that purpose. Basically, you only store a reference to the previous response record plus the headers of the new response but drop the entire new response body. But doing that after the fact is hard. We've done it before in pipeline scripts, but that used warcio, which is now known to mangle WARCs in a number of ways. |
| 20:38:03 | <Rome> | Shame |
| 20:38:45 | <Rome> | I hope you get to your tool soon then |
| 21:18:36 | <@JAA> | As do I, since I need it as well for other things. :-) |
| 21:25:40 | | BlueMaxima joins |
| 22:36:48 | | pabs (pabs) joins |