| 00:00:06 | <@JAA> | I.e. it isn't. |
| 00:00:51 | | dm4v quits [Read error: Connection reset by peer] |
| 00:01:14 | <pabs> | aha |
| 00:04:39 | | dm4v joins |
| 00:04:41 | | dm4v is now authenticated as dm4v |
| 00:04:41 | | dm4v quits [Changing host] |
| 00:04:41 | | dm4v (dm4v) joins |
| 00:16:45 | | dfgffgd joins |
| 00:37:16 | | BlueMaxima joins |
| 01:02:41 | | dm4v quits [Read error: Connection reset by peer] |
| 01:03:34 | | dm4v joins |
| 01:03:36 | | dm4v is now authenticated as dm4v |
| 01:03:36 | | dm4v quits [Changing host] |
| 01:03:36 | | dm4v (dm4v) joins |
| 01:22:56 | | duce1337 quits [Client Quit] |
| 02:40:20 | | duce1337 joins |
| 03:28:41 | | qw3rty_ joins |
| 03:32:19 | | qw3rty__ quits [Ping timeout: 258 seconds] |
| 05:29:20 | <systwi> | JAA: I saw you earlier mention using zstd before. How would you say it compares to LZMA4? I've always gone with 7-Zip's highest compression using LZMA4 and, in general, the files have tended to be the smallest. I know it ultimately depends on the input complexity, but is zstd generally better? |
| 05:30:12 | <systwi> | I personally care more about a smaller output rather than the time it takes to compress/decompress. |
| 05:55:17 | <pabs> | how deep does ArchiveBot go when downloading a site? does it also archive pages on external sites linked from the site? |
| 06:01:08 | <systwi> | pabs: It will go one level outside the hostname specified. E.g., if you run "!a http://example.com/" and it has a link to "https://eff.org/", it will grab the latter, too, but only that page and its assets. It won't start grabbing all of eff.org, too. |
| 06:23:57 | | grawity quits [Remote host closed the connection] |
| 06:24:56 | | grawity (grawity) joins |
| 06:25:36 | <pabs> | and it won't grab pages that are linked to on the second domain? |
| 07:21:40 | <systwi> | No, it will skip them. |
| 07:23:43 | <pabs> | ok, thanks |
| 07:24:12 | <systwi> | You're welcome |
| 07:26:32 | | Matthww86 joins |
| 07:27:32 | | Matthww8 quits [Ping timeout: 250 seconds] |
| 07:27:32 | | Matthww86 is now known as Matthww8 |
| 07:38:22 | | Matthww8 quits [Ping timeout: 250 seconds] |
| 07:43:24 | | BlueMaxima quits [Client Quit] |
| 07:52:27 | | Matthww8 joins |
| 08:09:56 | | godane (godane) joins |
| 09:03:44 | | HP_Archivist quits [Ping timeout: 250 seconds] |
| 10:00:31 | | duce1337 is now authenticated as duce1337 |
| 10:00:31 | | duce1337 quits [Changing host] |
| 10:00:31 | | duce1337 (duce1337) joins |
| 10:20:27 | | spirit joins |
| 12:03:13 | | yano quits [Read error: Connection reset by peer] |
| 12:03:57 | | yano (yano) joins |
| 12:04:00 | | psy quits [Ping timeout: 250 seconds] |
| 12:04:15 | | noteness quits [Read error: Connection reset by peer] |
| 12:04:32 | | noteness (noteness) joins |
| 12:06:44 | | psy (psy) joins |
| 12:44:31 | | Matthww88 joins |
| 12:45:05 | | bsmith093 quits [Ping timeout: 258 seconds] |
| 12:45:28 | | Matthww8 quits [Ping timeout: 258 seconds] |
| 12:45:28 | | Matthww88 is now known as Matthww8 |
| 12:45:34 | | bsmith093 joins |
| 12:45:34 | | bsmith093 is now authenticated as bsmith093 |
| 13:00:59 | | Iki joins |
| 13:33:13 | | dfgffgd leaves |
| 13:38:02 | | Matthww8 quits [Ping timeout: 250 seconds] |
| 13:42:06 | | Matthww8 joins |
| 13:54:56 | | lunik1 quits [Quit: :x] |
| 13:55:27 | | lunik1 joins |
| 13:59:18 | | lunik1 quits [Client Quit] |
| 13:59:44 | | lunik1 joins |
| 14:21:53 | | lunik1 quits [Client Quit] |
| 14:22:52 | | lunik1 joins |
| 14:24:22 | | Stiletto quits [Ping timeout: 258 seconds] |
| 15:17:39 | | Arcorann_ quits [Ping timeout: 258 seconds] |
| 16:02:53 | | thuban quits [Ping timeout: 258 seconds] |
| 16:19:00 | <@JAA> | systwi: I haven't experimented much with LZMA, but I believe it generally (with the right settings) achieves a higher compression ratio with a much longer runtime than zstd. But ultimately, you'll have to try it with the specific data you want to compress. |
| 17:06:24 | <spirit> | generally as a rule of thumb: zstd is like zip but usually way faster, lzma is a bit advanced compression ratio wise but super slow |
| 17:06:32 | <spirit> | zstd allows training with a dictionary though iirc |
| 17:07:31 | <spirit> | in other news, why the flying fuck are raspberry 4s still murdering SD cards :( |
| 17:10:26 | <@JAA> | Uh, no, zstd isn't like ZIP at all. It's purely a compression algorithm and format like Brotli, Deflate, LZ4, LZMA, and what have you. ZIP is an archive file format and more comparable to e.g. tar, but it has direct support for compression with a number of different algorithms. |
| 17:10:47 | <spirit> | sorry i meant it is pretty close to DEFLATE afaik |
| 17:10:58 | <@JAA> | Yeah, that's the most common one. ZIP also supports LZMA though. |
| 17:11:23 | <spirit> | huh |
| 17:12:48 | <@JAA> | But comparing zstd to ZIP with LZMA is still not fair. You'd have to put the zstd stream(s) into a ZIP (without another compression layer, obviously) for that. |
| 17:14:45 | <@JAA> | Also, Lempel-Ziv and all its derived algorithms (including LZMA and zstd) are all dictionary-based as far as I know. Not sure if any common implementations of the others allow you to train and specify a custom dict though. |
| 17:15:10 | | Stiletto joins |
| 17:15:48 | <@JAA> | But you can use a custom dict with zlib in Python, for example. |
| 17:16:13 | | spirit backs away into a hedge and pretends he never said anything |
| 17:17:57 | <@JAA> | I mean, your rule of thumb is a good one with s/zip/deflate/. :-) |
| 18:22:36 | | DogsRNice (Webuser299) joins |
| 18:24:55 | | HP_Archivist (HP_Archivist) joins |
| 18:25:27 | <lunik1> | LZMA is slower to compress and decompress, but you can generally get slightly better ratios out of it |
| 18:26:49 | <lunik1> | I don't think there's much point in using zip or bzip2 over zstd at this point, other than compatibility |
| 18:29:06 | <lunik1> | *using gzip |
| 19:47:24 | | thuban joins |
| 20:07:18 | | spirit quits [Client Quit] |
| 21:04:22 | | Barto quits [Ping timeout: 250 seconds] |
| 21:04:43 | | Barto (Barto) joins |
| 23:05:08 | | Arcorann_ joins |
| 23:17:24 | | HP_Archivist quits [Ping timeout: 250 seconds] |