| 01:17:44 | | qwertyasdfuiopghjkl quits [Client Quit] |
| 01:20:03 | | qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins |
| 01:24:34 | | qwertyasdfuiopghjkl quits [Client Quit] |
| 01:38:54 | | qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins |
| 02:19:18 | | Maturion quits [Ping timeout: 265 seconds] |
| 02:31:58 | | Maturion joins |
| 06:45:25 | <Maakuth|m> | OrIdow6: reminding about the mbnet warc thing |
| 08:40:19 | <@OrIdow6> | Will do it tomorrow |
| 09:50:25 | | qwertyasdfuiopghjkl quits [Client Quit] |
| 11:07:44 | | qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins |
| 14:00:48 | | systwi quits [Read error: Connection reset by peer] |
| 14:01:32 | | systwi (systwi) joins |
| 16:02:34 | | atphoenix_ quits [Ping timeout: 252 seconds] |
| 19:51:49 | | systwi_ (systwi) joins |
| 19:56:21 | | systwi quits [Ping timeout: 265 seconds] |
| 20:17:05 | | atphoenix_ (atphoenix) joins |
| 20:28:20 | | atphoenix_ quits [Remote host closed the connection] |
| 20:28:54 | | atphoenix_ (atphoenix) joins |
| 21:37:39 | <@OrIdow6> | Assuming these are acceptable to go into the WBM, would it be acceptable to have a few WARCs per IA item? Seems they're in the 100s of MBs range so I'd put them in groups of 5 GB or so and upload those |
| 21:40:21 | <pokechu22> | I don't see why that'd cause a problem - archivebot does split warcs like that too |
| 21:40:34 | <pokechu22> | (but I haven't done any manual warc uploading myself) |
| 21:41:46 | <@JAA> | What's the total size? |
| 21:49:33 | <@OrIdow6> | JAA: 27 .3GiB |
| 21:49:38 | <@OrIdow6> | *27.3 |
| 21:50:13 | <@JAA> | Cute |
| 21:50:20 | <@JAA> | That can just go in a single item. |
| 21:50:45 | <@OrIdow6> | Oh, ok |
| 21:51:05 | <@OrIdow6> | What's the recommended item size these days? I thought I remember a soft limit below that |
| 21:51:50 | <@JAA> | The hard limit is 1 TiB, and it's usually best to not fly too close to the sun. |
| 21:52:28 | <@JAA> | Recommendations would depend on who you ask. |
| 21:53:19 | <@JAA> | I don't even hesitate to put 800 GiB in a single item when that's how large a self-contained collection of files is. |
| 21:54:07 | <@JAA> | One problem here might be the number of files though. It's ~36k WARCs, right? |
| 21:54:59 | <@OrIdow6> | Yeah, but I intend to megawarc or otherwise merge them |
| 21:55:33 | <@JAA> | Yep, that will be necessary. |
| 21:56:58 | <@JAA> | One megawarc per tar listed in the pad seems reasonable. Or just a single megawarc would be fine, too, of course. |
| 21:58:58 | <@OrIdow6> | Alright |
| 22:25:36 | <@OrIdow6> | What Python version does megawarc work with? |
| 22:27:18 | <@JAA> | Pretty sure it's still Python 2 only. |
| 22:33:16 | <@OrIdow6> | Hm, that's what I thought |
| 22:36:59 | <@OrIdow6> | If I'm not mistaken https://github.com/ArchiveTeam/megawarc/commit/5468d80e35b3dcb85d36624580c813326af706fe broke tar input by checking the extension "filename" in test_gz; filename is the extension of the compressed WARC when operating on individual files, but of the input .tar when operating on it |
| 22:38:22 | <@JAA> | We never use tar input, so that's certainly possible. |
| 22:39:06 | <@OrIdow6> | *extension of |
| 22:39:22 | <@OrIdow6> | *is the name of the |
| 22:49:03 | <@OrIdow6> | Alright, off it goes |