HomeSearchPrevious dayNext day

00:05:43Arcorann (Arcorann) joins
00:14:49DogsRNice_ joins
00:17:50DogsRNice quits [Ping timeout: 240 seconds]
00:46:36qwertyasdfuiopghjkl quits [Remote host closed the connection]
01:02:44DogsRNice__ joins
01:04:42qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins
01:06:47DogsRNice_ quits [Ping timeout: 272 seconds]
04:22:26<nicolas17>hm
04:22:47<nicolas17>JAA: is x-archive-size-hint supposed to the the size of the whole item?
04:24:20<nicolas17>"ia upload --debug myitem data/severalfiles*" shows it's sending the size of each individual file in the x-archive-size-hint
04:24:47<nicolas17>so it uploads a small file with x-archive-size-hint:176 Content-Length:176, then a larger file with x-archive-size-hint:129105920 Content-Length:129105920, seems to defeat the whole point of that header
04:24:52<nicolas17>however I may have an old version of ia
04:25:32<@JAA>nicolas17: Yes, the total size of the item.
04:25:58<@JAA>I thought that the CLI does it right, but I never really checked, I think.
04:26:25<@JAA>There's definitely code iterating over all files and summing their sizes. Or at least there was a couple years ago.
04:27:11<nicolas17>then again I also don't see x-amz-auto-make-bucket:1 in the first upload, or metadata
04:27:23<nicolas17>so maybe --debug is busted
04:59:36DogsRNice__ quits [Read error: Connection reset by peer]
08:12:24qwertyasdfuiopghjkl quits [Remote host closed the connection]
10:42:41qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins
11:50:47driib quits [Quit: The Lounge - https://thelounge.chat]
11:53:48murmur quits [Read error: Connection reset by peer]
11:53:52murmur joins
12:09:54driib (driib) joins
12:21:49driib quits [Client Quit]
12:22:58driib (driib) joins
13:09:50Arcorann quits [Ping timeout: 240 seconds]
14:59:29Iki1 joins
15:03:25Iki quits [Ping timeout: 272 seconds]
15:07:03qwertyasdfuiopghjkl quits [Remote host closed the connection]
15:12:07qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins
15:37:44c3manu (c3manu) joins
18:34:12katia_ (katia) joins
18:34:49katia_ quits [Remote host closed the connection]
18:45:13katia_ (katia) joins
19:01:10katia_ quits [Remote host closed the connection]
19:03:22nyany_ quits [Quit: (516): and then you went into taco bell without pants...and surprisingly you weren't the only one there without pants]
19:03:32nyany (nyany) joins
20:28:43DogsRNice joins
20:52:07line quits [Remote host closed the connection]
20:52:26line joins
20:55:51line quits [Remote host closed the connection]
20:58:23line joins
20:58:24line quits [Remote host closed the connection]
21:00:55line joins
21:01:37line quits [Remote host closed the connection]
21:02:33line joins
21:03:51line quits [Remote host closed the connection]
21:05:22line joins
21:06:30line quits [Remote host closed the connection]
21:06:55line joins
21:09:45line quits [Remote host closed the connection]
21:10:41line joins
21:16:10line quits [Remote host closed the connection]
21:17:01line joins
21:22:26line quits [Remote host closed the connection]
21:22:46line joins
21:24:38line quits [Remote host closed the connection]
21:25:48line joins
21:28:31line quits [Remote host closed the connection]
21:29:01line joins
21:34:36line quits [Remote host closed the connection]
21:34:56line joins
21:59:42line quits [Remote host closed the connection]
22:00:03line joins
22:05:29line quits [Remote host closed the connection]
22:05:50line joins
22:13:12Arcorann (Arcorann) joins
22:17:56c3manu quits [Read error: Connection reset by peer]
22:18:02c3manu (c3manu) joins
22:57:35<@arkiver>nicolas17: JAA: yeah the ia cli does it wrong. i pointed it out before, but no effort has been made to fix it
23:00:00<nicolas17>I'm uploading multi-file items from the largest file to the smallest, so it reduces the impact of that, and also reduces redundant derive tasks etc
23:00:35<@JAA>arkiver: Jake's normally quite responsive to PRs at least, so maybe I'll take a look.
23:07:04<@JAA>Oh, yeah, it doesn't even attempt to set it, it seems.
23:08:28<@JAA>There's a `--size-hint` option or you can specify the header directly, but that's it.
23:08:41<@JAA>And then on each individual upload, it sets the hint to the file's size if there isn't a header already.
23:17:35<@JAA>Should be easy to fix though. It already iterates through all files to get the total file count to know when to queue a derive. Adding a stat call and returning the size along with that is pretty easy.
23:19:40<nicolas17>does it send "don't derive" until the last file?
23:20:18<@JAA>Yes
23:21:24<nicolas17>redundant derives was the main thing I was trying to avoid by uploading files largest-first
23:21:35<nicolas17>seems that was unnecessary, but it still helps with the bad size hint
23:22:30<@JAA>There are cases where the derive thing doesn't work: https://github.com/jjjake/internetarchive/issues/288
23:23:16<@JAA>I've been uploading without deriving and then explicitly triggering a derive for a long time now.
23:24:23<@JAA>There's also this fun quirk: https://github.com/jjjake/internetarchive/issues/253
23:31:03<nicolas17>ugh
23:31:30<nicolas17>I also noticed --checksum sometimes doesn't work and re-uploads the file, idk if it's because IA didn't process the upload yet and it's not in the file list?
23:32:01<@JAA>That'd definitely be my first guess, yes.
23:32:45<@JAA>https://github.com/jjjake/internetarchive/issues/289
23:32:54<@JAA>Filed by yours truly five years ago. :-)
23:34:16<nicolas17>aaaaa stupid broken tooling >:(
23:34:50<@JAA>You can use ia-wait-item-tasks from my little-things to restart as soon as all pending tasks are processed.

HomeSearchPrevious dayNext day