| 01:10:11 | | rewby quits [Ping timeout: 252 seconds] |
| 01:28:55 | | AlsoHP_Archivist joins |
| 01:32:11 | | HP_Archivist quits [Ping timeout: 252 seconds] |
| 02:13:57 | | rewby (rewby) joins |
| 04:03:16 | | Barto quits [Ping timeout: 252 seconds] |
| 04:36:11 | <nicolas17> | JAA: does archive.org support multi-part uploads? |
| 04:37:22 | <nicolas17> | I'm getting awful speeds uploading 1 file |
| 04:37:31 | <nicolas17> | uploading 2 files I don't seem to get a further slowdown |
| 04:38:20 | <nicolas17> | so it seems if I could upload 10 files at once it would be good for throughput... but I can't fit 10 files in my VPS at once :P |
| 04:39:25 | <@JAA> | nicolas17: Yes, but the processing after completion is very slow. |
| 04:39:44 | <@JAA> | ia-upload-stream in my little-things repo implements it. |
| 04:40:02 | <@JAA> | As the name suggests, originally written to upload a stream of data rather than a file from disk. |
| 04:41:40 | <@JAA> | Which means it buffers in memory, of course. |
| 04:41:55 | <nicolas17> | I have even less RAM than disk |
| 04:41:56 | <nicolas17> | :D |
| 04:42:17 | <@JAA> | Who doesn't? :-P |
| 04:42:22 | <nicolas17> | I mean like |
| 04:42:29 | <nicolas17> | these are 7GB files |
| 04:42:41 | <nicolas17> | my VPS has a total of 25GB SSD and 1GB RAM |
| 04:42:47 | <@JAA> | Ah |
| 04:42:49 | <@JAA> | Welp |
| 04:42:54 | <fireonlive> | f |
| 04:43:46 | <nicolas17> | uploading from VPS I get 1MiB/s, uploading from home I get like 6s/MiB |
| 04:44:23 | <@JAA> | Upload from home but route your traffic through the VPS? |
| 04:44:58 | <JTL> | so that isn't a typo of 6MiB/s? |
| 04:45:01 | <@JAA> | Can be as simple as a SOCKS proxy with `ssh -D`. |
| 04:45:09 | <JTL> | I concur that low specs like that would be an issue |
| 04:45:28 | <@JAA> | JTL: 6s/MiB is totally realistic for IA uploads, sadly. |
| 04:45:35 | <nicolas17> | JTL: no, the internetarchive python module uses tqdm for the progress bar, which switches to "seconds per megabyte" when it goes below 1.0 |
| 04:45:41 | <JTL> | ahh |
| 04:45:43 | <JTL> | welp |
| 04:45:46 | <nicolas17> | instead of KiB/s |
| 04:45:55 | <@JAA> | mebibyte* |
| 04:45:57 | <JTL> | yeah.... |
| 04:46:18 | <JTL> | If you have a "better" system locally I'd start by futzing with SOCKS and other proxy things |
| 04:46:55 | <nicolas17> | well, locally I have a lot of storage and a lot of RAM but the maximum upload speed I can possibly get is 3MiB/s, so that's still suboptimal :) |
| 04:47:07 | <nicolas17> | would be better than the current 6s/MiB but... |
| 04:47:41 | <nicolas17> | <nicolas17> someone should give me ssh to their well-connected-to-IA server >.> |
| 04:49:35 | <@JAA> | Do you have the data locally anyway? |
| 04:50:19 | <nicolas17> | no, I can get it from login-walled developer.apple.com |
| 04:50:28 | <@JAA> | Ah, right. |
| 04:50:52 | <nicolas17> | would digitalocean in SFO be any better than digitalocean in NYC? |
| 04:51:10 | <nicolas17> | I don't know what's the bottleneck here |
| 04:51:38 | | Barto (Barto) joins |
| 04:51:57 | <@JAA> | Very possible. Network proximity seems to be a very significant factor. |
| 05:11:14 | <nicolas17> | yesss this is much better |
| 05:11:47 | <nicolas17> | of course when I get 200Mbps I wish for 1Gbps but |
| 05:16:29 | <fireonlive> | "now I want more >:(" |
| 05:16:49 | <fireonlive> | i feel ya |
| 05:17:50 | <@JAA> | https://i.kym-cdn.com/photos/images/original/001/791/790/48d.png |
| 05:29:47 | <nicolas17> | peaks of 80MiB/s, I'm gonna cry of joy |
| 05:41:33 | | nicolas17 quits [Remote host closed the connection] |
| 05:56:54 | | nicolas17 joins |
| 06:35:11 | | datechnoman quits [Quit: The Lounge - https://thelounge.chat] |
| 06:36:17 | | datechnoman (datechnoman) joins |
| 08:55:22 | <nicolas17> | finished uploading, that was over 200GB |
| 09:17:15 | | nicolas17 quits [Client Quit] |
| 12:36:52 | | sonick quits [Client Quit] |
| 14:35:11 | | AlsoHP_Archivist quits [Client Quit] |
| 14:35:32 | | HP_Archivist (HP_Archivist) joins |
| 14:39:50 | | spirit quits [Quit: Leaving] |
| 19:01:37 | | nicolas17 joins |
| 19:57:47 | | HP_Archivist quits [Client Quit] |
| 23:37:41 | | wessel1512 quits [Ping timeout: 252 seconds] |
| 23:59:29 | | Iki1 quits [Read error: Connection reset by peer] |
| 23:59:47 | | Justin[home] joins |
| 23:59:47 | | Justin[home] is now authenticated as DopefishJustin |
| 23:59:50 | | Iki1 joins |