| 02:27:12 | | pabs quits [Quit: Don't rest until all the world is paved in moss and greenery.] |
| 02:29:49 | | pabs (pabs) joins |
| 03:13:01 | | DogsRNice quits [Read error: Connection reset by peer] |
| 08:03:26 | | dxrt quits [Read error: Connection reset by peer] |
| 08:37:59 | | Dango360 quits [Ping timeout: 252 seconds] |
| 09:43:13 | | dxrt (dxrt) joins |
| 09:56:44 | | MrMcNuggets quits [Ping timeout: 252 seconds] |
| 10:05:29 | | MrMcNuggets (MrMcNuggets) joins |
| 12:10:32 | | systwi_9 joins |
| 12:13:43 | | nothere quits [Ping timeout: 268 seconds] |
| 12:15:44 | | Dango360 (Dango360) joins |
| 12:46:02 | | nothere joins |
| 17:02:30 | | thalia (thalia) joins |
| 17:31:20 | | dxrt quits [Quit: ZNC - http://znc.sourceforge.net] |
| 17:31:30 | | dxrt (dxrt) joins |
| 17:49:44 | | hexagonwin (hexagonwin) joins |
| 17:50:32 | <hexagonwin> | i'm attempting to upload more than 100TB total of data to IA, but upload speed is insanely slow (2s/MiB...) are there some tips to make it faster? |
| 17:51:06 | <hexagonwin> | i'm using the ia cli tool. tried using protonvpn(free) server located in san jose but doesn't seem to help much.. |
| 17:53:39 | <justauser> | There are some on the wiki. |
| 17:53:55 | <justauser> | https://wiki.archiveteam.org/index.php/Internet_Archive#Upload_speed |
| 18:30:39 | | HP_Archivist quits [Quit: Leaving] |
| 19:11:26 | | HP_Archivist (HP_Archivist) joins |
| 19:52:44 | | leo60228 quits [Read error: Connection reset by peer] |
| 19:52:53 | | leo60228 (leo60228) joins |
| 19:59:13 | <hexagonwin> | thanks a lot |
| 21:54:23 | <klea> | hexagonwin: If you want faster speed, using a VPN will not help, and in fact it will harm your goal, due to now adding more latency between you and the VPN, unless wherever you're does rate limiting for IA's IP space, but not the VPNs. |
| 22:16:09 | <TheTechRobo> | klea: The idea behind VPNing is getting better peering. IIRC some people have had success with it. |
| 22:26:36 | <nicolas17> | hexagonwin: on Linux, I found "sysctl net.ipv4.tcp_congestion_control=bbr" makes the biggest difference |
| 22:27:15 | <nicolas17> | also did you talk to IA (or at least arkiver) about that dataset? 100TB is a lot to upload without coordination |
| 22:41:58 | <hexagonwin> | i haven't; i guess i should. |
| 22:42:34 | <hexagonwin> | i did send a mail requesting a new collection to be made a few hours ago though |
| 22:49:20 | | DogsRNice joins |
| 22:49:35 | <klea> | huh, TIL. |
| 23:22:25 | <@JAA> | Yeah, that's big enough to require communication before the upload. |
| 23:22:40 | <@JAA> | arkiver: ^ hexagonwin wants to upload 100 TB of unspecified 'data'. |
| 23:29:44 | <nicolas17> | how much does 200TB of hard disks cost nowadays? ._. |
| 23:37:17 | <@JAA> | 3.5 unobtainium |
| 23:53:27 | <TheTechRobo> | If one requests all CDX records for a particular query to index them into their own database, is there a good way of incrementally adding to it later on? I thought of date filtering at first, but indexing delays would mess that up. |
| 23:55:43 | <TheTechRobo> | And simply requesting $(pagenum) until another page is available wouldn't work either except for matchType=exact because captures are sorted by urlkey first. |