| 02:12:56 | | Webuser446283 joins |
| 02:16:16 | <Webuser446283> | SPN has been intermittently overloaded for around a week. When will the situation improve? It takes around 30 minutes to an hour just for saving one web page. |
| 03:31:08 | | DogsRNice_ quits [Read error: Connection reset by peer] |
| 04:56:20 | <Webuser446283> | The overload is probably over. Thank you. |
| 04:56:34 | | Webuser446283 leaves |
| 07:40:52 | | SootBector quits [Remote host closed the connection] |
| 07:42:03 | | SootBector (SootBector) joins |
| 11:02:00 | | igloo22225 quits [Quit: The Lounge - https://thelounge.chat] |
| 11:02:39 | | igloo22225 (igloo22225) joins |
| 15:03:58 | | kline (kline) joins |
| 15:04:58 | <kline> | any tips on what determines `ia`s performance ? for some files of roughly the same size it can vary from 2.5h to 20 mins to upload, and im naturally keen on it going faster. i dont think my local pipe is the issue. https://paste.sqt.wtf/a7c0de |
| 15:05:42 | <justauser> | Our wiki has some tips involving TCP congestion control. |
| 15:06:02 | <justauser> | https://wiki.archiveteam.org/index.php/Internet_Archive#Upload_speed |
| 15:55:46 | | nulldata-alt1 quits [Read error: Connection reset by peer] |
| 15:56:40 | | nulldata-alt1 (nulldata) joins |
| 16:07:49 | <kline> | thank you |
| 16:53:34 | | BearFortress_ joins |
| 16:57:15 | | BearFortress quits [Ping timeout: 272 seconds] |
| 17:03:35 | | BearFortress_ quits [Ping timeout: 272 seconds] |
| 17:05:07 | | BearFortress joins |
| 19:52:01 | | HP_Archivist quits [Quit: Leaving] |
| 20:50:42 | | HP_Archivist (HP_Archivist) joins |
| 21:40:30 | <IDK> | Nice to see IA properly distributing uploads to its fleet of datanodes, hopefully this will reduce load 🙏 |
| 21:42:13 | <IDK> | it used to be just a few |
| 23:06:04 | | Webuser589692 joins |
| 23:10:55 | <Webuser589692> | SPN has been intermittently overloaded for around a week, and it currently takes an hour and 20 minutes to start a capture. When will the situation improve? The overload occurs almost every day. |
| 23:11:58 | <pokechu22> | I don't think anyone here knows for sure. I've seen that notice but my SPNs have been going through pretty quickly (but that might be because I'm signed in or because I'm not making a lot of them) |
| 23:18:48 | <klea> | Sometimes I noticed that yes, there's queues sometimes, but not always. |
| 23:42:52 | <Webuser589692> | I think the overload is caused by too many brozzler instances bacause I have ovserved PDF files are saved quickly. |
| 23:44:25 | <klea> | I wonder if IA has separate queues for pdf/static files and the ones where brozzler has to run. |
| 23:45:42 | <pokechu22> | I'm pretty sure it does work that way; the SPN2 google doc mentions that there's a field you can set to bypass brozzler as well if you know ahead of time it's a file and not a webpage that needs JS (or something like that) |
| 23:46:47 | <klea> | Yeah, I suppose they want to avoid using resources when not needed. |
| 23:47:05 | <@JAA> | As I understand it, by default, it does a HEAD request, and if it gets a 200 with a Content-Type known to not need a browser, it will be archived without brozzler. |
| 23:47:26 | <@JAA> | And you can suppress that behaviour and force the use of brozzler via force_get on the API. |
| 23:51:00 | <klea> | I guess that's why my requests have mostly gone to brozzler except some txt files. |
| 23:58:10 | <TheTechRobo> | The weird part is that check is done *in* the brozzler worker. So not sure how that works, unless there's a second check earlier in the chain. |