| 02:02:08 | | TastyWiener95 (TastyWiener95) joins |
| 02:34:50 | | Meli quits [Ping timeout: 252 seconds] |
| 02:36:19 | | Meli (Meli) joins |
| 03:36:26 | | Meli quits [Ping timeout: 252 seconds] |
| 03:39:11 | | Meli (Meli) joins |
| 04:00:00 | | PredatorIWD joins |
| 04:01:25 | <PredatorIWD> | Few quick questions: 1. Can I set the Warrior to 6 concurrent items and "ArchiveTeam’s Choice" project and not worry about that being too much and getting IP banned from some sites? Does the warrior get limited by the archive team's central server based on the project or? |
| 04:01:31 | <PredatorIWD> | 2. The Warrior worked a day on the imgur project and the VirtualBox .vdi went from ~1GB to ~4GB+ in size, what is being stored here exactly and when is it cleared? Thanks |
| 04:04:44 | <@JAA> | 1. The only tracker-side rate limits we typically have are global. So no, at 6, you might get banned depending on which project is AT's Choice, unfortunately. |
| 04:06:14 | <@JAA> | 2. Nothing is kept indefinitely, but virtual disks aren't normal files, and they might be bigger than their contents. Cf. https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior#The_Warrior_virtual_machine_is_using_up_disk_space,_even_though_it's_not_running_a_project! |
| 04:46:02 | | fireonlive (fireonlive) joins |
| 05:01:37 | | Hans5958 quits [Quit: Reconnecting] |
| 05:01:50 | | Hans5958 (Hans5958) joins |
| 05:35:31 | | Meli quits [Ping timeout: 265 seconds] |
| 05:39:28 | | Meli (Meli) joins |
| 05:47:11 | <myself> | it would be brilliant if each project could have a recommended concurrency that's handed down from the tracker, and then "archiveteam's chosen project at archiveteam's chosen concurrency" could be the true fire-and-forget selection. Feature wishlist! |
| 05:47:27 | <myself> | Failing that, concurrency 1 is always safe and gets infinitely more work done than zero. |
| 05:54:25 | <@JAA> | No disagreement there :-) |
| 06:02:01 | | Munchkin joins |
| 06:05:14 | <Munchkin> | On the warrior docker gitHub under the mapping section there is an ENV for WARRIOR_ID. Where can I find/get this? |
| 06:07:04 | <@JAA> | I've been wondering why that's exposed at all. It shouldn't be. |
| 06:07:22 | <@JAA> | Consider it useful only for development/testing. |
| 06:07:37 | <Munchkin> | Okay ill ignore it |
| 06:07:40 | <Munchkin> | thanks |
| 06:10:03 | <Munchkin> | On another note where does the warrior store the temp data its downloading on the docker? I'd like to path it to a host dir |
| 06:36:38 | <PredatorIWD> | Yes @myself, that was the reason I asked my first question above. Some other nice things would be: Giving a set amount of PC resources that get automatically spread out across different projects, especially the simpler ones which just require clean URLs to do basic requests, |
| 06:36:53 | <PredatorIWD> | no reason most people cant do 10 of those at the same time as long as each project hits different websites |
| 06:38:48 | <PredatorIWD> | What we have now is enough for a lot of things I guess but there are a lot of improvements possible, with AI assisted coding it should get much easier to add these things in the coming years. I'm a nodejs dev so I doubt I could do too much quickly here but I'll look over the code at some point surely. |
| 06:41:52 | <@JAA> | lol AI |
| 06:42:10 | <@JAA> | But yes, improvements are certainly possible. |
| 06:43:05 | <@JAA> | The current production tracker isn't open-source, so adding that is more involved. |
| 06:44:20 | <PredatorIWD> | AI wont code complex projects anytime soon, but with a proper database/internet/fs access, even current 30B llama could easily handle the busywork like boilerplate code, code completion, overall improvement suggestion, writing tests, documentation etc. |
| 06:45:23 | <PredatorIWD> | Just having AI that can search the documentation for a language online and explain what specific functions do better is quite a lot as well |
| 06:45:47 | <@JAA> | For simpler code, you spend more time checking that the generated code is indeed correct than you would just writing it directly. For complex code, it can't be trusted anyway. |
| 06:46:36 | <PredatorIWD> | I dont know, in my experience its quicker to look things over than write everything out yourself, but it does depend on the codebase |
| 06:47:00 | <@JAA> | That's how you end up with subtle logic bugs that are hard to find. |
| 06:47:17 | <@JAA> | The internet is full of shitty code. An LLM trained on that isn't going to produce anything better. |
| 06:47:42 | <PredatorIWD> | I'll look over the architecture quickly now but the tracker sending # of recommended instances for a particular project doesn't seem like something too complex to add. It can just be a hardcoded value set by admins that gets changed when you get new info on site request limits |
| 07:26:51 | <PredatorIWD> | From what I see in the code, the quickest way to implement the feature of "Let Archive Team decide number of concurrent items for a project" is for the tracker to send a suggested number of those instances with each "GetItemFromTracker" client request which will then be used as the first argument in "LimitConcurrent" in the project specific pipeline file |
| 07:28:44 | <PredatorIWD> | @JAA, you mentioned that prod tracker isnt open source, is there anything that I wrote in the msg above that wouldn't fit in with its current code base, anything that I'm missing or do you think there is a better way to do this? |
| 07:32:42 | | Munchkin quits [Client Quit] |
| 10:18:13 | | TokieSan joins |
| 10:58:06 | | TokieSan quits [Ping timeout: 252 seconds] |
| 11:04:42 | | TokieSan joins |
| 11:16:56 | | vantec (vantec) joins |
| 11:31:51 | | monoxane quits [Remote host closed the connection] |
| 11:44:18 | | TokieSan quits [Ping timeout: 265 seconds] |
| 12:01:42 | | TokieSan joins |
| 12:14:54 | <@kaz> | limitconcurrent doesn't limit concurrent jobs (haha, naming is fun) in that way - I think we only use it to limit the amount of concurrent rsync threads used for upload |
| 12:18:04 | <PredatorIWD> | kaz: From what I see in an example here it's used for both? https://github.com/ArchiveTeam/seesaw-kit/blob/699b0d215768c2208b5b48844c9f0f75bd6a1cbc/examples/picplz-example-pipeline.py#LL61C19-L61C19 |
| 12:38:03 | | monoxane (monoxane) joins |
| 12:41:23 | | monoxane quits [Remote host closed the connection] |
| 12:47:04 | | monoxane (monoxane) joins |
| 13:15:00 | <@kaz> | ah in the example yes |
| 13:15:05 | <@kaz> | in the real world, I don't think we use that at all |
| 13:15:35 | <@kaz> | we wouldalso need a way to bypass it for people that don't want to use the 'standard' limit, via a flag or something |
| 13:18:07 | <masterX244> | could be done by setting "default" ratelimit as -1 and going to this logic when ratelimit is -1 since thats a unplausible value |
| 13:33:12 | | bilboed quits [Quit: The Lounge - https://thelounge.chat] |
| 13:33:34 | | bilboed joins |
| 15:06:42 | | zhongfu quits [Ping timeout: 252 seconds] |
| 16:28:50 | | andrew quits [Quit: ] |
| 16:29:46 | | andrew (andrew) joins |
| 16:43:30 | | TokieSan quits [Remote host closed the connection] |
| 18:39:55 | | inomyabcs joins |
| 20:37:52 | <TheTechRobo> | Might be better for that to be put in Seesaw in full rather than project-specific scripts?% |
| 21:36:28 | | woans (WOANS) joins |
| 22:45:52 | | Jake quits [Quit: Leaving for a bit!] |
| 22:47:33 | | Jake (Jake) joins |
| 22:50:53 | | woans quits [Client Quit] |
| 23:09:46 | <vokunal|m> | I'm getting this on every docker container "Tracker returned status code 403. The tracker has probably malfunctioned. Retrying after 300 seconds" |
| 23:10:41 | <vokunal|m> | That was after restarting the watchtower. before I restarted the watchtower, every container had this error for about a day"error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value" |
| 23:10:51 | <vokunal|m> | * That was after restarting the watchtower. before I restarted the watchtower, every container had this error for about a day. |
| 23:10:51 | <vokunal|m> | "error from daemon in stream: Error grabbing logs: invalid character '\\x00' looking for beginning of value" |
| 23:10:57 | <vokunal|m> | * That was after restarting the watchtower. before I restarted the watchtower, every container had this error for about a day. |
| 23:10:57 | <vokunal|m> | "error from daemon in stream: Error grabbing logs: invalid character '\\x00' looking for beginning of value" |
| 23:11:06 | <vokunal|m> | * I'm getting this on every docker container |
| 23:11:06 | <vokunal|m> | "Tracker returned status code 403. The tracker has probably malfunctioned. Retrying after 300 seconds" |
| 23:11:22 | <@JAA> | TheTechRobo: It *has* to go at that level. You can't control the overall concurrency from within the pipeline. |
| 23:12:26 | <@JAA> | PredatorIWD: It can't be done at the GetItemFromTracker level because ^ |
| 23:12:56 | <@JAA> | Probably the most sensible way to put it is in projects.json. |
| 23:28:15 | | brecheche joins |
| 23:47:36 | | inomyabcs quits [Client Quit] |
| 23:56:28 | | woans (WOANS) joins |