| 00:01:16 | | Chris5010 (Chris5010) joins |
| 02:26:22 | | Exorcism8 (exorcism) joins |
| 02:28:01 | | Exorcism quits [Read error: Connection reset by peer] |
| 02:28:01 | | Exorcism8 is now known as Exorcism |
| 03:20:07 | | Peroniko joins |
| 03:20:12 | | Peroniko is now authenticated as Peroniko |
| 03:22:15 | | RetiredTurtle quits [Ping timeout: 265 seconds] |
| 06:27:53 | | project10 quits [Ping timeout: 252 seconds] |
| 06:29:17 | | project10 (project10) joins |
| 06:36:08 | | datechnoman quits [Quit: The Lounge - https://thelounge.chat] |
| 06:38:20 | | datechnoman (datechnoman) joins |
| 06:38:29 | | @flashfire42 quits [Client Quit] |
| 06:38:29 | | kiska quits [Client Quit] |
| 06:40:09 | | flashfire42 joins |
| 06:41:24 | | kiska (kiska) joins |
| 07:00:45 | | flashfire42 is now authenticated as flashfire42 |
| 07:00:45 | | @ChanServ sets mode: +o flashfire42 |
| 11:44:14 | | Exorcism quits [Remote host closed the connection] |
| 11:45:24 | | Exorcism (exorcism) joins |
| 12:45:41 | | Peronikola joins |
| 12:48:29 | | Peroniko quits [Ping timeout: 252 seconds] |
| 15:04:32 | | datechnoman quits [Ping timeout: 265 seconds] |
| 15:19:46 | | kiryu quits [Client Quit] |
| 15:28:45 | | datechnoman (datechnoman) joins |
| 15:34:09 | | Peronikola is now authenticated as Peroniko |
| 15:34:13 | | Peronikola is now known as Peroniko |
| 17:21:38 | | Peronikola joins |
| 17:21:48 | | Peronikola is now authenticated as Peroniko |
| 17:24:02 | | Peroniko quits [Ping timeout: 252 seconds] |
| 17:29:30 | | kiska quits [Client Quit] |
| 17:29:30 | | @flashfire42 quits [Client Quit] |
| 17:32:19 | | flashfire42 joins |
| 17:33:33 | | kiska (kiska) joins |
| 19:03:27 | | Exorcism quits [Remote host closed the connection] |
| 19:04:11 | | Exorcism (exorcism) joins |
| 19:20:30 | | Exorcism quits [Remote host closed the connection] |
| 19:22:57 | | Exorcism (exorcism) joins |
| 20:34:05 | | Exorcism quits [Remote host closed the connection] |
| 20:34:58 | | Exorcism (exorcism) joins |
| 20:46:17 | | Maturion joins |
| 20:46:29 | | Peronikola is now known as Peroniko |
| 20:51:32 | | Exorcism quits [Remote host closed the connection] |
| 20:53:01 | | Exorcism (exorcism) joins |
| 21:12:38 | | Maturion quits [Remote host closed the connection] |
| 21:14:50 | <@arkiver> | pokechu22: AB still gets banned on 2 second delay? |
| 21:15:56 | <pokechu22> | It's kinda random, but the job does get banned with the random delay from 1s to 2s sometimes. I *think* it happens when it retrieves a bunch of pages that don't exist in a row but I'm not 100% sure. Other jobs were able to run at a constant 1s delay without getting banned. (Though latency might factor into this as well which I'm not sure of) |
| 21:16:42 | <@arkiver> | could AB not be respecting the 2 second pause when a page throws some error? |
| 21:17:24 | <@JAA> | AB should always respect the delay setting. |
| 21:17:38 | <pokechu22> | I'm pretty sure it does, but there might be some odd behaviors with the way the redirects work or something. It definitely waits between each step of the redirect |
| 21:17:53 | <imer> | pokechu22: i can burn through some ips I can't use for archival anyways if we want to test things? |
| 21:18:05 | <@JAA> | But latency could play a role in this, too, if those 404s are returned much faster than 200s, for example. |
| 21:18:30 | <imer> | do we know how quick/long the bans are? |
| 21:18:54 | <pokechu22> | 24 hours |
| 21:22:32 | <imer> | and a ban was a connection timeout? |
| 21:23:17 | <imer> | for test urls, should I just use the ones from above https://coturnix.monsite-orange.fr/ (working) and https://coturnix.monsite-orange.fr/son/historique.mid 404? or is "non existant pages" something else? |
| 21:52:02 | <pokechu22> | Yeah, connection timeouts. I think I also got connections refused on a local connection but I'm not 100% sure |
| 21:52:16 | <pokechu22> | The nonexistent pages thing is a guess that I'm not 100% sure of |
| 21:56:11 | <pokechu22> | The refused connections might have been with a delay shorter than 1s actually |
| 21:56:37 | <pokechu22> | I think that was 500ms or 500ms-1000ms |
| 22:27:28 | | qwertyasdfuiopghjkl quits [Remote host closed the connection] |
| 22:29:45 | <imer> | ban only seems to apply to new connections, doesnt kill the old ones |
| 22:32:49 | <imer> | you can burst a lot from the looks of it, so this might take a while |
| 22:41:37 | <imer> | Worked 0000429, 1.0 req/s, 433s elapsed and then it timed out |
| 22:49:04 | <imer> | Worked 0000480, 1.5 req/s, 327s elapsed this one started timing out at ~430 requests again but recovered temporarily and got completely blocked at ~480 |
| 22:52:20 | <imer> | i'm guessing there's both a min. time and min. req number component to this, some of the earlier tests I went way faster (> 1k requests), but i'll retest that |
| 22:52:43 | <imer> | these are all on 404 urls for now |
| 22:53:03 | <pokechu22> | Are you measuring delay between requests, or between starting each request (disallowing concurrent requests)? |
| 22:53:21 | <pokechu22> | and are you following the redirects for 404 URLs? |
| 22:53:43 | <pokechu22> | for a DPoS job we'd probably not want to follow them (but AB will always follow them and there's no option to disable it) |
| 22:54:13 | <imer> | I've got (somewhat bad) code that adds an item to queue and then sleeps to match the rate, the x req/s is calculated in log output is calculated, so thats the real rate |
| 22:54:17 | <imer> | not following redirects |
| 22:54:50 | <imer> | i've got a number of workers, so if requests take longer than the rate it'll do multiple at once |
| 22:55:13 | <imer> | Worked 0000099, 0.5 req/s, 201s elapsed -> timeout |
| 22:55:28 | <imer> | gonna try the same on a non-404 now |
| 22:55:41 | | Peroniko quits [Ping timeout: 252 seconds] |
| 22:55:50 | <imer> | actually, going to make a proper build and run multiple at once |
| 22:55:58 | <imer> | (on different ips of course) |
| 22:56:36 | <pokechu22> | One other thing that'd be useful is determining rate-limits for stuff on woopic |
| 22:56:49 | <pokechu22> | I *think* they tie into the same rate limit but I'd need to double-check |
| 23:00:14 | <pokechu22> | hmm, actually, I don't have data either way on that one |
| 23:00:57 | <pokechu22> | it never ended up requesting anything on woopic while it was banned |
| 23:01:28 | <pokechu22> | but I guess *you* could easily test it by trying to load e.g. https://monsite.woopic.com/383/f/300x/p/bienetre-artisanat02/img/de96401a61f9936cc59ebf45fd037fd5.jpg on a banned IP |
| 23:02:19 | | Peroniko joins |
| 23:02:31 | | Peroniko is now authenticated as Peroniko |
| 23:13:02 | <imer> | ban aplies to both |
| 23:36:18 | <imer> | I think 404's have a higher penalty for whatever they're using, non-404 ones are definitely going longer without ban so far |
| 23:41:02 | <imer> | yeah, got a ban on 404-only at 4s/req |
| 23:49:29 | <imer> | i did get one ban on 200 at 1req/s (after 800req), so probably have to go slightly slower, just not sure what the strategy for 404s should be, quite slow probably to make sure there's no bans |
| 23:49:47 | <imer> | and by 404 I mean the redirect |
| 23:49:59 | <pokechu22> | Does the penalty behavior change if you follow the redirects? |
| 23:50:19 | <imer> | that'd be quite advanced |
| 23:50:38 | <imer> | will give these a bit longer and then i'll try |
| 23:51:13 | <pokechu22> | Well, it'd be as simple as having a lighter penalty for the redirects themselves (e.g. a 1s penalty) and then things would average out to a lower penalty overall maybe |