| 01:22:07 | | le0n quits [Client Quit] |
| 01:22:07 | | Jake quits [Client Quit] |
| 01:22:43 | | le0n (le0n) joins |
| 01:22:48 | | Jake (Jake) joins |
| 06:04:44 | | Atom-- joins |
| 06:06:36 | | Atom quits [Ping timeout: 252 seconds] |
| 06:21:02 | | Atom joins |
| 06:23:18 | | michaelblob quits [Read error: Connection reset by peer] |
| 06:24:12 | | Atom-- quits [Ping timeout: 252 seconds] |
| 06:26:40 | | michaelblob (michaelblob) joins |
| 07:06:35 | | Shjosan quits [Ping timeout: 265 seconds] |
| 07:30:38 | | systwi__ (systwi) joins |
| 07:31:18 | | systwi quits [Ping timeout: 252 seconds] |
| 09:55:47 | | qwertyasdfuiopghjkl joins |
| 11:07:59 | <Somebody2> | OK, looking at dy-si |
| 11:08:13 | <Somebody2> | You missed a "^" at the beginning of the regex |
| 11:11:25 | | systwi__ is now known as systwi |
| 11:12:25 | <h2ibot> | [AT] URLTeam tracker https://tracker.archiveteam.org:1338/api/health is up (200,Success) |
| 11:12:45 | <Somebody2> | Added HTTP 400 to cstu-io banned codes |
| 11:12:49 | <Somebody2> | and cleared the errors |
| 11:13:37 | <Somebody2> | OKY dy-si is generating data |
| 11:16:43 | <Somebody2> | and filled in ed-gr per your notes, Ryz and turned it on |
| 11:16:54 | <Somebody2> | Let me know if/when you have more questions! |
| 11:18:14 | <Somebody2> | Anyone remember why we turned off the queue for isgd_6 ? |
| 11:21:29 | <datechnoman> | We are zooming along now! |
| 11:22:11 | <datechnoman> | Didnt even know we were running ipv6 shortners? |
| 12:01:48 | <Somebody2> | Where's the ipv6 shortener results you saw? |
| 12:10:53 | | eroc1990 quits [Client Quit] |
| 12:12:27 | | eroc1990 (eroc1990) joins |
| 12:20:57 | <datechnoman> | Sorry no results. Just saw you said isgd_6 and assumed that is ipv6 |
| 13:26:22 | | mrHedgehog0 quits [Remote host closed the connection] |
| 13:27:10 | | mrHedgehog0 (mrHedgehog0) joins |
| 13:28:43 | <Somebody2> | Nope -- isgd_6 is the shortener is.gd |
| 13:28:57 | <Somebody2> | That we'd been running for a while, and I think broke |
| 15:12:37 | | Shjosan (Shjosan) joins |
| 15:13:31 | <Ryz> | Uhhh, I'm stopping the dy-si project Somebody2 |
| 15:13:35 | <Ryz> | Something is still wrong |
| 15:14:08 | <Ryz> | It's been grabbing invalid or unavailable IDs that just redirect to http://www.dynamicsignal.com/ |
| 15:14:34 | <Ryz> | I don't think the "Location header reject regular expression" thing matches it |
| 15:16:32 | <Ryz> | Literally the only thing I can think of is that the reject only blocks HTTPS and not HTTP |
| 15:17:07 | <Ryz> | ...Unfortunately it looks like something knowledge-wise transferred from ArchiveBot when doing regexes is incorrectly applied here |
| 15:33:08 | <Ryz> | Gonna run it again, hopefully what I set up is correct |
| 15:37:05 | <Ryz> | Hmm, wondering what's wrong with this code being: ^((http|https)?://dynamicsignal\.com|(http|https)?://www.dynamicsignal\.com)$ |
| 15:37:21 | <Ryz> | The other thing I can do is just simplifying to www.dynamicsignal\.com$ |
| 15:41:25 | <Ryz> | Oops, forgot the '/', so it should be www.dynamicsignal\.com/$ |
| 15:51:52 | <Ryz> | Regarding ed-gr, gonna have to change ^https?://rebrandly\.com/404$ to rebrandly\.com/404$ out of caution; since it's still finding valid results so far |
| 16:09:00 | <@JAA> | datechnoman: The '_6' means six characters. |
| 18:28:34 | | Icyelut|3 quits [Ping timeout: 265 seconds] |
| 19:20:43 | | Craigle quits [Quit: The Lounge - https://thelounge.chat] |
| 19:22:10 | | Craigle (Craigle) joins |
| 22:35:12 | <datechnoman> | Haha well i feel dumb now :P |
| 22:41:57 | <Ryz> | So I'm gonna probably resume running one of my projects that got halted because of rate limiting or banning; maybe 1 second delay instead of the default |
| 22:47:35 | <Ryz> | On project cstu-io, checking from default 0.5 to 1 on time between requests (seconds) |
| 22:49:24 | <Ryz> | Time to resume running it and hope the rate limits or bans are lifted or something |
| 23:00:15 | <Ryz> | "Number of attempts exceeded for 932600", hmm, stopping s: |
| 23:00:24 | <@phuzion> | Howdy |
| 23:00:30 | <@phuzion> | How's it going Ryz? |
| 23:00:35 | <Ryz> | Heya phuzion~ |
| 23:00:51 | <Ryz> | Pretty alright, some of the projects got halted because it suffered rate limits and bans~ |
| 23:01:04 | <Ryz> | Some of them I tried starting out just immediately ended with bans |
| 23:01:17 | <Ryz> | Others I was just waiting after setting up and then Somebody2 helped out |
| 23:04:22 | <Ryz> | As you can see the amount of activity above while waiting <#>; |
| 23:11:19 | <@phuzion> | Added git.io |
| 23:14:17 | <Ryz> | Welp, if GitHub's rate limiting is any indication, I foresee a flood of 429s phuzion ><; |
| 23:15:52 | <@phuzion> | I'm seeing a pretty consistent results so far |
| 23:16:06 | <@phuzion> | If we start seeing bans, I'll drop the rate limiting. |
| 23:16:29 | <Ryz> | It's more like 429s; and uhh, tis based on what happened at ArchiveBot |
| 23:18:47 | <fuzzy8021> | [2022-04-30_14:20:22] <@J_AA> arkiver: I ran a quick test with qwarc and 10 connections. Didn't see any rate limit issues, just poor response times of 0.5 to 1 s (average). |
| 23:18:57 | <fuzzy8021> | regarding git.io |
| 23:20:13 | <@JAA> | Cc arkiver ^ |
| 23:29:56 | | @phuzion quits [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.] |
| 23:30:02 | | phuzion (phuzion) joins |
| 23:30:02 | | @ChanServ sets mode: +o phuzion |
| 23:37:36 | <Ryz> | Regarding l8r-it and postly-app, phuzion, what to do when running 'em, it just stops because of 500s on the former, and 405s on the latter? s: |
| 23:41:59 | <@arkiver> | we're not creating any WARCs here yet right |
| 23:42:16 | <@arkiver> | so for the git.io URLs that do exist, we'll cover them again into WARCs later |
| 23:43:21 | <@JAA> | Correct, no WARC here, just mapping text files. |
| 23:46:43 | <Ryz> | Yeah, that's one of the reasons why there hasn't been much activity here overall... |
| 23:47:10 | <Ryz> | And/or not processing it thru #// which may have super blow up the project with too many links |