| 00:07:46 | | burnmjd quits [Client Quit] |
| 00:22:51 | <OrIdow6> | JetBrains TeamCity (CI thing) possibly implicated in the SolarWinds affair - https://www.nytimes.com/2021/01/06/us/politics/russia-cyber-hack.html |
| 00:24:52 | <OrIdow6> | atphoenix: I may resume that in the future, but yesterday I spent about as much time with Twitter, Youtube, and the people who occupy them (and post about politics) as I normally do in two months, and I don't think I can take much more in a short time |
| 00:28:38 | <OrIdow6> | I plan to look at So-Net U+ over the weekend, I think there were a few tens of thousands of users in WBM CDX |
| 00:41:26 | <atphoenix> | OrIdow6, ok. It's easy to get overwhelmed. |
| 01:20:51 | | purplebot quits [Read error: Connection reset by peer] |
| 01:21:20 | | purplebot joins |
| 01:26:56 | <OrIdow6> | Thank you |
| 02:00:01 | | Dallas quits [Client Quit] |
| 02:03:38 | | Dallas (Dallas) joins |
| 02:07:26 | | chip quits [Quit: leaving] |
| 02:35:20 | <OrIdow6> | Betsy DeVos resigns - https://www.nytimes.com/2021/01/07/us/politics/betsy-devos-resigns.html |
| 02:59:07 | <OrIdow6> | Also might be useful: https://www.nytimes.com/article/trump-resignations.html |
| 02:59:14 | <OrIdow6> | List of people who have resigned |
| 03:16:01 | <mgrandi> | Did we get their twitter accounts? https://twitter.com/SecElaineChao and https://twitter.com/BetsyDeVosED |
| 03:17:32 | <@JAA> | Yes |
| 03:40:44 | | Doran quits [Ping timeout: 240 seconds] |
| 03:58:27 | | sec^nd quits [Ping timeout: 240 seconds] |
| 04:05:38 | | Doran joins |
| 04:13:34 | | sec^nd (second) joins |
| 04:27:27 | | lukash7 quits [Quit: The Lounge - https://thelounge.chat] |
| 04:28:05 | | qw3rty_ joins |
| 04:31:48 | | lukash7 joins |
| 04:32:09 | | qw3rty__ quits [Ping timeout: 265 seconds] |
| 04:58:50 | | etnguyen03 quits [Client Quit] |
| 05:04:36 | | godane (godane) joins |
| 05:24:56 | | DogsRNice quits [Read error: Connection reset by peer] |
| 06:29:54 | <@kiska> | rsync.hel1.... is now going down for a system rebuild, est time for it to come back onto #//, #gitgud, #nearlylostmygoogles, #mediaonfire should be < 1 hour depending on how many times I need to reinstall due to ZFS fuck ups |
| 06:31:03 | | tzt quits [Ping timeout: 265 seconds] |
| 06:36:17 | <@kiska> | Fuck 1... Doing an reinstall cause of me accidently activating swap |
| 07:02:40 | <@kiska> | Attempt 2, wrong rootfs config |
| 07:12:13 | | LeGoupil joins |
| 07:45:02 | | godane quits [Read error: Connection reset by peer] |
| 09:01:34 | | burnmjd joins |
| 09:42:44 | | Sanqui is now known as Sa |
| 09:43:14 | | Sa is now known as Sanqui |
| 11:16:53 | | Arcorann_ joins |
| 11:20:40 | | Arcorann quits [Ping timeout: 252 seconds] |
| 12:11:19 | | Jon quits [Ping timeout: 265 seconds] |
| 12:12:01 | | sec^nd quits [Remote host closed the connection] |
| 12:12:27 | | sec^nd (second) joins |
| 12:29:24 | | icedice joins |
| 12:33:03 | | tzt joins |
| 12:50:13 | <icedice> | Might be worth archiving tweets from the hashtags of Shueisha's most popular manga titles |
| 12:50:28 | <icedice> | https://twitter.com/ReadTokyoGhoul2/status/1347420776526442498 |
| 12:50:36 | <icedice> | https://myanimelist.net/manga/magazine/83/Shounen_Jump_Weekly |
| 12:55:12 | <icedice> | I'd say most titles below Hokuto no Ken with the exception of Burn the Witch are too irrelevant for Shueisha to care about if they're only going after their most popular ones |
| 13:04:03 | <icedice> | Seems to be because of new chapter leaks posted to Twitter, so this would probably mostly be regarding ongoing series |
| 13:13:44 | | LeGoupil quits [Ping timeout: 240 seconds] |
| 13:28:22 | | godane (godane) joins |
| 13:44:21 | | godane quits [Client Quit] |
| 14:01:34 | | britm0b joins |
| 14:05:23 | | Matthww quits [Client Quit] |
| 14:05:52 | | LeGoupil joins |
| 14:09:18 | | Matthww joins |
| 14:20:18 | | Dallas quits [Client Quit] |
| 14:20:30 | | Dallas (Dallas) joins |
| 14:26:33 | <@kiska> | fionera: Not me, I don't even know what you're rate limiting |
| 14:27:20 | <@kiska> | I've invited you to the channel I think you're referring to |
| 14:29:32 | <@arkiver> | fionera: can probably best ask JAA about archivebot stuff |
| 14:29:38 | <fionera> | kk |
| 14:30:03 | <fionera> | JAA: One of our Domains at work is currently getting a lot of requests every night between 1-3am UTC apparently |
| 14:30:07 | <fionera> | UA is "ArchiveTeam ArchiveBot/20210105.3af42e7" |
| 14:31:24 | <fionera> | www.landtag-bw.de/files/live/sites/LTBW/files/dokumente/WP15/Drucksachen/6000/15_6741_D.pdf |
| 14:33:44 | | Jon (jon) joins |
| 14:39:25 | | Jonboy345 joins |
| 14:45:01 | | Arcorann_ quits [Ping timeout: 265 seconds] |
| 15:16:07 | <mgrandi> | Apparently the trump admin has created an account here for videos in case they get kicked off youtube : https://odysee.com/@DonaldJTrump:9 |
| 15:47:02 | <benjins> | latest youtube-dl release supports odysee it seems |
| 16:24:53 | <mgrandi> | I think I read it's mirroring his youtube at the moment |
| 16:55:42 | | LeGoupil quits [Client Quit] |
| 17:56:58 | | etnguyen03 (etnguyen03) joins |
| 18:11:24 | | Jonboy345 quits [Ping timeout: 240 seconds] |
| 18:11:50 | | Jonboy345 joins |
| 18:15:02 | | DogsRNice (Webuser299) joins |
| 18:21:52 | <fionera> | JAA: Thanks for checking is it possible to make limits for a domain globally? |
| 18:24:06 | <@JAA> | fionera: Unfortunately no. Each job is completely independent, and there are no options for host-specific delays or similar either. |
| 18:24:25 | <fionera> | ah shoot then we will add a ratelimit in our scrubbing pipeline |
| 18:24:46 | <fionera> | because sadly it rings the on-call admin all the time because the site isnt that nice |
| 18:24:47 | <@JAA> | We can't even ignore those URLs and then run them slower in a separate job because they're all redirects, not links. |
| 18:24:59 | <fionera> | yeah no problem ^^ |
| 18:25:17 | <@JAA> | Please make it a 429 if possible, then we can reduce the rate when it retries the errors. |
| 18:25:25 | <fionera> | sure thing |
| 18:26:04 | | fionera is adding this as a needed feature to his crawling system |
| 18:26:59 | <@JAA> | :-) |
| 18:29:10 | <@JAA> | fionera: Oh, actually, the structure of kleineanfragen.de might allow us to ignore it. Checking. |
| 18:29:25 | <fionera> | that would be awesome tho reducing the speed would also be fine |
| 18:30:04 | <@JAA> | Yeah, we don't want to slow it down in general though because it's unclear when the site will go down. |
| 18:30:04 | <fionera> | the site runs on a very suicidal java framework. it always crashes and is pure garbage |
| 18:30:14 | <@JAA> | And can't slow down just these particular URLs. |
| 18:30:23 | <@JAA> | Ah, fun. |
| 18:30:35 | <fionera> | ah i c |
| 18:31:46 | <@JAA> | Ok, I added an ignore that should cover these. |
| 18:31:53 | <fionera> | thank you <3 |
| 18:31:58 | <@JAA> | Will then rerun them when the job finishes at a slower pace. |
| 18:32:12 | <@JAA> | Let me know if the requests don't stop. :-) |
| 18:33:15 | <fionera> | sure thing currently not on my work pc will check later |
| 18:33:32 | <fionera> | also I think a coworker already blocks them rn |
| 18:35:15 | <@JAA> | Last requests from that job to //www.landtag-bw.de/ were about 5 hours ago as far as I can see. Still 200 at the time. |
| 18:36:36 | <fionera> | ah ok |
| 18:36:55 | <fionera> | yeah we just thought its AT because the log was full of it last night |
| 18:38:58 | <@JAA> | Yeah, it happens in bursts due to how AB/wpull recurses on that site. It'd come across https://kleineanfragen.de/baden-wuerttemberg/16, then do each of those shortly after one another, and it gets worse with the pagination at the bottom as it later does the same thing for four pages at once. |
| 18:40:01 | <@JAA> | Ideally, we'd have URL prioritisation, which would partially solve this as we could deprioritise those download URLs and then reduce the request rate when it starts processing them, but that's still WIP. |
| 18:40:21 | <@JAA> | Also requires that someone actually sees it when it starts going through those. |
| 18:40:58 | <@JAA> | URL-dependent delays are also something I've had in mind for a while, but it wouldn't work properly here because there are parallel downloads and other URLs in the queue. |
| 18:52:42 | <@kiska> | arkiver: please don't start projects where I have provided a target, but haven't said in the channel if I have "resumed service" |
| 18:53:04 | <@kiska> | Its likely those folders... never made the migration :D |
| 19:01:15 | | sec^nd quits [Remote host closed the connection] |
| 19:01:34 | | sec^nd (second) joins |
| 19:02:54 | | sec^nd quits [Remote host closed the connection] |
| 19:03:18 | | sec^nd (second) joins |
| 19:09:21 | | Dj-Wawa quits [Quit: Dj-Wawa] |
| 19:09:37 | | Dj-Wawa joins |
| 19:09:37 | | Dj-Wawa is now authenticated as Dj-Wawa |
| 19:12:14 | | Dj-Wawa quits [Client Quit] |
| 19:12:47 | | Dj-Wawa joins |
| 19:12:47 | | Dj-Wawa is now authenticated as Dj-Wawa |
| 20:13:12 | <@Kaz> | kiska: problem wouldn't exist if you used docker, thanks |
| 20:53:42 | | sembiance (sembiance) joins |
| 21:07:29 | | Jonboy345 quits [Read error: Connection reset by peer] |
| 21:34:13 | <purplebot> | Webzdarma edited by Sanqui (+25, job 19) just now -- https://www.archiveteam.org/?diff=46093&oldid=46061 |
| 21:44:14 | <purplebot> | Parler created by Sanqui (+89, stub out the page), edited by Sanqui (+0, typo) just now -- https://www.archiveteam.org/?diff=46095&oldid=0 |
| 21:54:15 | | godane (godane) joins |
| 22:27:08 | | flirora (flirora) joins |
| 23:31:17 | <Craigle> | Twitter just banned Trump |
| 23:32:04 | <Craigle> | So I guess there will be an influx of people asking if we "saved" it... |
| 23:32:10 | <@JAA> | surprised_pikachu.png |
| 23:33:10 | <Aoede> | https://twitter.com/TwitterSafety/status/1347684877634838528 |
| 23:33:13 | <Aoede> | it's permanent |
| 23:33:44 | <Craigle> | Curious if he will try to use @Potus or another account to evade the ban |
| 23:39:10 | | Arcorann_ joins |
| 23:43:12 | <flashfire42> | Craigle that was my thought TBH |
| 23:43:18 | <flashfire42> | Retweet for my mutuals lmfao |
| 23:43:44 | <tech234a> | Or perhaps use http://twitter.com/DonaldTrump |
| 23:44:00 | <tech234a> | Idk if he officially has that account but it looks possible |
| 23:45:01 | <benjins> | *crab rave* |
| 23:48:30 | | burnmjd quits [Client Quit] |
| 23:49:19 | <mgrandi> | I'm pretty sure several people were saving every tweet he made, including in WBM |
| 23:49:40 | <mgrandi> | I dunno if WBM gets the videos |
| 23:51:06 | | flirora quits [Remote host closed the connection] |
| 23:54:22 | <mgrandi> | Like I saw tweets that were tweeted pretty much live and I tried to WBM them and they were already saves |