| 00:06:29 | | Matthww quits [Client Quit] |
| 00:06:50 | | Matthww joins |
| 00:20:23 | | sliccricc quits [Remote host closed the connection] |
| 00:29:35 | | Matthww quits [Client Quit] |
| 00:29:55 | | Matthww joins |
| 00:31:29 | <icculus> | Are y'all still pulling in wiki.libsdl.org, or can I turn on redirects to GitHub? |
| 00:32:54 | | britmob quits [Read error: Connection reset by peer] |
| 00:39:00 | | britmob joins |
| 00:46:41 | | ThreeHeadedMonkey quits [Remote host closed the connection] |
| 00:47:10 | <@JAA> | icculus: That's still running. http://dashboard.at.ninjawedding.org/ if you want to keep an eye on it. |
| 00:48:59 | <icculus> | JAA: That can run more quickly, if it's still being throttled, if you like. |
| 00:49:54 | | ThreeHeadedMonkey (ThreeHeadedMonkey) joins |
| 00:53:16 | | ThreeHea1 (ThreeHeadedMonkey) joins |
| 00:54:52 | | ThreeHeadedMonkey quits [Ping timeout: 265 seconds] |
| 00:55:15 | | ThreeHea1 is now known as ThreeHeadedMonkey |
| 00:57:04 | <icculus> | Also, you're welcome to archive discourse.libsdl.org, but it isn't going away. |
| 00:57:58 | <@JAA> | icculus: I've increased the request rate for the wiki job, but I just noticed that I think it will not be able to archive the past revisions of all pages. Looking into options to cover those. |
| 00:58:26 | <@JAA> | Well, and now the wiki is throwing 503. |
| 00:58:40 | <icculus> | Someone is pounding bugzilla to download all the attachment data |
| 00:58:52 | <icculus> | I'm blocking that person, which hopefully will help. |
| 00:58:56 | <@JAA> | Ah |
| 00:59:21 | <@JAA> | I wonder whether we fully covered that. I know that I had to do some special stuff in the past for getting attachments from Trac. |
| 01:05:07 | <@JAA> | I'll run another job to be sure. |
| 01:05:37 | <@JAA> | Just the bug and attachment pages, iterating over the IDs. |
| 01:10:15 | <@hook54321> | JAA: it's been doing that most of today |
| 01:10:37 | <@hook54321> | not doing it on my connection thoughg |
| 01:12:41 | <@JAA> | hook54321: I checked the logs and found holes. |
| 01:13:00 | <@JAA> | It definitely didn't retrieve all attachments. Not sure about bugs. |
| 01:13:23 | <@hook54321> | the wiki i mean, sorry |
| 01:13:30 | <@JAA> | Ah |
| 01:28:03 | | BlueMaxima joins |
| 01:29:51 | <@JAA> | icculus: FYI, I have launched that additional job for Bugzilla. |
| 01:36:28 | | sliccricc (sliccricc) joins |
| 01:53:51 | | Wayward- (wayward) joins |
| 01:54:37 | | Wayward quits [Ping timeout: 252 seconds] |
| 02:24:45 | | aleph quits [Client Quit] |
| 02:27:26 | | aleph joins |
| 02:29:13 | <SketchTheCow> | Was icculus murdered |
| 02:29:36 | <icculus> | Still alive, just putting the kid to bed. :) |
| 02:35:47 | <icculus> | JAA: here's _all_ the attachment data for Bugzilla: https://libsdl.org/bugzilla.libsdl.org-attachments.tar.bz2 |
| 02:35:52 | <icculus> | 209 megabytes |
| 02:36:14 | <icculus> | Takes five seconds to pull with an SQL query, hours to coerce Bugzilla to hand it to you. |
| 02:37:27 | <icculus> | I have no idea if this is helpful if it doesn't come in by pulling it directly from the website, but if it saves you time or gives you certainty that you got everything, feel free to use that instead. |
| 02:39:36 | <@JAA> | icculus: Thanks, that's certainly useful! Part of the effort is also to preserve the URLs though, e.g. when they were linked elsewhere. |
| 02:45:33 | | benjins quits [Ping timeout: 265 seconds] |
| 03:02:05 | | benjins joins |
| 03:04:00 | | benjins is now authenticated as benjins |
| 03:11:17 | <mgrandi> | Also, did you see what we decided earlier? Probably the entire database (minus sensitive user data) + sqlite mapping of github/bugzilla IDs that you are using, and plus those attachments , we should be good to throw those in a IA item |
| 03:42:10 | | Arcorann quits [Read error: Connection reset by peer] |
| 03:50:48 | | balrog quits [Ping timeout: 265 seconds] |
| 03:56:39 | | balrog joins |
| 03:56:40 | | balrog is now authenticated as balrog |
| 04:08:33 | | DogsRNice quits [Read error: Connection reset by peer] |
| 04:29:07 | <icculus> | mgrandi: I have to figure out what data to filter out, which probably won't happen tonight, but I'll get you an SQL dump after that. |
| 04:29:58 | <icculus> | The bugzilla -> github bug id mapping will happen once all the bugs import. We're mostly done with that, got to do all the satellite libraries still, but SDL itself (which is ~5000 bugs) are migrated already. |
| 04:31:03 | <icculus> | The attachment data will _also_ be in the SQL dump, but the tarball of all the attachments (with names cleaned up...early Bugzilla clamped the file names, losing extensions) is posted above. |
| 04:31:33 | <icculus> | also: y'all are hardcore, I love it. :) |
| 04:33:08 | <mgrandi> | Sure, we'll take it when you are ready! And thanks, lots of people contribute to make things like the ddos-as-a-service (archive bot) happen! |
| 04:45:24 | | icculus quits [Remote host closed the connection] |
| 04:45:40 | | icculus joins |
| 04:56:24 | | qw3rty_ joins |
| 04:59:55 | | qw3rty__ quits [Ping timeout: 265 seconds] |
| 05:10:57 | <@JAA> | Fusl, jodizzle: Turns out it isn't just the inQuake forums that are shutting down: https://starcraft2.ingame.de/forum/announcement.php?f=700 https://warcraft.ingame.de/forum/threads/216062-Das-Forum-ist-wahrscheinlich-bald-Geschichte?p=5027446&viewfull=1#post5027446 |
| 05:11:01 | <@JAA> | Not sure if there are more. |
| 05:12:55 | <jodizzle> | Oh. I had done a quick check of the other forums a few days ago, looking for the same header notice that the quake forums had, so I guess I missed these announcements |
| 05:13:10 | <@JAA> | Yeah, no header notice on those. |
| 05:13:22 | <jodizzle> | So should we just get all the forums? |
| 05:13:30 | <@JAA> | I just happened to read a reference to them on one of the 'when does inQuake finally shut down?' threads. |
| 05:13:56 | <@JAA> | https://transfer.notkiska.pw/inline/105SYc/aatt.png |
| 05:17:23 | | etnguyen03 quits [Client Quit] |
| 05:17:31 | <@JAA> | inWoW (inwow.de, wow.ingame.de) is already down. Last snapshots in the WBM are from December. |
| 05:17:48 | <@JAA> | So yeah, let's assume that all forums on ingame.de are going to disappear. |
| 05:18:10 | <@JAA> | inWoW had 2.8 million posts... :-| |
| 05:18:19 | <@JAA> | https://web.archive.org/web/20201204211256/https://wow.ingame.de/forum/forum.php?s=eb406a80dcb8a69e94ec85592f0f4c69 |
| 05:18:35 | <jodizzle> | :( |
| 05:19:41 | <jodizzle> | Okay, so should I start putting them in AB then? Or are you planning to qwarc them? |
| 05:19:54 | <@JAA> | https://cs.ingame.de/forum/ https://diablo3.ingame.de/forum/ |
| 05:20:00 | <@JAA> | No announcements on these as far as I can see. |
| 05:24:28 | <@JAA> | https://guildwars.ingame.de/forum/ vanished some years ago already. Hardly anything in the WBM. |
| 05:25:47 | <@JAA> | http://callofduty.ingame.de/ seems to have gone down sometime last year. |
| 05:30:16 | <flashfire42> | https://torchlight.ingame.de/forum us dead |
| 05:31:05 | <@JAA> | More tombstones: https://bns.ingame.de/forum/ (ca 2016), https://torchlight.ingame.de/forum (2018), https://lol.ingame.de/ aka inLoL.de (2019?), https://hearthstone.ingame.de/forum aka inHearthstone.de (after 2017?) |
| 05:31:38 | <flashfire42> | Should we run all the main sites of those anyway or? |
| 05:32:41 | <jodizzle> | I think we should probably run everything on ingame.de at this point |
| 05:32:52 | <@JAA> | Yeah |
| 05:33:20 | <flashfire42> | I will get started |
| 05:33:29 | <@JAA> | Take care with the session ID crap. |
| 05:33:35 | <jodizzle> | Yeah |
| 05:33:50 | <jodizzle> | launch with '?archiveteam' at the end, use the forums igset, and use `--no-offsite-links` |
| 05:34:09 | <flashfire42> | oh shit |
| 05:34:14 | <jodizzle> | For forums, I mean. For the non-forum parts, it probably doesn't matter |
| 05:34:29 | <flashfire42> | I will abort that first one then and just leave https://dnsdumpster.com/static/xls/ingame.de-202102110528.xlsx here |
| 05:34:54 | <jodizzle> | Okay, let's continue this in #archivebot |
| 05:34:56 | <flashfire42> | I think I will give dealing with session IDs a miss lol |
| 05:35:27 | <@JAA> | I'll grab the forum threads of starcraft2, warcraft, cs, diablo3 with qwarc tomorrow. If you find any other live forums, please let me know. |
| 05:36:19 | <@JAA> | http://dota2.ingame.de/ died ca 2019 it seems. |
| 05:37:32 | <@JAA> | (aka inDota2.de) |
| 05:37:52 | <@JAA> | Maybe earlier actually, last good snapshot in the WBM is from late 2017. |
| 05:39:23 | <@JAA> | Based on reading those announcements and discussions on starcraft2 and warcraft, it looks like even the admins don't really know much about this shutdown... |
| 05:44:23 | <@JAA> | Accounts are apparently shared across all forums, which also explains why so few members had posts on inQuake. |
| 05:44:58 | <@JAA> | Also further reason to grab all of it since they're likely just going to take the entire thing down. |
| 05:45:00 | <jodizzle> | Ahhh |
| 05:45:05 | <jodizzle> | Interesting |
| 05:45:15 | <@JAA> | End of month (28 February) is given as a date on starcraft2 and warcraft. |
| 05:46:23 | <@JAA> | Here's a list of sites, by the way: https://starcraft2.ingame.de/forum/threads/243445-Aus-f%C3%BCr-pq-de?p=6798550&viewfull=1#post6798550 |
| 05:47:10 | <@JAA> | According to that, those four forums above are the only ones (apart from inQuake) that still exist. |
| 05:49:31 | <@JAA> | Oh right, attachments aren't publicly viewable. Meh. |
| 06:24:06 | <icculus> | mgrandi: https://www.dropbox.com/s/ncwfgfyf3d2gaw1/libsdlbugmap.sql?dl=1 |
| 06:24:55 | <icculus> | https://www.dropbox.com/s/1l1n007m6qdg4j5/libsdlbugmap.sqlite3?dl=1 |
| 06:25:22 | <icculus> | it's just bugzilla.libsdl.org bug id, the github repo name, the github issue id |
| 06:27:08 | <icculus> | So 5454|SDL_ttf|121 means https://bugzilla.libsdl.org/show_bug.cgi?id=5454 maps to https://github.com/libsdl-org/SDL_ttf/issues/121 |
| 06:36:47 | <icculus> | I'm going to sleep for a few hours. When I wake up, I really need to turn on the redirects, so hopefully y'all will be done archiving by then. Feel free to pull bugzilla and wiki as fast as the server will let you. If you 503, just try again without delay. The server is basically useless to me until the redirects are in place now. |
| 06:53:49 | | godane1 quits [Ping timeout: 252 seconds] |
| 06:53:51 | | godane joins |
| 06:53:51 | | godane is now authenticated as godane |
| 07:59:39 | | daalekz (daalekz) joins |
| 08:00:47 | | daalekz quits [Client Quit] |
| 08:09:03 | | HP_Archivist quits [Read error: Connection reset by peer] |
| 08:28:55 | | Arcorann (Arcorann) joins |
| 08:34:53 | | hooway joins |
| 08:56:01 | | nepeat_ quits [Quit: ZNC 1.8.2 - https://znc.in] |
| 08:56:41 | | nepeat joins |
| 09:12:03 | | Craigle quits [Quit: Ping timeout (120 seconds)] |
| 09:12:41 | | Craigle (Craigle) joins |
| 09:39:17 | | Jonimus quits [Ping timeout: 265 seconds] |
| 09:52:12 | | Jonimus joins |
| 09:53:59 | | ragu joins |
| 09:54:16 | | ragu is now authenticated as ragu |
| 10:34:21 | | BlueMaxima quits [Read error: Connection reset by peer] |
| 12:40:04 | | ragu_ joins |
| 12:43:55 | | ragu quits [Ping timeout: 265 seconds] |
| 12:44:58 | | LeGoupil joins |
| 13:25:20 | <mgrandi> | JAA ^ |
| 13:25:53 | <mgrandi> | Also see ping in archivebot , I'm just pinging here cause it's spammy in there |
| 13:30:14 | <OrIdow6> | Do you know if that applies to Mercurial as well mgrandi? |
| 13:31:11 | <mgrandi> | The HG repo probably finished super fast, and is staying up for a while I believe |
| 13:31:37 | <mgrandi> | He has a fancy script that redirects if it's a browser but serves the HG response if it's a hg client |
| 13:32:18 | <OrIdow6> | OK; it's still desirable to make a copy of the HG repos with the mercurial project but that's waiting on approval |
| 13:39:10 | | lennier1 quits [Ping timeout: 252 seconds] |
| 13:40:11 | | lennier1 (lennier1) joins |
| 13:41:43 | <mgrandi> | I think that should be around for a while, if I read his messages right |
| 13:43:18 | <icculus> | Mercurial will be around (in read-only mode) for awhile. |
| 13:43:25 | <icculus> | But it _will_ go away eventually. |
| 13:43:57 | <icculus> | URL redirection will be there permanently. |
| 13:44:42 | <icculus> | And likely we'll make a final commit that deletes all the files and replaces it with a readme that says "you really should be over at the new location now" |
| 13:45:27 | <icculus> | Also, we have a git repo that's a final snapshot of the Mercurial repo here: https://github.com/libsdl-org/SDL-historical-archive |
| 13:45:43 | <OrIdow6> | As long as it's there for a few days, it should be enough |
| 13:45:50 | <icculus> | it will be. |
| 13:46:08 | | Zerote quits [Quit: Leaving] |
| 13:54:27 | | Zerote joins |
| 13:59:48 | | Atom-- quits [Ping timeout: 265 seconds] |
| 14:00:39 | | Atom joins |
| 14:14:26 | | britmob quits [Read error: Connection reset by peer] |
| 14:17:39 | | britmob joins |
| 14:56:50 | | Arcorann quits [Ping timeout: 265 seconds] |
| 15:09:11 | | LeGoupil quits [Client Quit] |
| 15:09:22 | | HP_Archivist (HP_Archivist) joins |
| 15:19:59 | <@hook54321> | icculus: could the wiki be ratelimiting us? |
| 15:20:19 | <icculus> | hook54321: It _shouldn't_ be, but I'm looking into that now |
| 15:21:09 | <@hook54321> | it seems to be returning 503s only to the pipelines |
| 15:22:41 | <icculus> | try now |
| 15:22:56 | <icculus> | surge_action_limits = None # disable surge protection |
| 15:22:58 | <icculus> | yolo |
| 15:23:13 | <@hook54321> | lmao |
| 15:23:42 | <@hook54321> | yeah, works fine now |
| 15:23:53 | <@hook54321> | let me know if it breaks |
| 15:24:33 | <icculus> | short of the machine literally catching fire, hit it as hard as you can |
| 15:24:44 | <icculus> | I really need y'all to finish so I can, too. :) |
| 15:36:13 | | ragu__ joins |
| 15:39:51 | | ragu_ quits [Ping timeout: 265 seconds] |
| 15:51:09 | <@hook54321> | icculus: where's the revision history pages? |
| 15:52:42 | <icculus> | hook54321: https://wiki.libsdl.org/FrontPage?action=info |
| 15:52:53 | <icculus> | ?action=info on any page should do it |
| 15:53:31 | <icculus> | Apparently it doesn't show that link unless you are logged in, but it works anyhow. |
| 15:53:52 | <icculus> | Lemme see if I can change that, in case it automates things |
| 16:02:26 | | britmob2 quits [Client Quit] |
| 16:11:45 | | no112 quits [Ping timeout: 265 seconds] |
| 16:17:29 | | no112 joins |
| 16:21:05 | <icculus> | ok, I hardcoded a "(Page info.)" link at the bottom of every page. |
| 16:21:12 | <icculus> | So crawls should find this now. |
| 16:21:37 | <icculus> | (that was for hook54321) |
| 16:24:02 | | britmob2 joins |
| 16:26:29 | <@hook54321> | thanks! |
| 16:40:29 | | britmob2 quits [Client Quit] |
| 16:45:00 | <purplebot> | File:Webs-logo-white-background.png uploaded by Arkiver (+0) just now -- https://www.archiveteam.org/?diff=46314&oldid=0 |
| 16:53:56 | <@EggplantN> | okie kiska gave your target a break |
| 16:54:04 | <@EggplantN> | added on mf/urls |
| 16:54:06 | <@kiska> | Huh? |
| 16:54:14 | <@EggplantN> | added atr3 on urls/mf |
| 16:54:23 | <@EggplantN> | turns out they're >1Gbit combined |
| 16:54:24 | <@EggplantN> | lol |
| 16:55:10 | <@kiska> | Also broken files... |
| 16:56:35 | <@EggplantN> | yeah thats fine |
| 16:56:38 | <@EggplantN> | atr3 can handle |
| 17:19:52 | | britmob2 joins |
| 17:51:02 | | britmob2 quits [Client Quit] |
| 18:04:31 | | britmob2 joins |
| 18:29:01 | | Barto quits [Ping timeout: 252 seconds] |
| 18:29:46 | | britmob2 quits [Client Quit] |
| 18:35:34 | | Barto (Barto) joins |
| 18:41:25 | | britmob2 joins |
| 19:03:49 | | britmob2 quits [Client Quit] |
| 19:04:40 | | sec^nd quits [Remote host closed the connection] |
| 19:05:10 | | sec^nd (second) joins |
| 19:14:48 | | DogsRNice (Webuser299) joins |
| 19:17:36 | | britmob2 joins |
| 19:20:22 | | etnguyen03 (etnguyen03) joins |
| 19:21:05 | | britmob2 quits [Client Quit] |
| 19:32:54 | | britmob2 joins |
| 19:51:50 | <@JAA> | icculus: Wonderful, thanks for both the dumps and the ?action=info fix! The Bugzilla job is almost done with the bug pages, then it'll start the attachments. Might take another day or so since it's running quite slowly. |
| 20:17:35 | <mgrandi> | Is the archive bot User agent hard coded? You could leave the old site up for us and then redirect anyone else? |
| 20:52:12 | | icculus_ joins |
| 20:54:47 | | icculus quits [Ping timeout: 244 seconds] |
| 21:01:31 | <@hook54321> | is there a good way to grab telegram channels yet? |
| 21:28:29 | | nepeat is now authenticated as nepeat |
| 22:09:01 | | benjinsmith joins |
| 22:10:52 | | benjins quits [Ping timeout: 265 seconds] |
| 22:11:18 | | benjinsmith is now known as benjins |
| 22:11:24 | | benjins is now authenticated as benjins |
| 22:40:29 | | space quits [Read error: Connection reset by peer] |
| 22:40:35 | | space (space) joins |
| 22:40:43 | | mgrandi quits [Read error: Connection reset by peer] |
| 22:40:52 | | mgrandi (mgrandi) joins |
| 22:58:51 | | Eighty quits [Remote host closed the connection] |
| 22:59:24 | | Sylirana (Sylirana) joins |
| 23:03:17 | <Sylirana> | Not really sure if this is the right channel, but do we have CodePlex (https://archive.codeplex.com/) or is there a project for it? It will *shut down July 2021*. |
| 23:05:15 | <@JAA> | Sylirana: We don't yet, but we're aware of it (see https://archiveteam.org/index.php?title=Deathwatch for notable things about to shut down). |
| 23:07:47 | | britm0b joins |
| 23:08:07 | <@JAA> | Speaking of which, looks like we missed Unity Connect. :-| |
| 23:09:54 | <OrIdow6> | I looked at Unity Connect, and I don't think there was much there |
| 23:10:37 | <OrIdow6> | IIRC the important stuff was moved elsewhere |
| 23:11:07 | <Sylirana> | Thanks, so far all I could find was https://github.com/Ciastex/CodePlexScraper . I honestly don't know much about the strcture of CodePlex myself, but am interested in helping to save what's possible before it goes down. |
| 23:11:46 | | britmob quits [Ping timeout: 265 seconds] |
| 23:13:19 | <@JAA> | OrIdow6: Groups were only migrated to Unity Learn partially. |
| 23:15:12 | <@JAA> | Actually, not sure where those are at all on Learn. But that's what the announcement said: 'Select Groups will move to Unity Learn soon.' |
| 23:15:34 | | icculus_ quits [Ping timeout: 252 seconds] |
| 23:15:54 | <Sylirana> | The sitemap looks quite promising. Not sure if that's all though. |
| 23:25:19 | | benjinsmith joins |
| 23:25:32 | <mgrandi> | @hook54321: uh, we can apparently look at the channel preview but it's kinda jank |
| 23:26:28 | <mgrandi> | If you join the channel, you can export it using telegram desktop to a HTML file |
| 23:26:47 | <mgrandi> | (or json too actually) |
| 23:26:54 | | xdd quits [Remote host closed the connection] |
| 23:27:12 | | benjins quits [Ping timeout: 244 seconds] |
| 23:27:35 | <mgrandi> | I am also working on a proper telegram archiver myself and need time to finish it |
| 23:29:20 | <mgrandi> | There is no super official way of getting the telegram data, but the tdlib library that telegram has, uses json for send / recieve so I figure I would just maybe save those to a war? |
| 23:29:25 | <mgrandi> | WARC* |
| 23:31:55 | <Sylirana> | Looks like we're lucky: The project archives MS provides, contain wikis and issues as well instead of just the code of the project. |
| 23:33:06 | <mgrandi> | That's good! |
| 23:35:07 | <Sylirana> | Only downside is that the links and embedded image sources need to be rewritten as they refer to the to-be-shutdown site. But from what I can see, all the files are in the archive (attachments such as images have just a hash as name though and no ending, so need to be renamed). But at leastwe can get all the files this way. From the sitemap, it seems there are something over 100k project on the site. |
| 23:35:16 | <Sylirana> | *least we |
| 23:37:22 | | hooway quits [Client Quit] |
| 23:51:06 | | Ryz quits [Quit: Ping timeout (120 seconds)] |
| 23:53:53 | | Ryz (Ryz) joins |