HomeSearchPrevious dayNext day

01:26:07lun4 (lun4) joins
02:06:10mary3 joins
02:08:29mary quits [Ping timeout: 265 seconds]
02:08:29mary3 is now known as mary
02:16:38DopefishJustin quits [Remote host closed the connection]
02:20:36<flashfire42>https://discourse.libsdl.org/t/sdl-moving-to-github/28700
02:20:53DopefishJustin joins
02:32:38hexa- quits [Quit: WeeChat 2.9]
02:35:20hexa- (hexa-) joins
02:58:16ThreeHeadedMonkey quits [Changing host]
02:58:16ThreeHeadedMonkey (ThreeHeadedMonkey) joins
02:58:33NullPointer quits [Remote host closed the connection]
03:09:49Arcorann (Arcorann) joins
03:27:01sliccricc (sliccricc) joins
04:04:14DLoader quits [Ping timeout: 244 seconds]
04:05:10DogsRNice quits [Read error: Connection reset by peer]
04:22:31themadpro quits [Read error: Connection reset by peer]
04:22:31aarchi quits [Write error: Connection reset by peer]
04:23:02themadpro (themadpro) joins
04:23:27aarchi (aarchi) joins
04:26:38katocala quits [Remote host closed the connection]
04:28:17Dallas quits [Read error: Connection reset by peer]
04:28:27Dallas (Dallas) joins
04:30:17godane2 joins
04:30:24Dragnog quits [Read error: Connection reset by peer]
04:30:28Dragnog joins
04:33:00godane1 quits [Ping timeout: 265 seconds]
04:33:58Barto quits [Ping timeout: 265 seconds]
04:41:15godane1 joins
04:43:30godane2 quits [Ping timeout: 244 seconds]
04:50:41katocala joins
04:57:08qw3rty__ joins
05:01:02qw3rty_ quits [Ping timeout: 265 seconds]
05:16:35etnguyen03 quits [Client Quit]
05:45:11DopefishJustin quits [Remote host closed the connection]
05:48:38DopefishJustin joins
05:49:18OrIdow6 quits [Remote host closed the connection]
05:49:36OrIdow6 (OrIdow6) joins
06:29:01<mgrandi>@flashfire42: https://twitter.com/icculus/status/1359379854639190016 lol
06:30:43<mgrandi>@hook54321 @JAA ^ since you two started the AB jobs
06:31:05<mgrandi>Also https://twitter.com/icculus/status/1359382148688596995
06:31:36Barto (Barto) joins
06:44:15<flashfire42>oops we already killed it lol
06:46:37<mgrandi>Hmm, dunno if there is a way to properly use the database but still create WBM entries for it
06:46:56<mgrandi>He mentioned that if it finished in 8 hours we can probably do it
06:48:04<flashfire42>Yeah but we already killed the mercurial I think
06:48:08<flashfire42>SketchTheCow
06:48:16<flashfire42>If you want to jump on this
06:55:05<mgrandi>Icculus is referring to the bugzilla
06:55:48<mgrandi>I checked https://bugzilla.libsdl.org/buglist.cgi?chfield=%5BBug%20creation%5D&chfieldfrom=24h and there are 5543 bugs total I think, we can probably get it done in 8 hours easily
06:56:03<mgrandi>But I think a job is already running
06:58:02<@hook54321>mgrandi: I've just been adding ignores
06:58:23<mgrandi>For the bugzilla?
06:58:50<@hook54321>yeah
06:58:58<mgrandi>I'm on my phone and can't search for the bugzilla job, but I think it should finish by the time he wakes up
06:59:12benjins quits [Read error: Connection reset by peer]
07:08:04<mgrandi>Yeah I can't find it, If someone could drop the job ID that would be neat :)
07:15:50LeGoupil joins
07:16:08<thuban>mgrandi: 8kkomzrqu0ufo8a9gjatcs6ke
07:31:59<@hook54321>flashfire42: in the future please try to add ignores for jobs you start. particularly the mercurial interface (which there isn't really much of a point to) and bugzilla
07:43:52daalekz (daalekz) joins
08:15:04hooway joins
08:19:10Jonboy3451 quits [Ping timeout: 252 seconds]
08:32:02Jonboy3451 joins
08:39:23daalekz quits [Client Quit]
08:46:20jtt123 quits [Ping timeout: 244 seconds]
08:46:21tzt quits [Ping timeout: 244 seconds]
08:47:17tzt joins
08:47:18jtt123 (jtt123) joins
09:00:18britmob2 quits [Client Quit]
09:17:16godane2 joins
09:19:40godane1 quits [Ping timeout: 252 seconds]
10:12:37systwi quits [Ping timeout: 244 seconds]
10:16:52BlueMaxima quits [Client Quit]
10:18:46systwi (systwi) joins
10:35:30systwi quits [Ping timeout: 265 seconds]
10:52:44systwi (systwi) joins
10:54:49nathan quits [Ping timeout: 252 seconds]
10:55:59nathan joins
10:58:42Arcorann quits [Ping timeout: 265 seconds]
10:58:43Arcorann_ joins
12:50:44benjins joins
12:56:22Arcorann_ quits [Ping timeout: 252 seconds]
13:07:24Arcorann (Arcorann) joins
13:28:44yanome quits [Quit: The Lounge - https://thelounge.chat]
13:28:50yanome (yano) joins
13:35:28jonboy3452 joins
13:38:46Jonboy3451 quits [Ping timeout: 244 seconds]
13:50:07britmob2 joins
13:59:28etnguyen03 (etnguyen03) joins
14:01:33nathan quits [Client Quit]
14:01:44nathan joins
14:38:31Jonboy345 joins
14:41:58jonboy3452 quits [Ping timeout: 252 seconds]
14:54:12Church quits [Ping timeout: 244 seconds]
15:08:07sliccricc quits [Remote host closed the connection]
15:08:27sliccricc (sliccricc) joins
15:15:21Arcorann quits [Ping timeout: 265 seconds]
15:16:28Jonboy345 quits [Read error: Connection reset by peer]
15:22:29Jonboy345 joins
15:52:28ThreeHeadedMonkey quits [Remote host closed the connection]
15:53:58ThreeHeadedMonkey (ThreeHeadedMonkey) joins
16:27:22ThreeHeadedMonkey quits [Ping timeout: 265 seconds]
16:35:39aleph joins
16:36:33atphoenix quits [Read error: Connection reset by peer]
16:37:11atphoenix (atphoenix) joins
16:37:19LeGoupil quits [Remote host closed the connection]
16:37:30LeGoupil joins
16:47:14ThreeHeadedMonkey (ThreeHeadedMonkey) joins
17:16:56ThreeHea1 (ThreeHeadedMonkey) joins
17:19:49ThreeHeadedMonkey quits [Ping timeout: 252 seconds]
17:23:24ThreeHea1 is now known as ThreeHeadedMonkey
17:32:58aleph quits [Client Quit]
17:33:10aleph joins
17:39:09aleph quits [Client Quit]
17:39:21aleph joins
17:40:30aleph quits [Client Quit]
17:40:48aleph joins
17:49:18<yawkat>jcenter update: a week into concurrent wpull, not banned yet. i think this may work
17:49:53<yawkat>if this ever finishes, where can i upload it? it'll probably be 1tbish, many small files
18:02:35divi quits [Ping timeout: 265 seconds]
18:24:47godane1 joins
18:25:03divi (divi) joins
18:27:28godane2 quits [Ping timeout: 252 seconds]
18:51:35<@JAA>FYI, I just dumped the Mercurial repos of SDL (https://hg.libsdl.org/) using `hg clone --noupdate --stream` + `hg bundle --all --type zstd-v2`.
19:17:56aleph quits [Client Quit]
19:20:38aleph joins
19:22:29aleph quits [Client Quit]
19:24:44HackMii quits [Remote host closed the connection]
19:25:03HackMii (hacktheplanet) joins
19:26:59<mgrandi>Icculus is a bro and probably will give us whatever stuff we need to properly archive
19:29:24aleph joins
19:29:58<OrIdow6>yawkat: Put it into a .tar or .zip, upload it to archive.org (though it won't get into the Wayback Machine)
19:30:17<OrIdow6>Is the Mercurial project still running? That's slightly more comprehensive
19:30:40<OrIdow6>than a regular stream clone
19:32:58<@JAA>Not sure if there are still any workers or targets running for it.
19:33:22aleph quits [Read error: Connection reset by peer]
19:35:54aleph joins
19:49:22<mgrandi>How is it more comprehensive?
19:49:45<mgrandi>Just the WBM part?
19:50:13<mgrandi>And icculus just mentioned that he left the bugzilla up for us, I said it's finished and he is taking it down now
19:53:56DogsRNice (Webuser299) joins
19:56:49<@hook54321>thanks! a dump of the wiki and bugzilla would be appreciated as well, if possible.
19:57:48<@hook54321>we may not have gotten some of the more recent bugs on bugzilla
19:57:55@hook54321 shrugs
19:58:40britm0b quits [Read error: Connection reset by peer]
20:02:04<OrIdow6>mgrandi: At least one of obsmarkers, pushkeys, bookmarks (I've forgotten what specifically at this point)
20:02:48<OrIdow6>Which, again, are minor and ancillary
20:05:36britmob joins
20:06:44britmob quits [Read error: Connection reset by peer]
20:48:44icculus joins
20:49:10<icculus>Hi, this twitter thread brought me here: https://twitter.com/icculus/status/1359379854639190016
20:49:41<icculus>Specifically this part: https://twitter.com/AuroraPenguin/status/1359599254138884096
20:50:16<@EggplantN>Welcome icculus
20:50:21<icculus>If anyone wants the Bugzilla database (as opposed to just the scraped pages ArchiveTeam pulled in last night), feel free to hit me up.
20:51:54<icculus>SDL's Mercurial repositories are still talking to Mercurial clients (even after web browsers start getting redirected to GitHub), so you can just "hg clone https://hg.libsdl.org/$REPONAME" if you want final copies of any of the repositories.
21:00:53LeGoupil quits [Client Quit]
21:16:25<mgrandi>We got a clone of the HG repo, but that might not be 100% of the information , as @OrIdow6 mentioned
21:16:26britmob joins
21:17:05<mgrandi>@JAA ran this' 10:51:35 <@ JAA> FYI, I just dumped the Mercurial repos of SDL (https://hg.libsdl.org/) using `hg clone --noupdate --stream` + `hg bundle --all --type zstd-v2`.
21:19:59<mgrandi>But if you have a handy download of the bugzilla database, I can download it and throw it up on IA ,njust make sure it's not the part that has the sensitive user information :) @icculus
21:21:54<@hook54321>icculus: it looks like we're getting 503s for the wiki, do you want us to slow down?
21:33:48<icculus>hook54321: Slowing down is not a bad idea. It's...fragile.
21:34:45<icculus>hook54321: That being said: feel free to run it into the ground until you get all the info you need out of it.
21:36:30<icculus>mgrandi: as far as I know, if you get a clone of the Mercurial repo, you've got everything there is to get there. Note that there are several repos, though, if the "SDL" one is the only one that got grabbed. The list is on the main page for now: https://hg.libsdl.org/
21:36:43<mgrandi>This is mainly so if people want to look up the URL in WBM then they can
21:37:06<mgrandi>Like an old url that references a wiki page or bugzilla ticket
21:37:17<icculus>ah, okay
21:37:55<@JAA>icculus: I dumped all repositories listed on https://hg.libsdl.org/. However, I'm still looking into archiving it in another way such that people can directly `hg clone` from the Wayback Machine in the future.
21:38:30<@arkiver>JAA: we could try to use the existing mercurial project for that?
21:38:34<@arkiver>how many repos are there?
21:38:35<@JAA>The bundles can be found in https://archive.org/details/hg.libsdl.org_bundles_20210210 for future reference.
21:38:41<@arkiver>and do you have an example?
21:38:59<icculus>fwiw, all of those are going to redirect to the equivalent at GitHub, so https://bugzilla.libsdl.org/show_bug.cgi?id=4076 will automatically kick over to https://github.com/libsdl-org/SDL-1.2/issues/772 with all the content intact
21:39:00<@JAA>arkiver: Just the 10 listed on https://hg.libsdl.org/.
21:39:14<icculus>(same for most URLs under hg.libsdl.org)
21:39:33<@JAA>Yay for URL stability. :-)
21:39:45<icculus>Broken URLs drive me nuts. :)
21:40:06<@JAA>arkiver: I guess the question is whether there are still workers and targets/factories running for mercurial. But yes, that would be ideal. Otherwise, I'll run it manually.
21:40:44<@arkiver>JAA: i assume the zip/gz/bz2 files are already in wayback?
21:40:51<mgrandi>Fancy redirect logic, do you have a database of what they migrated to and then do mod_rewrite or similar?
21:41:00<icculus>JAA: the current plan is to have user-agents that start with "mercurial/proto-" talk to the Mercurial server. Other user agents will get redirected to the equivalent thing on GitHub.
21:41:22<@JAA>arkiver: I think the AB job should've grabbed those, but I didn't check.
21:41:44<icculus>mgrandi: yeah, it's just a sqlite3 database that maps between the two places and a little PHP to look it up and redirect.
21:45:07Matthww2 joins
21:45:15<atphoenix>Google Play Music drop dead date is Feb 24, 2021 per https://www.androidauthority.com/google-play-music-delete-data-1198079/ and as mentioned at https://twitter.com/princetrunks/status/1358172587440881665
21:45:40Matthww quits [Read error: Connection reset by peer]
21:45:40Matthww2 is now known as Matthww
21:46:19<@hook54321>arkiver: those were ignored
21:46:45<@JAA>hook54321: Even the ones linked on the homepage for the tip?
21:46:55<@hook54321>no idea
21:48:04<icculus>If you want the zip/gz/bz2 files for any revision, you can run "hg archive -t zip -r $REVISION /path/to/write/archive.zip" on your local copy and not destroy my server with all the compression effort. :)
21:48:27<@JAA>Yeah, we just want the tip archives. :-)
21:49:07<icculus>Many many crawlers have triggered the OOM killer on that server by just trying to get .zip files for 11,000 revisions. :)
21:49:23<icculus>Not gonna miss that!
21:50:27<@JAA>Can confirm that the AB job grabbed all ten tip archives in all three formats.
21:51:57<@JAA>arkiver: So yeah, we'd just want to run mercurial-grab on those ten I guess.
21:52:34<@JAA>Doesn't really make sense to power up the full machinery for that. It's only a couple hundred MB for the compressed bundles.
21:54:03<mgrandi>I can start some workers if needed
21:57:21DogsRNice quits [Ping timeout: 244 seconds]
22:01:49DogsRNice (Webuser299) joins
22:02:51Matthww quits [Client Quit]
22:03:10Matthww joins
22:07:15<icculus>mgrandi: (to follow up here) I have the bugzilla database, but I have to figure out where user info and email addresses and stuff end up before I hand it over. Do you want this as a big SQL dump or the actual MySQL binary files?
22:07:50<icculus>The database won't be deleted in any case, we aren't trying to free up disk space here. :)
22:09:50<mgrandi>Hmm, I feel like the mysqldump output is probably better, I feel like the binary files are non portable and are probably tied to mysql version and dialect
22:13:03<mgrandi>Maybe someone else here knows better?
22:14:07<mgrandi>But yeah, a zip of the mysql dump and the sqlite file of the bugzilla <-> github mappings (sqlite IS portable, I know that for a fact) and zip em up and someone here can upload them to IA
22:14:28<@JAA>Agreed, a dump is preferable.
22:17:02<mgrandi>As for the users, you might be able to just null out the password hash / email columns
22:17:47<@JAA>Wow, this is disgusting. I found a list of MoinMoin wikis (because it features the SDL one), and most of the wikis listed there are dead. :-|
22:17:52<@JAA>http://moinmo.in/MoinMoinWikis
22:18:00Arcorann (Arcorann) joins
22:19:08<OrIdow6>Checking that mercurial-grab gets it
22:29:45<OrIdow6>Looks like it should work
22:30:06<OrIdow6>(Have successfully done an hg clone from a local pywb instance)
22:37:55<mgrandi>Should throw those on AB to get ones that are not dead then
22:38:27<@arkiver>yeah checking as well
22:39:14<@JAA>mgrandi: Yes. We need an igset though before we do that.
22:40:45lennier1 quits [Ping timeout: 244 seconds]
22:42:10lennier1 (lennier1) joins
22:44:42lennier2 joins
22:47:45lennier1 quits [Ping timeout: 265 seconds]
22:47:52lennier2 is now known as lennier1
22:59:13Matthww quits [Client Quit]
22:59:33Matthww joins
23:09:32Matthww3 joins
23:10:43Matthww quits [Ping timeout: 252 seconds]
23:10:43Matthww3 is now known as Matthww
23:35:20divi quits [Client Quit]
23:36:45divi__ (divi) joins
23:36:52divi__ is now known as divi
23:37:33<@JAA>jodizzle: So uh, I have 7 million unique URLs from my inQuake forums grab. This is everything, including links within the forums. Needs a lot of filtering obviously. What did you want to target?
23:38:33hooway quits [Client Quit]
23:40:37<jodizzle>I think the only thing that was left that was particularly important was the member pages, but I captured those earlier via the member list you sent, and they're also being grabbed via the AB job currently
23:41:22<jodizzle>I guess we should do page requisites, but we probably have a lot of those already anyway, via the AB job
23:42:01<@JAA>There are only a couple hundred URLs within the domain that are not in /forum/, by the way.
23:42:37<jodizzle>Yeah, that was pretty weird. The AB job for that domain outside /forum/ was very small
23:42:42<jodizzle>Guess it was just a pretty small site
23:43:18<jodizzle>Outlinks would be nice to get, but those could also just be put through #//
23:43:24<@JAA>It's also quite broken. A lot of pages lead to redirect loops and whatnot.
23:43:37<jodizzle>Oh yeah, I noticed that too
23:45:29<@JAA>518k URLs on other domains

HomeSearchPrevious dayNext day