HomeSearchPrevious dayNext day

00:08:03Wayward (wayward) joins
00:19:22<mgrandi>Ty
00:39:24Wayward quits [Ping timeout: 240 seconds]
02:20:26nico_32 joins
03:05:02Dallas quits [Client Quit]
03:08:13Dallas (Dallas) joins
03:24:59qw3rty__ joins
03:28:56qw3rty_ quits [Ping timeout: 252 seconds]
04:24:56Nume quits [Quit: Ping timeout (120 seconds)]
06:09:26<jodizzle>mgrandi: Thanks for the wiki advice, but it looks like there was actually a dump of the wiki recently: https://archive.org/download/wiki-corebootorg.
06:09:44<jodizzle>Of course, I went to the trouble of setting up the wikiteam tools before I realized that
06:09:53<jodizzle>But I guess I have them for the future.
06:11:333INAAREDS joins
06:23:12rsn quits [Remote host closed the connection]
06:26:41Craigle quits [Quit: Ping timeout (120 seconds)]
06:30:02Craigle (Craigle) joins
06:31:28<mgrandi>Cool, glad it worked out
06:55:46Craigle quits [Client Quit]
06:56:14Craigle (Craigle) joins
07:01:26phuzion quits [Remote host closed the connection]
07:03:32phuzion (phuzion) joins
07:27:40<wickedplayer494>hah I came across flashkit and thought "oh this would be something good for archivebot" and then I go and look at the viewer then dashboard and see that we're already on it
07:27:41<wickedplayer494>nice.
07:53:57HP_Archiv quits [Read error: Connection reset by peer]
07:54:20HP_Archiv joins
07:54:32lennier2 joins
07:57:20lennier1 quits [Ping timeout: 252 seconds]
07:57:23lennier2 is now known as lennier1
08:37:26JeanFred quits [Remote host closed the connection]
09:01:36EggplantN2 joins
09:16:12Datechnom quits [Quit: Ping timeout (120 seconds)]
09:16:41Datechnom (Datechnom) joins
09:20:42sol joins
09:21:01<sol> Hello. Is anyone here archiving the NY Review of Books? All articles are open to (registered) non-subscribers until November 5th.
09:21:11<sol>It's a Wordpress-based site. But I'm having difficulty getting wget and httrack to work with the cookies.txt I supply. I feel they might both be downloading the "logout" link, despite my trying to exclude them from doing so.
09:22:52<sol>And should this ^^^ discussion go here or in #archiveteam-ot?
09:23:36<jodizzle>It should go here, I think
09:25:01<jodizzle>sol: Don't have much time to look into this atm, but you may want to try grab-site instead: https://github.com/archiveteam/grab-site
09:25:27<jodizzle>You can provide cookies, and also you get everything saved in WARCs
09:25:49<sol>Thanks. I'll take a look at grab-site today.
09:51:50<sol>So, doing `grab-site https://www.nybooks.com --wpull-args=--load-cookies=/home/sol/gs-venv/cookies.txt` should work?
09:53:59<sol>How do I check if grab-site is downloading the full articles available to registered users or only the partial articles available to unregistered users?
11:25:05sol quits [Remote host closed the connection]
11:55:53Jean-Fred joins
12:10:52dashcloud quits [Ping timeout: 252 seconds]
13:16:52<katocala>!ao https://www.dailymail.co.uk/news/article-8867267/Borats-Bulgarian-honey-trap-girl-Glamorous-BLM-supporting-24-year-old-plucked-obscurity.html
13:17:10<katocala>well...
13:53:58Arcorann quits [Read error: Connection reset by peer]
14:14:47@arkiver quits [Excess Flood]
14:15:16arkiver (arkiver) joins
14:15:16@ChanServ sets mode: +o arkiver
14:15:49<Lord_Nightmare>I just made an observation about ripping certain CDs: some CDs are encoded in a special/weird format called "HDCD" which allows dynamic range and other trickery to effectively store 20-bit audio using a 16-bit cd, but part of the data encoded is stored in the CD subchannel data
14:16:06<phuzion>Lord_Nightmare: Are you talking about SACDs?
14:16:17<Lord_Nightmare>so a straight audio rip of the CD loses this data and the HDCD decoder, without that data, has to guess at the multiplier for the dynamic range
14:16:18<phuzion>Or is that something completely different?
14:16:22<Lord_Nightmare>no, HDCD, which is a bit of a hack
14:16:33<phuzion>Interesting.
14:16:47<Lord_Nightmare>https://en.wikipedia.org/wiki/High_Definition_Compatible_Digital
14:16:53<phuzion>Yeah I'm reading up on it now
14:16:57<Lord_Nightmare>https://forum.doom9.org/showthread.php?s=3b8f9e12e85bd95f9e4926f49a80f69b&t=129136&page=4 explains how the subchannel thing works, kind of
14:17:18<Lord_Nightmare>this means that ripping any HDCD cd *MUST INCLUDE* the subchannel data to be an accurate rip
14:17:18<phuzion>I'm vaguely familiar with SACDs, and actually quite familiar with redbook CDs, so this is new and interesting.
14:17:35<Lord_Nightmare>so it has to be ripped as bin/cue with subchannels enabled
14:17:50<Lord_Nightmare>which is different from pretty much ANY OTHER audio cd
14:18:02<phuzion>I wonder if EAC can rip it
14:18:05<Lord_Nightmare>now, there are some audio cds which have audio in the pregaps
14:18:14<Lord_Nightmare>which is technicaly against the spec
14:18:55<Lord_Nightmare>they're supposed to be 2 seconds of silence encoded as audio
14:19:05<Lord_Nightmare>a proper cd ripper should rip those too
14:20:05<Lord_Nightmare>does archiveteam have a wiki about ripping CD data?
14:20:11<Lord_Nightmare>this should be noted there
14:20:14<Lord_Nightmare>if possible
14:20:20<phuzion>Lord_Nightmare: It's probably not on the AT wiki, but it might be on fileformats
14:20:35<phuzion>http://fileformats.archiveteam.org/wiki/Main_Page
14:21:37<Lord_Nightmare>the subchannel data on audio CDs uses the P and Q channels to hold the timecode data (as would be displayed on the LCD of a diskman etc) and should be a consistent format and timing from the end of the pregap, barring any offset craziness
14:21:39<@JAA>EAC and whipper definitely cover pre-gaps.
14:21:58<Lord_Nightmare>but on these HDCDs, I guess the remaining 6 subchannels are used to store other data
14:22:14<Lord_Nightmare>how subchannels are actually stored is WEIRD AS HELL
14:23:10<Lord_Nightmare>byuu had a series of tweets, which they have since deleted, about how EFM (https://en.wikipedia.org/wiki/Eight-to-fourteen_modulation ) works and how the 'noise-whitened' bits etc are stored as pits/lands on the cd surface
14:23:31<Lord_Nightmare>there was also a lot of discussion about this on the domesday duplicator discord
14:23:32<@JAA>We need a KryoFlux-like thing for CDs. :-|
14:23:41<Lord_Nightmare>since EFM is also used on laserdiscs
14:24:01<Lord_Nightmare>...and the domesday guys observed that many lasterdisc players have RF debug pickups from the laser head
14:24:03<@JAA>I suppose a microscope should work in principle.
14:24:07<Lord_Nightmare>which is also used to read CDs
14:24:09<Lord_Nightmare>in the LD player
14:24:19<Lord_Nightmare>and the CDs can be raw dumped the same way LDs can
14:25:04<Lord_Nightmare>so yes there *IS* a kryoflux thing for CDs, though you may need to patch the laserdisc drive firmware to make it play the entire CD sequentially from lead-in/TOC to lead-out
14:25:15<Lord_Nightmare>then decode the resulting 300GB analog file
14:26:07<Lord_Nightmare>laserdiscs are analog/PWM data stored as pits/lands, while cds are entirely digital, so the domesday duplicator board, which is meant for reading LDs, is storing a whole hell of a lot of redundant/unnecessary data
14:27:10<Lord_Nightmare>later laserdiscs which have digital/stereo audio use EFM encoding for that, similar to CDs
14:28:27<@JAA>Oof
14:28:39<balrog>someone needs to reverse-engineer MakeMKV's "libredrive" crap
14:30:40<balrog>there are a bunch of techniques that basically read a sector or a few, dump the drive cache, extract the pre-decoded EFM from it
14:30:52<balrog>rawdump/friidump for GC/Wii works like that
14:30:55<balrog>it is slow
14:33:34<balrog>it's not quite as low-level as using a domesday duplicator, but it's more than enough
14:55:09rsn joins
15:01:44<Jean-Fred>Hello! Upping the PSN topic − mgrandi any chance you had a look? I tried getting you these URL list you asked for (via VGPC folks who crawled the site ages ago) − it’s possible but did not get it done so far :-( (also cc lennier1 who raised the topic couple of days ago )
15:28:30@arkiver quits [Excess Flood]
15:28:59arkiver (arkiver) joins
15:28:59@ChanServ sets mode: +o arkiver
15:48:07@arkiver quits [Excess Flood]
15:48:35arkiver (arkiver) joins
15:48:35@ChanServ sets mode: +o arkiver
16:19:31lennier2 joins
16:22:04lennier1 quits [Ping timeout: 240 seconds]
16:22:11lennier2 is now known as lennier1
16:30:48@arkiver quits [Excess Flood]
16:31:18arkiver (arkiver) joins
16:31:18@ChanServ sets mode: +o arkiver
17:01:11dashcloud joins
17:04:46nerdguy1138 quits [Remote host closed the connection]
17:17:18@arkiver quits [Excess Flood]
17:17:46arkiver (arkiver) joins
17:17:46@ChanServ sets mode: +o arkiver
17:26:09atphoenix_ joins
17:29:08atphoenix quits [Ping timeout: 252 seconds]
17:32:01<jodizzle>sol: I think the only way to be sure would be to examine the contents of the WARC. There are a variety of tools that you can use to help you with that.
17:35:57Raccoon (wayward) joins
17:36:24Raccoon is now known as RJHacker81577
17:45:16dashcloud quits [Ping timeout: 252 seconds]
18:06:48atphoenix__ joins
18:10:56atphoenix_ quits [Ping timeout: 252 seconds]
18:27:05atphoenix__ quits [Ping timeout: 252 seconds]
18:27:37atphoenix joins
18:30:35atphoenix_ joins
18:34:24atphoenix quits [Ping timeout: 240 seconds]
18:46:30Eggplant joins
18:50:04EggplantN2 quits [Ping timeout: 240 seconds]
19:15:02RJHacker81577 is now known as Wayward
19:22:38atphoenix__ joins
19:25:24atphoenix_ quits [Ping timeout: 240 seconds]
19:39:10<OrIdow6><tech234a> YouTube-DL just got DMCA’d by the RIAA https://github.com/github/dmca/blob/master/2020/10/2020-10-23-RIAA.md
19:39:17<OrIdow6><tech234a> GitHub repo is gone but PyPi listing is still up
19:39:46<OrIdow6>"DMCA'd" meaning that they're basically trying to remove all public copies
19:40:07<OrIdow6>(First two from -ot)
19:40:07<tech234a>https://pypi.org/project/youtube_dl/#files
19:41:50<@JAA>Oof
19:43:05<OrIdow6>Now to find all the issue trackers etc.
19:44:57<Wayward>updates don't work either
19:45:05<@JAA>Issues will be gone.
19:45:10<@JAA>Unless the repo is restored.
19:45:20<@JAA>Same with PRs.
19:46:16Eggplant quits [Read error: Connection reset by peer]
19:46:52<OrIdow6>Is the #gitgud far along to have gotten it? Haven't been paying attention to that
19:46:57<OrIdow6>very much
19:47:32<Wayward>One thing I'm not clear on is how these "rolling cipher technical protection measures" that YouTube is said to employ and that youtube-dl is said to circomvent, the main basis of this DMCA claim, is even a thing. What makes youtube-dl any different from a web browser that poles data from YouTube in order to display it or store it in memory, cache, harddisk space?
19:47:57<@JAA>Doesn't look like the GitHub project got it, unless it happened very recently.
19:48:07<Wayward>Every web browser MUST definitionally circomvent these measures
19:48:08<@JAA>Wayward: Please keep that discussion in -ot.
19:48:58<@JAA>Discussion about archival here, discussion about the notice and consequences and legal stuff and whatever in -ot.
19:51:16tar-xvf quits [Remote host closed the connection]
19:51:29tar-xvf joins
19:58:44<OrIdow6>The second top-level comment in the HN thread has a link to a WBM copy and tells people to be rebels and download it that way; if that keeps up, would not be surprised if an AB warc gets darked or similar
19:59:23<OrIdow6>Nonetheless, not much to change on AT's part even if that's the case
19:59:39<maxfan8>Wdym by darked?
20:01:00<Ajay>I think darked is when it is hidden from the public, but still kept in the archive for future preservation
20:03:01<maxfan8>Ah nice, thanks
20:04:16<@JAA>Good thing that Git is a DVCS, so every dev still has their local copy of nearly the entire repo.
20:13:20<maxfan8>Yeah I have a local clone cloned repo of it somewhere
20:13:50<@JAA>Same, I have a normal clone (i.e. no PRs) from end of September.
20:17:28<OrIdow6>Looks like a lot of the issues are still in Google Web Cache
20:17:48<OrIdow6>Presumably there will be many copies of the repo - I'm more concerned about the non-code
20:23:04<mgrandi>@Jean-Fred: not yet but tonight hopefully, I think if I login I can still see the ps3 stuff on the store which is good (https://store.playstation.com/en-us/product/UP9000-NPUI94199_00-0000000000000001?PlatformPrivacyWs1=exempt&psappver=19.15.0&scope=sceapp&smcid=psapp%3Alink%20menu%3Astore )
20:23:05sec^nd quits [Remote host closed the connection]
20:23:25sec^nd (second) joins
20:28:52<@JAA>OrIdow6: Yep, working on that.
20:29:36<mgrandi>Youtube dl got dcma-ed
20:29:39<mgrandi>https://twitter.com/scriptjunkie1/status/1319731273477328896
20:30:10<OrIdow6>Look up
20:30:49<mgrandi>Oh yeah
20:31:01<mgrandi>People are saying it will probably be restored
20:31:53<mgrandi>Should check back to see if it gets put up again so we can pull the issues
20:32:39<@JAA>Yep, just set up a monitor.
20:35:10dashcloud joins
20:37:31<mgrandi>So you can't git clone from way back machine copies of the repo? How did that work for the mercurial project, just because HG has a weird wire format that we also scraped?
20:38:40icedice joins
20:39:58<@JAA>There's a bundle on IA from late September: https://archive.org/details/github.com-ytdl-org-youtube-dl_-_2020-09-28_19-41-32
20:42:16<mgrandi>There is a way back machine url from the 18th but you can't seem to git clone it, but it got the tar archives at least.
20:43:10<@JAA>*cough* https://github.com/rbrito/pkg-youtube-dl *cough*
20:43:34<icedice>I have https://anonfiles.com/daH1Leibp9/youtube-dl.git.tar_xz
20:43:40<icedice>Source: https://gitee.com/mirrors/youtube-downloader
20:44:42<tech234a>Yep just saw that mirror on HN
20:45:06<tech234a>Seems to have a commit from today
20:45:11<icedice>I have youtube-dlc btw, but only the binary
20:45:43<icedice>(youtube-dlc was a fork of youtube-dl that was more up to date on fixing pull requests)
20:49:39nerdguy1138 (nerdguy1138) joins
20:50:05dashcloud quits [Ping timeout: 252 seconds]
21:01:18<@JAA>So here's what I have: PyPI including wheels and tars (via AB), website (via AB, not much on there), all versions of the Debian package, bundle of the Debian package repo
21:01:26<@JAA>Also a bundle of that Gitee mirror.
21:01:46<@JAA>I ran the Debian package repo through #gitgud as well.
21:01:57<@JAA>And as a precaution, same for Invidious.
21:02:52<@JAA>Debian package versions are being downloaded by AB at the moment.
21:03:38<@JAA>The Google cache attempt quickly failed due to rate limiting. I wanted to spread that across the different AB pipelines.
21:03:54dashcloud joins
21:04:15<@JAA>There are 54k URLs to be retrieved (27k each for issues and PRs), and it needs a high delay, so it'd take ~25 days on a single AB job...
21:11:46<@JAA>Arch Linux package archive done via AB.
21:12:32<Ajay>maybe run newpipe through gitgud if that's not already done?
21:13:15<@JAA>Heh, 30 seconds ago: https://github.com/TeamNewPipe/NewPipe/issues/4618
21:13:54<Barto>JAA: that's for all that job
21:14:11<@JAA>Ajay: Done and bundle grabbed.
21:14:53lennier2 joins
21:15:15<tech234a>Does anyone know if notices are posted to the DMCA repo before the repos are made inaccessible?
21:15:24<tech234a>Also https://github.com/github/dmca/pull/8124 :)
21:18:18lennier1 quits [Ping timeout: 252 seconds]
21:18:23lennier2 is now known as lennier1
21:21:37<hexa->https://old.reddit.com/r/programming/comments/jgub36/youtubedl_just_received_a_dmca_takedown_from_riaa/g9sphpg/?context=3
21:22:45<tech234a>Huh so they’re going after contributors also
21:23:07<OrIdow6>hexa-: See the frantic discussion above
21:23:56<tech234a>The link mentions contributors are facing legal action
21:24:14<OrIdow6>Oh, didn't see the context param
21:24:19<OrIdow6>Sorry hexa-
21:24:46<icedice>https://github.com/github/dmca/pull/7584
21:24:56<icedice>^ not very smart move
21:27:09<@JAA>icedice: You clearly didn't read what that is about. Also, unrelated.
21:27:48<icedice>I thought they were saying that it's reuploaded to Archive.org in the initial post
21:27:54<icedice>I might have been mistaken
21:28:57<@JAA>It's not even a DMCA notice. It's an explanation why GitHub makes the notices public.
21:29:02<@JAA>Or well, a quote about that.
21:49:04<icedice>https://notabug.org/, https://codeberg.org/, and https://repo.or.cz/ might be worth archiving to as well, according to https://github.com/TeamNewPipe/NewPipe/issues/4618#issuecomment-715599050
21:50:49<Ajay>icedice: no, they are saying that those are alternative places the repository could be hosted
21:51:57<icedice>archiving to
21:52:00<icedice>Not from
21:54:16<Ajay>ooh, sorry I misread
21:54:32<Ajay>to migrate issues, I think you need to be an admin of the repository
21:54:39<Ajay>for gitea and gitlab
22:02:57HP_Archiv quits [Read error: Connection reset by peer]
22:03:39HP_Archiv joins
22:42:50colona (colona) joins
23:12:14Arcorann (Arcorann) joins
23:52:03<nico_32>i have youtube-dl repo from commit d65d89183f645a0e95910c3861491a75c26000eb // Thu Sep 24 07:36:38 2020 +0700
23:52:58<nico_32>so the whole history
23:53:03<nico_32>git is really nice for that
23:54:05<@JAA>Funny, same commit I also have on a local clone. But also, there are full copies on GitHub and Gitee up to today.
23:57:40<nico_32>DMCA takedown make me keep local copy of everything
23:58:07<nico_32>mmorpg emulator, fanfiction, mashup
23:58:10<nico_32>everything
23:59:06<Ryz>Loot all the things, looooooooooot
23:59:28<nico_32>dozen of po on local nas are cheap
23:59:36<nico_32>s/po/to/
23:59:41<nico_32>sorry
23:59:52<nico_32>would be nice to have a few po :)

HomeSearchPrevious dayNext day