| 00:00:38 | | icculus_ joins |
| 00:32:04 | <Ryz> | Hey folks, has there been an instance where there's a Twitter account that still remains on the website even though it's suspended, but all the information has been edited with Twitter's message instead? Or is it fake and this is a pretend Twitter account? The one I stumbled across is https://twitter.com/Star_Wars_porn |
| 00:33:12 | <@Kaz> | don't think that 'Learn more' link is possible without that |
| 00:33:20 | <@Kaz> | so yes - I think that's legitimate |
| 00:33:51 | <AltroskyAS207616_Mystique> | Ryz: that's a recent thingie i think |
| 00:36:05 | <Ryz> | Ah, interesting, usually from my experience, suspended account just have the entire Twitter account down, not something like a shell of it's former self |
| 00:46:28 | | benjinsmith quits [Client Quit] |
| 01:00:31 | | ragu__ quits [Ping timeout: 265 seconds] |
| 01:02:27 | | benjins joins |
| 01:02:39 | | benjins is now authenticated as benjins |
| 01:06:24 | | icculus_ quits [Ping timeout: 244 seconds] |
| 01:07:56 | | BlueMaxima joins |
| 01:21:04 | | icculus_ joins |
| 01:29:35 | | britmob joins |
| 01:32:31 | | britm0b quits [Ping timeout: 252 seconds] |
| 01:42:05 | | godane quits [Ping timeout: 265 seconds] |
| 01:42:53 | <flashfire42> | No that happens when its only the images it posting violate the guildines the first time |
| 01:42:58 | <flashfire42> | Usulaly used for pornography |
| 02:04:48 | | benjins quits [Ping timeout: 265 seconds] |
| 02:09:07 | | icculus_ is now known as icculus |
| 02:09:31 | <@JAA> | I ran mercurial-grab manually on the SDL repos so it doesn't get forgotten or whatever: https://archive.org/details/hg.libsdl.org_clone_warc_20210211 |
| 02:09:41 | <icculus> | mgrandi: to answer the question: there were definite ArchiveBot user agents, but the wiki downloads did not appear to be using it. |
| 02:09:53 | <icculus> | But maybe I misunderstood what I saw |
| 02:10:12 | <icculus> | bugzilla can live a little longer, it's no big deal |
| 02:11:45 | <mgrandi> | @JAA: ^ |
| 02:12:09 | <mgrandi> | @Ryz: there is also the thing where embeds of twitter accounts that were suspended still work |
| 02:12:28 | <mgrandi> | See: trump's twitter account, however the media that was contained in it is still not accessible |
| 02:16:10 | <Ryz> | Huh, that's bewildering~ Maybe I need to see an example of it working, or maybe that one is a special case? |
| 02:27:09 | <@JAA> | icculus: Yeah, our first attempt for the wiki was with the ArchiveBot UA, but we often switch to a browser UA when we see bot blocks or similar things (which the 503s looked like, I guess; I didn't run or watch it). The current job uses the AB UA again. |
| 02:33:47 | <@JAA> | The hg derive finished, so once it gets picked up by the WBM index, `hg clone` from the WBM should work. :-) |
| 02:37:54 | <atphoenix> | Ryz, as described in the Q/A on https://www.electoral-vote.com/evp2021/Pres/Maps/Jan09.html ... "the content of the tweet is actually built into the html code that Twitter provides. And so, the pretty formatting and the information about likes/retweets/comments is now lost, but the actual content remains" |
| 02:43:22 | | Sylirana_ joins |
| 02:45:02 | | Eighty (Eighty) joins |
| 02:45:36 | | Sylirana quits [Ping timeout: 244 seconds] |
| 02:56:02 | | benjins joins |
| 02:56:20 | | benjins is now authenticated as benjins |
| 03:07:28 | | Hackerpcs quits [Quit: Hackerpcs] |
| 03:09:15 | | Hackerpcs (Hackerpcs) joins |
| 03:24:24 | | Sylirana_ quits [Client Quit] |
| 03:25:34 | | Sylirana (Sylirana) joins |
| 03:28:18 | <mgrandi> | i'm not sure if anyone saw my link, but apparently twitter is allowing access to twitter's historical data, but i'm not sure if it includes suspended users |
| 03:31:19 | <flashfire42> | weird https://sflan.net/ redirects to archive.org |
| 03:31:34 | | Wayward- is now known as Wayward |
| 03:36:22 | <Sylirana> | Who would be the person to ask if I'm interested in taking over the project for CodePlex? Currently there doesn't seem to be a project lead and from what I could see, nobody is currently working on saving it. |
| 03:36:49 | | benjins quits [Ping timeout: 252 seconds] |
| 03:38:31 | <@JAA> | Sounds right. There's nobody to ask for permission. Just start working on it and document your findings. :-) Discovery of repositories, what to grab for each repository, and a size estimate would be nice. |
| 03:39:27 | <mgrandi> | so you said that microsoft archived codeplex? is that available somewhere, like google code readonly mode? |
| 03:39:46 | <@JAA> | https://archive.codeplex.com/ |
| 03:39:50 | <@JAA> | That's what's shutting down in a few months. |
| 03:40:28 | <mgrandi> | ah ok |
| 03:40:41 | <mgrandi> | yeah, seems pretty simple, first thing is to figure out how the URL structure works and...generate a giant list of urls |
| 03:40:56 | <Sylirana> | Already did that. :-) |
| 03:41:18 | <mgrandi> | oh, well that is..most of the work? i guess what is left: stuff that needs to be ignored? |
| 03:41:28 | <Sylirana> | Going to edit the Wiki page with more info about it when I have a bit more time. |
| 03:42:04 | <mgrandi> | then the standard copy/paste/edit of the DPoS project structure with your findings / ignores and then we can test it and archive it with time to spare |
| 03:42:05 | <Sylirana> | Well I meant the list of urls. It's not downloaded yet, mostly due to how strictly they're limiting everything... And the fact that there are lots of tiny files... |
| 03:42:23 | <mgrandi> | limits is also important to figure out |
| 03:43:20 | <mgrandi> | oh, the most important thing: thinking of the IRC channel name |
| 03:43:29 | <Sylirana> | DPoS? o.o I already started with the downloads... If it continues at it's current rate, I should have everything within the next month. |
| 03:43:56 | <mgrandi> | oh, well that might be it then. as WARCs or just plain files? |
| 03:44:02 | <Sylirana> | Just need to figure out where and how to upload it... 100k+ files doesn't seem like the best idea. |
| 03:44:15 | <@JAA> | Yeah, we'll want it as WARCs. |
| 03:44:18 | <mgrandi> | ah, so plain files |
| 03:44:41 | <mgrandi> | we can possibly keep that copy and then try to get a proper copy as WARCs with a DPoS, and compare and make sure we got everything |
| 03:44:46 | <Sylirana> | I'm getting all of the zip files as they do contain all of the extra data about wikis and issues. Some of it even as JSON, so it can be parsed later. |
| 03:45:36 | <mgrandi> | benefits of getting it as WARCs: it perserves the HTTP headers, and since its coming from a URL, they could possibly be put into wayback machine |
| 03:46:12 | <mgrandi> | but finish the download, and if you don't have space for it, i'll take it and hold onto it while we work in the DPoS project, and we'll be extra safe |
| 03:47:04 | <Sylirana> | The structure is pretty simple. https://archive.codeplex.com/?p=photocollage |
| 03:48:11 | <Sylirana> | The list can be found in the sitemap.xml. 108516 projects. +1 if you count the start page with the shutdown message. |
| 03:48:48 | <@JAA> | Would be good to check if there are any projects which don't appear in the sitemap. |
| 03:50:22 | <Sylirana> | I've thought about thsat too, but I can't really think of a good way to look for that except hoping that some other site links to one that doesn't appear in the sitemap... |
| 03:51:19 | <mgrandi> | are you only downloading urls via the sitemap? |
| 03:51:48 | <Sylirana> | Something like "link:https://archive.codeplex.com/" comes to mind, but to make sure, you'd have to get all the Google results and then check if you find a new project link on the page... |
| 03:53:54 | <@JAA> | Yeah, it isn't always easy to find those things. |
| 03:54:03 | | benjins joins |
| 03:54:12 | <@JAA> | Search engines are of limited use as they're often based on the sitemap. |
| 03:54:16 | | benjins is now authenticated as benjins |
| 03:54:20 | <Sylirana> | The sitemap seems to list all of them. The site is kept rather simple. All I could find are the start page, project pages and the search. The sitemap seems to be just a generated list of all projects, sorted alphabetically. Warning, large xml! https://archive.codeplex.com/sitemap.xml |
| 03:55:13 | <mgrandi> | a site with an actual useful sitemap? what is this |
| 03:55:31 | | Doranwen quits [Ping timeout: 252 seconds] |
| 03:55:49 | <@JAA> | Insanity. Heresy. |
| 03:57:24 | <@JAA> | https://web.archive.org/web/*/codeplex.com/* might be useful for finding more projects possibly. |
| 03:58:23 | <Sylirana> | One thing to note: While the .zip archives urls do follow a simple pattern, the project page is always lowercase, while in some cases, the filename of the .zip is not, which will lead to a 404 if you just go by the sitemap. My current approach is to get the archies from the sitemap and then for each missing one (eg. the ones with uppercase letters) getting the project page which does contain the uppercase letters. |
| 03:59:35 | <Sylirana> | Now I could of course have less manual work and get the page for every project... But I would be making a lot of redundant requests since the majority of project are all lowercase anyway. |
| 03:59:56 | <Sylirana> | *projects |
| 04:01:57 | | godane (godane) joins |
| 04:03:51 | <ThreeHeadedMonkey> | Project pages seem like something we should be archiving, that content seems to be missing from the zip files as far as I can tell |
| 04:04:28 | <Sylirana> | I have checked multiple, it was always in the .zip. o.O |
| 04:06:02 | <@JAA> | We will for the WARC copy. |
| 04:06:12 | <Sylirana> | We could of course archive them all, but I'm not sure exactly how I'd do that. At least not with the resources I have currently. |
| 04:06:24 | <ThreeHeadedMonkey> | Nevermind, found it |
| 04:07:32 | <Sylirana> | I'd have to split it up. Not sure how well/if that works with WARCs though. Only have around 50 GB after which I have to move things over before I can download the next batch. |
| 04:08:42 | <Sylirana> | If I can somehow easily split up the WARCs, I'd gladly work on WARCs. Got quite a bit of traffic to spare this month. ;-) |
| 04:10:18 | <ThreeHeadedMonkey> | wget has --warc-max-size, or you could just make separate WARCs for each project |
| 04:11:20 | <ThreeHeadedMonkey> | The project pages may be a bit annoying though, looks like they're completely loaded through JS |
| 04:12:22 | <Ajay> | are the API endpoints predictable? |
| 04:12:34 | <Ajay> | if so, wget on the API should work |
| 04:13:21 | <Sylirana> | https://archive.codeplex.com/metadata/photocollage.json |
| 04:15:46 | <ThreeHeadedMonkey> | Seems easy enough, assuming those filenames are consistent |
| 04:16:28 | <Ajay> | there is https://archive.codeplex.com/projects/photocollage/discussions/discussions.json and https://archive.codeplex.com/projects/photocollage/issues/issues.json |
| 04:16:56 | <ThreeHeadedMonkey> | Yeah, that and individual JSON files for each issue/discussion |
| 04:17:02 | <Ajay> | yep |
| 04:17:19 | <Ajay> | that would have to be discovered from the issues json |
| 04:17:20 | <ThreeHeadedMonkey> | Although those seem to already be part of the ZIP archive |
| 04:17:29 | <Ajay> | https://archive.codeplex.com/projects/photocollage/issues/20607/20607.json |
| 04:17:53 | <Ajay> | it'd be nice to get them as warc for wayback machine support |
| 04:18:24 | <Ajay> | but maybe the files from the archive can be used, not sure how that works |
| 04:18:45 | <Sylirana> | But wouldn't we have to rewrite the JS for Wayback Machine support? |
| 04:19:08 | <Ajay> | wayback machine rewrites urls automatically |
| 04:19:23 | <Sylirana> | Even in obfuscated JS? |
| 04:19:24 | <Ajay> | so, the files would just have to be in the right place |
| 04:21:50 | <Ajay> | Sylirana: I just took at look at the JS and it seems to already use relative urls |
| 04:22:33 | <mgrandi> | @Sylirana thats what a DPoS project is for, splitting up the work |
| 04:22:34 | <Ajay> | not for the "download archive" button though |
| 04:22:46 | <Ajay> | or search |
| 04:22:48 | <Ajay> | the rest do |
| 04:22:48 | <Sylirana> | https://archive.codeplex.com/static/js/main.3a3229e5.js.map |
| 04:23:01 | <Sylirana> | Don't ask me why, but the source map is public apparently. |
| 04:23:02 | | icculus quits [Ping timeout: 265 seconds] |
| 04:23:43 | <Ajay> | there are a lot of variable names kept, so I think it is just minified |
| 04:24:09 | <Ajay> | is there a channel for this project? |
| 04:25:41 | <Sylirana> | According to the Wiki it's #archiveteam on EFnet. (Which has a topic telling you to go to the same channel on hackint, so probably a lot of the IRC links on the Wiki need updating...) |
| 04:26:05 | <mgrandi> | we have to come up with some cool pun-y name for this |
| 04:26:14 | <Ajay> | I think that's the default before any are created |
| 04:26:15 | <OrIdow6> | This is going down in July? |
| 04:26:19 | <Ajay> | yes |
| 04:26:22 | <@arkiver> | yeah July |
| 04:26:28 | <Sylirana> | I've asked here because "Please only post urgent things here (e.g. websites' shutdown announcements) and use the channels mentioned in the topic for everything else". |
| 04:26:37 | <@arkiver> | I can get a project up later for this |
| 04:26:42 | <Ajay> | yes, this is the right channel |
| 04:26:43 | <@arkiver> | most important it the list of items |
| 04:26:50 | <@arkiver> | Sylirana: you go lists? |
| 04:27:15 | <Sylirana> | What do want? Project name or links to pages? |
| 04:27:21 | <@arkiver> | sitemap.xml takes a long time to load for me |
| 04:27:36 | <Sylirana> | It loads forever in the browser... I've just used curl. |
| 04:27:43 | <@arkiver> | ok |
| 04:28:04 | <ThreeHeadedMonkey> | The search function might be a way to discover more projects, the API seems to just return everything if you give it an empty search string |
| 04:28:06 | <ThreeHeadedMonkey> | https://codeplexarchive-search.search.windows.net/indexes/codeplexarchive-index/docs?api-version=2016-09-01&api-key=44C8CF90A6561D9EC9A1BBA09250FEA1&$top=1000&highlight=Title&search= |
| 04:28:29 | <@arkiver> | Sylirana: does the sitemap list all project? |
| 04:28:32 | <Sylirana> | Afaik that's limited though. |
| 04:28:53 | <Sylirana> | (The search, I mean.) |
| 04:29:13 | <@arkiver> | i see this contains git and mercurial |
| 04:29:18 | <Sylirana> | The sitemap does appear to list all of them, but that is not confirmed. |
| 04:29:25 | <Sylirana> | For some projects only. |
| 04:29:25 | <@arkiver> | lets wait until i have the next phase up for github |
| 04:29:36 | | maxfan8 is now authenticated as maxfan8 |
| 04:29:37 | <@arkiver> | also lets make a channel |
| 04:29:41 | <@arkiver> | ideas? |
| 04:30:13 | <ThreeHeadedMonkey> | The search stops at 1000 items/request, but it gives you a pre-made link for the next query if you ask for more than that |
| 04:30:29 | <@arkiver> | ThreeHeadedMonkey: found anything not in the sitemap? |
| 04:30:46 | <Sylirana> | I'm not really a creative person... DontDeleteMyCodePlex? o.O |
| 04:31:12 | <ThreeHeadedMonkey> | arkiver: Haven't looked through it yet, just checked if it works |
| 04:31:15 | <@arkiver> | mgrandi: creative? ^ |
| 04:32:30 | <Ajay> | codemby |
| 04:32:34 | | DogsRNice quits [Read error: Connection reset by peer] |
| 04:32:49 | <Ajay> | alternative to plex movie software |
| 04:33:16 | <@arkiver> | nice and confusing |
| 04:33:31 | <@arkiver> | i like it :P |
| 04:33:39 | <@JAA> | lol |
| 04:33:56 | <@arkiver> | #codemby |
| 04:33:57 | <@JAA> | Let's ... see if we can come up with something better... :-P |
| 04:34:07 | <@arkiver> | awwwwww sad |
| 04:34:19 | <@arkiver> | alright JAA says no :P |
| 04:34:26 | <@arkiver> | gotta come up with something better Ajay |
| 04:34:45 | <thuban> | #codewrecks |
| 04:35:38 | <@arkiver> | or |
| 04:35:41 | <@arkiver> | plexicode |
| 04:35:43 | <@arkiver> | from plexiglass |
| 04:35:58 | <Ajay> | lol |
| 04:36:05 | <@arkiver> | does JAA approve ^ |
| 04:36:08 | <Ajay> | I like it |
| 04:36:08 | <thuban> | plexiglass is legit tho |
| 04:36:15 | <@JAA> | Sounds good :-) |
| 04:36:21 | <@arkiver> | HE APPROVES |
| 04:40:59 | <purplebot> | CodePlex edited by JustAnotherArchivist (+91, Update channel, status, URL) just now -- https://www.archiveteam.org/?diff=46315&oldid=46273 |
| 04:51:23 | <atphoenix> | codeflex |
| 04:52:23 | <atphoenix> | okay I see you guys went to #plexicode |
| 04:56:05 | | qw3rty__ joins |
| 04:59:17 | | qw3rty_ quits [Ping timeout: 265 seconds] |
| 05:00:12 | <atphoenix> | otherwise I'd also say try on 'plexstore' or 'decodeplex' or 'diplocodeplexicus' |
| 05:00:56 | <thuban> | aw, i like the last one |
| 05:11:05 | <atphoenix> | Nothing wrong with plexicode or codewrecks either, but hey, I was AFK, and channel naming is very much a must-be-present-to-win game :) |
| 05:11:53 | | etnguyen03 quits [Client Quit] |
| 05:12:32 | | icculus joins |
| 05:13:24 | | nertzy quits [Read error: Connection reset by peer] |
| 05:13:44 | | nertzy (nertzy) joins |
| 06:23:41 | | Doranwen (Doranwen) joins |
| 06:35:05 | | daalekz (daalekz) joins |
| 06:48:00 | <mgrandi> | This was forwarded to me , for lols https://usercontent.irccloud-cdn.com/file/gT8zZhIR/IMG_20210211_224715_513.jpg |
| 06:56:11 | | programmerq quits [Ping timeout: 244 seconds] |
| 06:58:06 | | programmerq (programmerq) joins |
| 07:05:06 | | lennier1 quits [Remote host closed the connection] |
| 07:20:54 | | no112 quits [Ping timeout: 265 seconds] |
| 07:29:02 | | Wayward- (wayward) joins |
| 07:30:05 | | Wayward quits [Ping timeout: 265 seconds] |
| 07:33:54 | | icculus quits [Ping timeout: 244 seconds] |
| 07:36:16 | | icculus joins |
| 07:36:41 | | daalekz quits [Read error: Connection reset by peer] |
| 07:38:16 | <flashfire42> | lmfao |
| 07:39:49 | <@kiska> | "Do not fuck with a librarian, bitch we will catalogue you" - HCross |
| 07:39:50 | <@kiska> | :D |
| 07:57:59 | | Arcorann (Arcorann) joins |
| 08:06:37 | | Mineroboter joins |
| 08:08:47 | | Mineroboter quits [Client Quit] |
| 08:11:23 | <avoozl> | anyone here using youtube-dl to archive live recordings (like twitch)? It works well overall but every once in a while when the stream terminates my ffmpeg gets stuck on closing the .mp4.part file and the entire process hangs. I haven't found a good way to get out of that situation without corrupting the resulting file :/ |
| 08:13:16 | | BlueMaxima quits [Client Quit] |
| 08:15:15 | <@jrwr> | @mgrandi best part, they didn't use the word hacker |
| 08:25:35 | | daalekz (daalekz) joins |
| 08:26:54 | | daalekz quits [Client Quit] |
| 08:57:38 | | hooway joins |
| 09:23:02 | | Wayward- is now known as Wayward |
| 09:35:07 | <mgrandi> | @avoozl hey |
| 09:35:32 | <mgrandi> | A: you might want to report that to the youtube-dl github, sounds like a kinda nasty bug |
| 09:36:09 | <mgrandi> | B: I use a combination of twitch leecher https://github.com/Franiac/TwitchLeecher/releases/ and TCD https://github.com/PetterKraabol/Twitch-Chat-Downloader to download twitch videos and streams |
| 09:36:59 | <mgrandi> | youtube-dl claims it can get the chat (as like subtitles) but it isn't complete, and B: it tries to fix the resulting file and i have not seen any documentation on why that is needed? the resulting video (a bunch of .ts files merged together) seems fine |
| 09:41:59 | <mgrandi> | i personally use a combination of all 3: twitch leecher to download the video, youtube-dl to download the info-json / description files, and then TCD to download the chat |
| 09:43:39 | <Sylirana> | I've edited my user page on the Wiki and added information about our findings to https://wiki.archiveteam.org/index.php?title=CodePlex . Hopefully a Wiki moderator can review those soon. :-) |
| 09:46:25 | | nathan quits [Ping timeout: 252 seconds] |
| 09:48:25 | <Sylirana> | avoozl, if you can get a non-corrupted mp4 with the exact same parameters (preferrably from the same stream), you can use https://github.com/ponchio/untrunc to repair it. From what I understand, it copies the headers from the working file to the broken one. This has saved me countless streams that broke due to being unfinished. |
| 09:48:58 | <mgrandi> | youtube-dl is reencoding at least the audio part because it claims that twitch has broken audio |
| 09:49:07 | <mgrandi> | i dunno if ffmpeg does it in place or not |
| 09:49:42 | | LeGoupil joins |
| 09:50:49 | | nathan joins |
| 09:52:57 | <mgrandi> | i think you can also just tell it to NOT 'fix up' the file and then it should be fine |
| 09:52:58 | <Sylirana> | I haven't done it with youtube-dl, but other streams that I've been saving (also using ffmpeg), were fixable. So I think ffmpeg does it directly. The only thing that's done at the end, is adding the header. (That's also why you can't preview mp4s if a server doesn't support partial (skipping to the end) downloads. A bit unfortunate, but that's just how the format was designed. |
| 09:53:42 | <Sylirana> | An mp4 without its header will never play, afaik. |
| 09:54:03 | <mgrandi> | the format that twitch uses is just a bunch of "TS" files that literally gets concatenated together, i don't think it needs any headers or anything |
| 09:56:59 | <Sylirana> | I doubt those ever make it to the disk though. Not sure how that implementation for that Twitch downloader works, but telling ffmpeg to download and convert a series of .ts files (usually in a playlist or so), will result in it writing the new format on-the-fly. So if you request mp4, it will save mp4. One could of course use a different format which does not have that issue... |
| 09:58:02 | <mgrandi> | how twitch leecher works is it downloads the playlist, and then that lists all the TS files, then it downloads all the TS files |
| 09:58:24 | <mgrandi> | if you have the checkbox to disable encoding then it will be done there, but you can also have it encoded as MP4, and then it will call ffmpeg to convert it to MP4 |
| 09:58:34 | | manoel joins |
| 09:58:39 | <mgrandi> | however I just leave it as .TS as that is as cloose as we get to the source |
| 09:59:21 | <mgrandi> | also, i dunno who was the lead on the #scrapevoat project, but apparently we all forgot to try and use searchvoat.co to get anything remaining that we missed |
| 09:59:55 | <mgrandi> | if you could upload the full URL list @manoel we can probably have a look at it tomorrow, i think a lot of people are asleep atm |
| 10:00:21 | <manoel> | sure thing, doing it right now :) |
| 10:03:16 | <manoel> | so, I managed to get it on github zipped: https://github.com/manoelhortaribeiro/voat_missing_urls |
| 10:04:13 | <Sylirana> | Yes, keeping the individual TS files will probably get around that issue. I guess for a lot of people and mp4 with slightly lower quality (and thus lower filesize) and better compatibility withplayers, is preferrable, so that's why this is often the default for stream downloaders... |
| 10:04:40 | <mgrandi> | i have personally uploaded .TS files to the internet archive and it seems to convert them fine |
| 10:05:16 | <manoel> | (These were obtained by extensively crawling all ~4mi threads in https://searchvoat.co/, and then comparing the tbreads with what was already in WBM. What was NOT in WBM and IS in searchvoat.co is in the repo above!) |
| 10:05:17 | <Sylirana> | Considering that I've seen this exact issue on various sites though, makes me think that this might be an ffmpeg issue rather than an issue with the downloader calling ffmpeg... |
| 10:06:03 | <mgrandi> | https://archive.org/download/aoc_among_us_twitch_stream_2020_10_20 |
| 10:06:37 | <mgrandi> | i think .TS is technically MP4 or something, maybe in a slightly different format meant to be split up |
| 10:07:18 | <mgrandi> | thanks @manoel , i'll follow up tomorrow and make sure we decide whether we want to put this in AB or another DPoS project to finish it up |
| 10:07:22 | <Sylirana> | Oh I don't doubt that IA can handle them. I meant the general public, who might want to play those recordings directly on phones or TVs etc.... Although on phones, VLC should support .ts. |
| 10:07:43 | <mgrandi> | the IA creates a .mp4 version of the file for the 'web view' so at least that part is done |
| 10:08:14 | <mgrandi> | i follow the guidelines that the source file (or as close as you can get) is always preferable as then you can derive it (which is what IA does) to a more popular format |
| 10:08:17 | <Sylirana> | Yep, similar to GDrive or YT. |
| 10:10:03 | <Sylirana> | I definitely agree, source is always preferrable. But most of the people out there will continue to use mp4s for the time being, so it's worth having methods to fix them. :-) |
| 10:10:13 | <mgrandi> | So regardless of the program, twitch hosts .TS files, youtube-dl might concatenate them and then just rename it to be a .mp4 file , but it also by default tries to fix the AAC audio track for some reason |
| 10:10:40 | <mgrandi> | I tried looking in the source for ytdl to figure out why it is doing that but couldn't find it |
| 10:10:48 | <Sylirana> | It just concatenates them? o.O Is that even valid? |
| 10:11:39 | <mgrandi> | yep |
| 10:12:07 | <Sylirana> | What audio format do the source files have? Is that AAC? Or the the result AAC? Or both? |
| 10:12:29 | <Sylirana> | It might just change it due to better compatibility with the container. |
| 10:12:34 | <mgrandi> | https://en.wikipedia.org/wiki/MPEG_transport_stream |
| 10:13:29 | <Sylirana> | One can force a lot of technically unsupported audio or video formats into a container and depending on the player, they might even play without any hiccups... |
| 10:15:53 | <mgrandi> | https://usercontent.irccloud-cdn.com/file/ZgUUFYIB/image.png |
| 10:19:08 | <mgrandi> | https://www.irccloud.com/pastebin/qG1C7iHw/ |
| 10:19:35 | <mgrandi> | so its h264 as a 'transport stream' and then AAC audio, although VLC calls it ADTS |
| 10:19:44 | <Sylirana> | Hmm but just simply concatenating the files, won't result in a valid mp4, from my understanding. I guess you check what "file" says about it. I would guess they have differnet magic bytes. |
| 10:19:51 | <Sylirana> | https://superuser.com/questions/692990/use-ffmpeg-copy-codec-to-combine-ts-files-into-a-single-mp4/693009#693009 |
| 10:20:14 | <avoozl> | hmm I'll see what I can do, thanks for the tips :) |
| 10:20:18 | <Sylirana> | That's what that second step does, convert it to a proper mp4. |
| 10:20:33 | <mgrandi> | well that answer mentions that TS can just be concattenated, thats what the "transport stream" format is |
| 10:21:18 | <mgrandi> | but i think the 'transport stream' format is just...the same as MP4 but each 'TS' file just has its own headers and other stuff so it can be streamed without requiring the entire file |
| 10:22:35 | <Sylirana> | But it won't be a valid mp4. It still needs to be converted. Although that "conversion" might just be changing the magic bytes and header and leaving most of the actual audio and video streams intact (just a guess). |
| 10:22:49 | <mgrandi> | i mean, a mp4 is a contianer format, you are right it won't be a mp4 i guess |
| 10:23:04 | <mgrandi> | ffmpeg probably just copies the actual 'video/audio' data into an actual mp4 container |
| 10:23:36 | <Sylirana> | Yep that's how I think it works. |
| 10:23:37 | <mgrandi> | its like moving data that is inside a .rar file into a .zip file, its the same data, just different containers |
| 10:24:04 | <Sylirana> | And VLC is known for... Playing even very zombiefied files that won't play in any other player. :D |
| 10:24:08 | <mgrandi> | @avoozl let me know if you have any issues, i have done this lots |
| 10:24:31 | <mgrandi> | while true, .TS isn't that exotic of a format |
| 10:24:55 | <mgrandi> | part of the mpeg standard i believe |
| 10:25:40 | <Sylirana> | Well, I think .rar and .zip use different algorithms, so it would be more also converting the streams to other formats. |
| 10:26:17 | <mgrandi> | the container format is different, but the underlying data of whatever you are compressing (since video codecs are basically just compression algorithms) is different |
| 10:26:51 | <mgrandi> | at least with a high level, ffmpeg might be doing some magic on the actual video data, who knows, which is why i don't bother and just keep the .TS files for downloading twitch streams |
| 10:29:50 | <Sylirana> | I would say those have 3 layers, the container (zip), the compression algorithm (DEFLATE, etc.) and the files inside the archive itself. For archives you can change either the container (and keep the algorithm) or change the algorithm (and keep the container) or both. Similar to video containers and their formats, with the exception that archives are lossless and most video formats are lossy, even if just by a few bits. |
| 10:31:38 | <avoozl> | mgrandi: if there is any way to stop youtube-dl from hanging my ffmpeg that would be awesome.. now I need to manually kill it every once in a while because otherwise I miss parts of the stream when they start broadcasting again |
| 10:31:39 | <Sylirana> | Speaking of, last year I have tried to convert a set of PNGs to a video and then back to PNGs. The goal was to have a completely lossless conversion. But no matter what video format or container I've used, no matter which color space or encoder settings... It never resulted in the exact same image. The best I got was that a few pixels were off by 1 bit. |
| 10:31:51 | | Zerote quits [Read error: Connection reset by peer] |
| 10:32:23 | <mgrandi> | yeah, ffmpeg and most 'conversion' tools don't really ever consider a lossless conversion in terms of round trip |
| 10:32:43 | <mgrandi> | @avoozl , i might consider just...telling it to not do conversions for twitch streams? i don't know why its doing it for twitch streams |
| 10:33:25 | <mgrandi> | i think its --fixup "never" |
| 10:33:42 | <mgrandi> | might want to report it to youtube-dl though, i've never had that happen |
| 10:34:24 | <avoozl> | it mostly seems to happen for files over 10GB, but that may be just luck |
| 10:34:36 | <avoozl> | I do not have an easy way of reproducing it, since it is all coming from live streams |
| 10:35:13 | <mgrandi> | youtube-dl can download from a live stream? |
| 10:35:21 | <mgrandi> | or do you mean a VOD of a stream that is happening life |
| 10:35:23 | <mgrandi> | *live |
| 10:36:01 | <avoozl> | even with fixup never it seems to use ffmpeg to download the ts and concatenate them |
| 10:36:11 | <Sylirana> | I've personally had it happen for files with all kinds of sizes. From single-digit MBs (yes, those were still playable after fixing) to 10+ GB files. |
| 10:36:18 | <avoozl> | youtube-dl can download from a live twitch stream |
| 10:36:26 | <Sylirana> | And this was without youtube-dl, which really makes me think it's an ffmpeg issue. |
| 10:36:44 | <avoozl> | Sylirana: could very well be |
| 10:37:00 | <mgrandi> | i don't think its an "actual" live stream view |
| 10:37:12 | <mgrandi> | how twitch works is there is a vod that is like updated every 30 seconds with a new chunk |
| 10:37:44 | <mgrandi> | and youtube-dl doesn't use ffmpeg to download, unless you are using the download command option |
| 10:38:31 | <Sylirana> | Afaik it does for some sites. |
| 10:38:31 | <avoozl> | ok. all I'm doing is basically running youtube-dl https://www.twitch.tv/hroth_gar_gaming |
| 10:38:33 | <mgrandi> | and you can just wait till the stream is over and the same url will work, but the length won't change anymore |
| 10:38:54 | <Sylirana> | I've had it use ffmpeg with no extra parameters telling it to. |
| 10:39:23 | <Sylirana> | (That's how I've learned that ffmpeg is capable of downloading on its own.) |
| 10:39:43 | <mgrandi> | the default policy is " fix file if we can, warn otherwise |
| 10:40:00 | <mgrandi> | it might depend on the extractor its using |
| 10:40:15 | <mgrandi> | but ffmpeg isn't guaranteed to be installed so i dont know why it would require it, ffmpeg is pretty bad at downloading things |
| 10:41:12 | <mgrandi> | @avoozl so what i THINK that is doing is basically finding the latest 'vod' for that streamer and then downloading that, so that should be the equiv of passing in "https://www.twitch.tv/videos/910703831" |
| 10:41:36 | <mgrandi> | but i wonder if its having problems because the length of the VOD is changing every 30 or so seconds , since the person is streaming live |
| 10:43:25 | <avoozl> | Not all streamers have VOD on |
| 10:43:50 | <Sylirana> | Not sure if this applies to Twitch, but I've noticed it happening when the source stream is temporarily cut off. A bit like a buffer underrun. |
| 10:43:59 | <avoozl> | I'll experiment a bit with the youtube-dl options, see if things like --hls-prefer-native help. thanks |
| 10:44:07 | <mgrandi> | i didn't know you could take it off |
| 10:44:19 | <mgrandi> | like twitch streams are basically First in first out for a month, unless they mark them as saved |
| 10:45:17 | <Sylirana> | I know a bunch that have no saved videos, just visible while they are streaming. So if you want to save it, you have to do it live. :/ |
| 10:45:18 | <avoozl> | mgrandi: https://help.twitch.tv/s/article/video-on-demand?language=en_US shows a toggle |
| 10:45:28 | <avoozl> | Sylirana: exactly |
| 10:46:18 | <avoozl> | Sylirana: attaching gdb to ffmpeg shows a stack trace blocked a __GI__poll all the way down inside a avio_read inside an av_read_frame... |
| 10:46:55 | <mgrandi> | i only download vods of finished streams, i dunno, ffmpeg is pretty jank |
| 10:47:42 | <Sylirana> | So it might actually be due to it not receiving the next frame on time? |
| 10:47:53 | <Sylirana> | Or perhaps it's corrupted? |
| 10:48:30 | <Sylirana> | Whatever it is, ffmpeg should not get stuck and instead retry or skip the frame... |
| 10:48:41 | <mgrandi> | youtube-dl has different extractors, maybe its calling ffmpeg to download the raw stream like you watch on twitch live, but then ffmpeg is jank and probably has a bug and dies |
| 10:55:44 | | Zerote joins |
| 11:30:01 | | sliccricc quits [Remote host closed the connection] |
| 11:30:44 | | sliccricc (sliccricc) joins |
| 11:32:14 | | mazet quits [Ping timeout: 265 seconds] |
| 11:38:43 | | mazet joins |
| 11:40:55 | | SpikedColaWasTaken (SpikedCola) joins |
| 11:41:59 | | SpikedCola quits [Read error: Connection reset by peer] |
| 12:09:27 | <mgrandi> | as a follow up, we are discussing the extra urls in #scrapevoat, if anyone left and wants to rejoin |
| 12:21:29 | | Ryz quits [Remote host closed the connection] |
| 12:21:40 | | Ryz (Ryz) joins |
| 12:44:24 | | thuban quits [Remote host closed the connection] |
| 12:55:16 | | ragu joins |
| 13:23:53 | <@EggplantN> | Okie so its GFW time of the year |
| 13:24:10 | <@EggplantN> | Haven't had the GFW spring up this bad since last "June 4th incident" |
| 13:27:44 | | manoel leaves |
| 13:39:21 | | lunik1 quits [Ping timeout: 265 seconds] |
| 13:42:18 | | ragu is now authenticated as ragu |
| 13:48:19 | | daalekz (daalekz) joins |
| 13:50:32 | | Guest69 joins |
| 13:55:40 | | Guest69 is now known as bithippo |
| 13:57:26 | | daalekz quits [Client Quit] |
| 13:59:46 | | britmob2 quits [Client Quit] |
| 14:00:00 | | britmob2 joins |
| 14:00:14 | | benjinsmith joins |
| 14:02:10 | | benjins quits [Ping timeout: 252 seconds] |
| 14:02:18 | | lunik1 joins |
| 14:08:21 | | jtt123 quits [Ping timeout: 265 seconds] |
| 14:08:50 | | balrog quits [Ping timeout: 265 seconds] |
| 14:09:07 | | balrog joins |
| 14:09:08 | | balrog is now authenticated as balrog |
| 14:09:41 | | jtt123 (jtt123) joins |
| 14:19:21 | | ragu_ joins |
| 14:23:20 | | ragu quits [Ping timeout: 265 seconds] |
| 14:27:26 | | ragu_ quits [Read error: Connection reset by peer] |
| 14:27:53 | | bithippo quits [Client Quit] |
| 14:34:59 | | benjinsmith is now known as benjins |
| 14:35:03 | | benjins is now authenticated as benjins |
| 14:37:26 | | sliccricc quits [Remote host closed the connection] |
| 14:37:40 | | sliccricc (sliccricc) joins |
| 14:48:28 | | sliccricc quits [Remote host closed the connection] |
| 14:49:06 | | sliccricc (sliccricc) joins |
| 14:52:13 | | thuban joins |
| 14:53:47 | | Arcorann quits [Ping timeout: 265 seconds] |
| 15:13:45 | | manoel joins |
| 15:24:14 | | lunik1 quits [Ping timeout: 265 seconds] |
| 15:33:56 | | sliccricc quits [Remote host closed the connection] |
| 15:34:42 | | sliccricc (sliccricc) joins |
| 15:44:14 | | sec^nd quits [Remote host closed the connection] |
| 15:44:43 | | sec^nd (second) joins |
| 16:00:11 | | godane1 joins |
| 16:01:47 | | godane quits [Ping timeout: 244 seconds] |
| 16:03:00 | <purplebot> | CodePlex edited by Sylirana (+4315, Added information about site structure …) just now -- https://www.archiveteam.org/?diff=46317&oldid=46315 |
| 16:05:37 | | lunik1 joins |
| 16:07:27 | | Ryz quits [Remote host closed the connection] |
| 16:08:18 | | Ryz (Ryz) joins |
| 16:12:55 | | Mateon2 joins |
| 16:14:43 | | Mateon1 quits [Ping timeout: 252 seconds] |
| 16:14:43 | | Mateon2 is now known as Mateon1 |
| 16:23:39 | | icculus_ joins |
| 16:26:04 | | icculus quits [Ping timeout: 244 seconds] |
| 16:26:54 | <@JAA> | avoozl: FWIW, I've had my share of issues with live streams and ffmpeg/youtube-dl and am using streamlink now. Doesn't support as many sites but is much more stable in my experience. |
| 16:40:25 | <avoozl> | Thanks, I'll have a look |
| 16:41:10 | <avoozl> | Looks good, I'll test |
| 16:52:56 | | benjins quits [Remote host closed the connection] |
| 16:53:22 | | benjins joins |
| 16:53:33 | | benjins is now authenticated as benjins |
| 17:05:53 | <avoozl> | JAA: hmm, it doesn't seem to work for current live streams, apart from that it looks nice |
| 17:06:21 | <@JAA> | avoozl: It definitely does, at least for Twitch. I use it regularly. |
| 17:06:33 | <avoozl> | Huh |
| 17:07:04 | <avoozl> | I'll need to read a bit more.. I was just doing error: No playable streams found on this URL: https://www.twitch.tv/gmhikaru |
| 17:07:46 | <@JAA> | Works fine for me. |
| 17:08:53 | <avoozl> | ok then my package is broken I guess.. I'll have a closer look |
| 17:09:14 | <avoozl> | [18:08:10,990][cli][debug] Python: 3.8.5 |
| 17:09:16 | <avoozl> | [18:08:10,990][cli][debug] Streamlink: 1.7.0 |
| 17:09:31 | <@JAA> | Update streamlink then. Current version is 2.0.0. |
| 17:10:09 | <avoozl> | doh. I was using their recommended ubuntu package repo. I'll use pip now instead :) |
| 17:10:13 | <@JAA> | Twitch changes stuff from time to time that breaks it. I remember it breaking a couple months ago. |
| 17:10:35 | <@JAA> | 1.7.0 is from October, so *ANCIENT*. :-P |
| 17:10:53 | <avoozl> | haha yeah, this repo looks stale |
| 17:11:03 | <avoozl> | I'll get things going with a virtualenv and pip |
| 17:11:19 | <avoozl> | much better |
| 17:12:03 | <@JAA> | Looks like that PPA they recommend for Ubuntu only has 2.0.0 for groovy, and it's only updated once in a blue moon. |
| 17:12:44 | <@JAA> | Debian has 2.0.0 even in stable-backports, by the way. :-) |
| 17:18:52 | <avoozl> | JAA: would something like this work, it is a bit hard to troubleshoot as tar is exiting and not showing the output... streamlink -l trace --fifo --player=tar -a "cvf out.tar {playerinput}" https://www.twitch.tv/lec best |
| 17:19:18 | <avoozl> | (probably a better way to do that) |
| 17:20:32 | <avoozl> | oh I forgot, tar won't read fifos |
| 17:22:29 | <avoozl> | ok this will work, thanks :) |
| 17:23:13 | <avoozl> | I'm just doing a raw dump for later processing. streamlink -l trace --fifo --player=dd -a "if={playerinput} of=out.raw" |
| 17:24:27 | <avoozl> | haha I'm an idiot. it has a -o option |
| 17:24:56 | | avoozl will stay of heavy machinery for the rest of the day |
| 17:25:31 | <@JAA> | lol yeah, I was about to say. I'm not sure if streamlink does any transcoding, but I believe not. |
| 17:26:01 | <avoozl> | It seems to just fetch and write, the resulting file is a transport stream, which is perfect for what I need |
| 17:26:25 | <avoozl> | I can always run this through avconv, ffmpeg, mencoder, or whatever is needed.. at least now when one part fails the stream is still saved |
| 17:26:31 | <@JAA> | Yeah, it just uses `ffmpeg ... -c:v copy -c:a copy ...` by default at least. |
| 17:34:33 | | rsn_ joins |
| 17:36:40 | | rsn quits [Ping timeout: 252 seconds] |
| 17:43:13 | | laapan joins |
| 18:11:28 | | godane1 quits [Ping timeout: 265 seconds] |
| 18:12:51 | | icculus_ is now known as icculus |
| 18:14:13 | <icculus> | mgrandi, JAA: looks like the bugzilla.libsdl.org archiving is done; is it cool for me to turn on the redirects to GitHub now, or do you need to grab anything else from there? |
| 18:14:55 | <icculus> | I see wiki.libsdl.org is still archiving, so I'm leaving that online to finish, in any case. |
| 18:20:11 | <@JAA> | icculus: Yep, Bugzilla should be fully covered now. :-) |
| 18:50:52 | | godane (godane) joins |
| 18:52:46 | | godane1 joins |
| 18:55:19 | | godane quits [Ping timeout: 252 seconds] |
| 18:56:51 | | LeGoupil quits [Client Quit] |
| 19:38:19 | | godane2 joins |
| 19:40:51 | | godane1 quits [Ping timeout: 244 seconds] |
| 19:42:58 | | DogsRNice (Webuser299) joins |
| 19:44:27 | <Barto> | damn, icculus is around town. I thought you'd hang around #ioquake3 every earthquakes or so. |
| 19:45:23 | <icculus> | Barto: I'm famously difficult to contact. :) |
| 19:46:06 | <Barto> | icculus: well, just a big thanks for libsdl, it's a really nice lib. |
| 19:46:26 | <icculus> | :) |
| 20:04:49 | | godane1 joins |
| 20:07:22 | | godane2 quits [Ping timeout: 252 seconds] |
| 20:12:37 | <@arkiver> | icculus: thank you for working with us on making sure a copy is archived for the Wayback Machine! |
| 20:26:31 | <Barto> | silly question, but how about having a full dump of icculus.org. It hasn't moved much recently. Maybe someone will restart fatelf dev, i don't know :D |
| 20:33:17 | <mgrandi> | If archivebot cannot archive that then we are failing at writing software |
| 20:33:52 | <mgrandi> | Although it probably doesn't know how to handle the source code repos without pain |
| 20:35:21 | <@JAA> | Yeah, archiving source code via web interfaces is almost never a good idea. |
| 21:10:14 | | manoel quits [Ping timeout: 244 seconds] |
| 21:21:27 | | rsn joins |
| 21:24:11 | | rsn_ quits [Ping timeout: 244 seconds] |
| 21:29:39 | | teej joins |
| 21:49:31 | <Sylirana> | Just wondering, is there something one can do to not have to wait for wiki edits to be reviewed? (Asking because I'm updating the stats on the CodePlex page from time to time.) |
| 22:00:59 | | BlueMaxima joins |
| 22:01:19 | <thuban> | Sylirana: yes, you need to be added to the automoderated group. i believe JAA usually does this. |
| 22:17:00 | <purplebot> | CodePlex edited by Sylirana (+0, Fixed wording and updated stats.) just now -- https://www.archiveteam.org/?diff=46318&oldid=46317 |