| 00:03:20 | <Suika> | Hm, more or less. There is patreon_data, patreon_inline, shared_data +fantia_data |
| 00:04:25 | <Suika> | patreon_data is basically `[Patreon ID]/[Post ID]/[data or another folder]` |
| 00:04:55 | <Suika> | patreon_inline and shared_data are loose data. |
| 00:05:49 | <Suika> | I think the patreon_inline uses [Post ID] for it's folder naming, let me check |
| 00:07:36 | <Suika> | And shared_data is user uploaded content. |
| 00:08:14 | | Terbium quits [Quit: No Ping reply in 180 seconds.] |
| 00:08:23 | <Suika> | Everything can be grouped by simply processing the HTML files to extract the links and then move the data in it's own contianers. |
| 00:09:26 | | Terbium joins |
| 00:10:06 | <Suika> | If you are interested, here are the html/json files. https://mega.nz/folder/Ik5yxIDR#Ah-5yJwBNd2ae_uBuCJuUA |
| 00:10:35 | <Suika> | Whatever links to local content can be found in every html file, will correspong to the location on disk. |
| 00:17:00 | <mgrandi> | I'll take a look |
| 00:21:05 | <Suika> | Mind you, json was always a broken mess. |
| 00:23:15 | <mgrandi> | Is that from an API on the site or something m |
| 00:23:43 | | @dxrt quits [Quit: ZNC - http://znc.sourceforge.net] |
| 00:25:03 | <Suika> | Json was server by the site and used ages ago for something internal. But never removed. It has broken links. The rest of the data is... fine I think. |
| 00:25:12 | | dxrt joins |
| 00:25:14 | | dxrt is now authenticated as dxrt |
| 00:25:14 | | dxrt quits [Changing host] |
| 00:25:14 | | dxrt (dxrt) joins |
| 00:25:14 | | @ChanServ sets mode: +o dxrt |
| 00:30:28 | <OrIdow6> | So I'll agree that, if it's found permissible, the best way to do it would be to split it up one user per item (unless there are 100 million users or something), then split the file such that the original is the union of all the per-user items (and perhaps some miscellaneous data) |
| 00:30:52 | | Jake5 (Jake) joins |
| 00:32:32 | | Jake quits [Ping timeout: 252 seconds] |
| 00:32:32 | | Jake5 is now known as Jake |
| 00:32:34 | <OrIdow6> | But that's dependent on that condition, and (apparently unlike some of the other people here), I don't seem to have too good of an understanding of what this data is |
| 00:32:42 | | @arkiver quits [Excess Flood] |
| 00:33:28 | <Suika> | Basically you pay on patreon for whatever you use, then you enter you session id into the site and it scrapes the content you have access to. |
| 00:33:29 | | arkiver (arkiver) joins |
| 00:33:29 | | @ChanServ sets mode: +o arkiver |
| 00:33:44 | <OrIdow6> | But I have other things to worry about, anyhow (#chromeweblore now has ops and is open for business) |
| 00:39:32 | <mgrandi> | Yeah, so it basically is like having one person pay for netflix, giving this site your session cookie and then the site scrapes and hosts for free all of netflix's content |
| 00:45:36 | <@JAA> | SketchTheCow: Yeah, we've been pushing a lot of stuff since FalconK's recent return and the addition of his pipeline. I'll try to divert some of the data directly to S3. |
| 00:55:54 | <SketchTheCow> | It's too much. It just is. |
| 00:56:02 | <SketchTheCow> | At least make him space it out |
| 00:56:21 | <SketchTheCow> | FOS does a lot of things. It should really stop being a pinchpoint |
| 00:57:31 | <@JAA> | Aye |
| 01:18:18 | | kyndigs joins |
| 01:21:33 | <kyndigs> | I archived entire iTunesU as I read rumors it was being closed, not sure the best to way preserve it online now, it needs a bit of organizing first, I grabbed as much meta data as possible for each institution and course, including artwork etc, any suggestions? |
| 01:22:26 | <kyndigs> | I was thinking archive.org, per institution/course uploads |
| 01:32:14 | <@JAA> | kyndigs: Just to clarify, it's not a rumour, iTunes U is indeed shutting down at the end of 2021: https://support.apple.com/guide/itunesu/welcome/web |
| 01:33:29 | <kyndigs> | looks like i got it just in time then |
| 01:33:32 | <kyndigs> | :| |
| 01:33:48 | <@JAA> | 2021 |
| 01:33:53 | <@JAA> | In a bit over a year. |
| 01:33:54 | <kyndigs> | is anyone else working on archiving it? |
| 01:34:38 | <kyndigs> | it was a bit brute force as the api are not great |
| 01:37:55 | <kyndigs> | The only thing is the organisation is not the best as it is in iTunesU app, as you essentially need to enroll the course then you can spoof the cookie to download the course structure data, but for 1000s of courses it was not feasible for me to do alone. |
| 01:38:30 | <kyndigs> | so only have the itunesu store data instead |
| 01:42:32 | <kyndigs> | I think I should share all the meta information with download links etc included and get suggestions on best way to organize, or can have a better effort to archive it better than I did. |
| 01:56:56 | | purplebot quits [Remote host closed the connection] |
| 01:57:30 | | purplebot joins |
| 02:02:54 | <Suika> | Either way, I'm off to sleep. I'm on a bouncer, so poke me if you decided something. I'll read it the coming days- |
| 02:08:20 | <@arkiver> | EggplantN: anything on fixing the discord thing? |
| 02:37:06 | <tech234a> | List of all iTunes versions since 7.3.0 which added support for the original iPhone: https://www.theiphonewiki.com/wiki/ITunes. Not sure if iTunes 12.6.5.3 would be helpful for archiving iTunes U courses (it was the last version before they removed a bunch of features, perhaps iTunes U was one of the removed features). |
| 02:42:24 | <kyndigs> | I already found how to archive it, and done so, I have all the meta with the direct links to download the media, if someone else wants to archive it with better organization, would be a matter of reading the meta and links and using that data to organize a bit better. |
| 02:42:37 | <kyndigs> | Institution like this |
| 02:42:39 | <kyndigs> | https://i.imgur.com/ySOCBaS.png |
| 02:42:47 | <kyndigs> | Then course like this |
| 02:42:48 | <kyndigs> | https://i.imgur.com/aBdiYe3.png |
| 02:44:51 | <kyndigs> | meta for a course: https://pastebin.com/raw/1cBNHWzn |
| 02:58:18 | <purplebot> | File:Nagi-logo.gif uploaded by Arkiver (+0) just now -- https://www.archiveteam.org/?diff=45765&oldid=0 |
| 03:02:10 | <OrIdow6> | kyndigs: By doing this a year in advance, you are about 11 months ahead of when ArchiveTeam will do it |
| 03:02:20 | <kyndigs> | hehe |
| 03:02:40 | <kyndigs> | headstart so no last minute panic :D |
| 03:04:11 | <@kiska> | We are the masters of doing last minute projects :P |
| 03:04:37 | <OrIdow6> | Anyhow, yeah, I'd say organize by course |
| 03:04:43 | <kyndigs> | But i will point out there is some 3rd party hosted content on itunesu |
| 03:04:50 | <OrIdow6> | This is actually contrary to what ArchiveTeam did with the UC Berkeley videos |
| 03:04:51 | <OrIdow6> | https://archive.org/details/ucberkeley-webcast |
| 03:04:55 | <kyndigs> | which is lost, so my worry was waiting longer might lose more |
| 03:05:02 | <OrIdow6> | Where there was 1 video per item, and an IA collection for each course |
| 03:05:41 | <kyndigs> | Was that manually organized do you know? |
| 03:06:09 | <OrIdow6> | What do you mean, manually? |
| 03:06:16 | <kyndigs> | As to get the course structure you need to subscribe to it and use the auth cookie to grab the structure |
| 03:06:44 | <kyndigs> | Allowing you to identify "Lecture 1" etc |
| 03:07:23 | <OrIdow6> | https://www.archiveteam.org/index.php?title=UC_Berkeley_Course_Captures says that there was easily accessible structure |
| 03:07:45 | <OrIdow6> | However, there was presumably some manual effort to create/coordinate the creation of the collections on IA |
| 03:07:50 | <OrIdow6> | As those can't be made by normal users |
| 03:08:20 | <OrIdow6> | (Sourced from Youtube playlists, apparently) |
| 03:10:01 | <OrIdow6> | Oh, rereading, I see what your problem is |
| 03:10:02 | <kyndigs> | yes those are the playlist, those are easy to get i was talking about this |
| 03:10:03 | <kyndigs> | https://cdn.cultofmac.com/wp-content/uploads/2014/07/itunesu.jpg |
| 03:10:37 | <kyndigs> | you see how the app shows it, organized its purely meta data on how to display but would be quite useful in organizing |
| 03:10:57 | <kyndigs> | we can get that info, but its a bit manual and tedious |
| 03:11:45 | <kyndigs> | "https://p2-u.itunes.apple.com/WebObjects/LZStudent.woa/ra/1/courses/{COURSE_ID}?pt=v" |
| 03:12:03 | <kyndigs> | and you can pass the authentication cookie in the request |
| 03:12:25 | <kyndigs> | but you need to subscribe to the course first to get that cookie which for 1000s of courses is a pain! |
| 03:14:48 | <OrIdow6> | I'm not familiar with Mac & all that too much, so if there is some way to automate it, I wouldn't know it |
| 03:15:03 | <OrIdow6> | And if there's no easy solution, there's no easy solution |
| 03:15:33 | <kyndigs> | yeh i even contemplated jailbreaking the ipad and writing something to do it but no idea how to do IOS stuff either! |
| 03:16:40 | <kyndigs> | worst comes to worst ill subscribe all the courses then get the data, main thing was the materials first which we got. |
| 03:17:09 | <OrIdow6> | Yeah, it sounds like the recordings are the most important thing |
| 03:17:32 | <OrIdow6> | If nothing else, you can piece them together after |
| 03:17:44 | <OrIdow6> | "You" not being you specifically |
| 03:17:52 | <kyndigs> | yeh what ill do now is upload all the meta and people can have a look also |
| 03:18:18 | <kyndigs> | the upload of materials will take some time for sure |
| 03:18:52 | <kyndigs> | kind of regretted starting it half way through :D |
| 03:19:30 | <OrIdow6> | So as it stands now, you just have a soup of videos, with nothing to indicate automatically that two are from the same course or school? |
| 03:19:49 | <kyndigs> | i can organize files to institution and course |
| 03:19:54 | <OrIdow6> | Oh, good |
| 03:19:55 | <kyndigs> | but not deeper |
| 03:20:05 | <kyndigs> | like lecture 1, assignment, quiz |
| 03:20:19 | <OrIdow6> | Well, it means that if or when you have all the metadata, you won't need to reupload everything into a different structure |
| 03:20:50 | <kyndigs> | exactly with the meta it can be used to organize how one likes |
| 03:21:45 | <kyndigs> | still a lot of meta atm but just not the app specific structure of the course which is much deeper and organized |
| 03:22:31 | <kyndigs> | i contacted IA also for some guidance |
| 03:22:52 | <kyndigs> | as i dont want to upload just a big dump of files :D |
| 03:23:24 | <OrIdow6> | If you upload more than 50? (I think that's the threshold) items of the same theme, you can email them and they'll put them into a collection for you |
| 03:24:04 | <OrIdow6> | But I suppose you've already done that, in a way, anyhow |
| 03:24:55 | <OrIdow6> | But again, one course per item is the way I would do it, but evidently it's been done differently in the past |
| 03:27:04 | <kyndigs> | yes makes most sense but will see what they say, "send us hdd we will do" ideally :D |
| 03:36:13 | <OrIdow6> | They've done it in the past, though I don't know how common it is |
| 03:36:26 | <OrIdow6> | But good luck with it |
| 03:57:46 | <mgrandi> | Any work you do should be added to the wiki so even if you aren't able to finish it, others can take it up. If it's not already added to deathwatch, you should do that too with a link to the wiki page |
| 03:58:18 | <@JAA> | It's on Deathwatch. |
| 04:18:44 | | benjins quits [Read error: Connection reset by peer] |
| 04:19:34 | | godane2 joins |
| 04:20:16 | | benjins joins |
| 04:21:53 | | godane1 quits [Ping timeout: 252 seconds] |
| 04:49:16 | | qw3rty__ joins |
| 04:53:14 | | qw3rty_ quits [Ping timeout: 252 seconds] |
| 05:51:01 | <kyndigs> | will do |
| 05:51:51 | | atphoenix is now authenticated as atphoenix |
| 06:39:06 | | Ryz quits [Remote host closed the connection] |
| 06:40:29 | | Ryz (Ryz) joins |
| 07:23:04 | | lukash quits [Ping timeout: 240 seconds] |
| 07:25:04 | | lukash (lukash) joins |
| 07:51:09 | | VADemon joins |
| 08:03:03 | | godane1 joins |
| 08:05:10 | | Doranwen quits [Ping timeout: 252 seconds] |
| 08:05:32 | | godane2 quits [Ping timeout: 252 seconds] |
| 08:06:24 | | Doranwen joins |
| 08:08:45 | | Doranwen is now authenticated as Doranwen |
| 08:54:43 | <FalconK> | SketchTheCow: sorry for filling your disk lol |
| 08:55:17 | <FalconK> | I have enough disk to hang onto WARCs for a while if needs must; I could also put them directly into IAS3 again if that's desirable |
| 08:56:23 | <FalconK> | ... I'm glad I'm still here. I just had the absolute worst flight round trip from seattle to portland. |
| 08:56:45 | | FalconK breathes |
| 09:00:18 | | britmob2 quits [Client Quit] |
| 09:38:43 | <fionera> | Fusl: how far are you from dusseldorf? |
| 09:38:43 | <fionera> | I feel offended |
| 09:40:08 | | Zopolis4 joins |
| 09:43:18 | <fionera> | even tho there is no reason. I just need smth to be angery |
| 09:44:03 | <@Fusl_> | fionera? |
| 09:45:04 | <fionera> | Just ignore me im bored |
| 09:46:28 | <@Fusl_> | uhhh, okayy? |
| 10:07:02 | | Arcorann quits [Read error: Connection reset by peer] |
| 10:16:28 | | kyndigs quits [Remote host closed the connection] |
| 10:34:56 | | Arcorann (Arcorann) joins |
| 10:36:02 | | Arcorann quits [Read error: Connection reset by peer] |
| 10:39:45 | | @arkiver quits [Excess Flood] |
| 10:40:41 | | arkiver (arkiver) joins |
| 10:40:41 | | @ChanServ sets mode: +o arkiver |
| 10:51:11 | | Arcorann (Arcorann) joins |
| 11:26:04 | | hooway joins |
| 11:30:16 | | VADemon quits [Read error: Connection reset by peer] |
| 12:15:46 | | benjins is now authenticated as benjins |
| 12:57:15 | | kyndigs joins |
| 13:51:17 | | britmob2 joins |
| 14:00:13 | | mutantmonkey quits [Remote host closed the connection] |
| 14:00:35 | | mutantmonkey (mutantmonkey) joins |
| 14:20:16 | | Doranwen quits [Ping timeout: 252 seconds] |
| 14:20:47 | | Doranwen joins |
| 14:25:14 | | Arcorann quits [Ping timeout: 252 seconds] |
| 14:32:44 | | Doranwen quits [Ping timeout: 252 seconds] |
| 14:32:57 | | Doran joins |
| 14:46:05 | | mutantmonkey quits [Remote host closed the connection] |
| 14:46:24 | | mutantmonkey (mutantmonkey) joins |
| 14:52:24 | | mutantmonkey quits [Ping timeout: 240 seconds] |
| 15:05:36 | | mutantmonkey (mutantmonkey) joins |
| 16:54:01 | | marked19578 joins |
| 16:55:56 | | marked1957 quits [Ping timeout: 252 seconds] |
| 16:55:56 | | marked19578 is now known as marked1957 |
| 17:02:03 | <OrIdow6> | arkiver: Anything on the .eu domains? |
| 17:07:01 | | Lord_Nightmare quits [Quit: ZNC - http://znc.in] |
| 17:09:51 | | Lord_Nightmare (Lord_Nightmare) joins |
| 17:21:32 | <SketchTheCow> | It's fine, just pace it. |
| 17:21:47 | <SketchTheCow> | Do a... I don't know, 1 minute sleep between WARCs |
| 17:55:40 | | Doran is now known as Doranwen |
| 17:55:42 | | Doranwen is now authenticated as Doranwen |
| 18:54:54 | | Doranwen quits [Ping timeout: 252 seconds] |
| 18:55:06 | | Doran joins |
| 18:59:09 | | Doran quits [Remote host closed the connection] |
| 18:59:41 | | Doran joins |
| 19:00:16 | | Doran is now known as Doranwen |
| 19:00:19 | | Doranwen is now authenticated as Doranwen |
| 19:12:30 | | Doranwen quits [Ping timeout: 252 seconds] |
| 19:12:36 | | Doranwen joins |
| 19:14:20 | | Doranwen is now authenticated as Doranwen |
| 19:15:58 | <@arkiver> | OrIdow6: not yet |
| 19:19:43 | | VerifiedJ quits [Quit: The Lounge - https://thelounge.chat] |
| 19:44:48 | | VerifiedJ (VerifiedJ) joins |
| 19:50:39 | | Zopolis4 quits [Ping timeout: 244 seconds] |
| 20:58:24 | | @arkiver quits [Excess Flood] |
| 20:59:19 | | arkiver (arkiver) joins |
| 20:59:19 | | @ChanServ sets mode: +o arkiver |
| 21:30:31 | <SketchTheCow> | Hey, fucksticks. |
| 21:31:03 | <EggplantN> | <Northern#2000> Ah shit Jason is losing it in lockdown |
| 21:31:07 | <SketchTheCow> | So, look. The new IRC is all nice and secure and all that but in a shocking twist, the guy who refuses to run https on his website keeps getting shit for his IRC client not running SASL or whatever |
| 21:31:29 | <SketchTheCow> | Now, this is fine. The only issue is Archivebot gives me guff when I make requests. |
| 21:31:32 | <SketchTheCow> | That's fine. |
| 21:31:35 | <SketchTheCow> | As well. |
| 21:31:52 | <SketchTheCow> | What I'd like to know is who is in that channel all the time, or here, who can go over to archivebot and do a save when I ask. |
| 21:32:06 | <SketchTheCow> | I ask, if you look back, every 3-10 days. Mostly because I noticed something or I was asked. |
| 21:32:09 | <SketchTheCow> | Speak up |
| 21:32:21 | <SketchTheCow> | Or I can make a phrase here that does it. |
| 21:32:47 | <SketchTheCow> | Otherwise, I'm just fine with things. I'm busy elsewhere with shit none of you can do because *makes jazz hands* and you wouldn't understand |
| 21:33:14 | <SketchTheCow> | I was trapped near the inner circle of thought. |
| 21:37:28 | <@JAA> | Ryz and/or I are usually around, so if you ping us in #archivebot, that'd probably work. Or just mention it here (so it doesn't get buried in the bot traffic) and whoever sees it first can throw it in. |
| 21:39:31 | <Ryz> | Loot! |
| 21:47:55 | <SketchTheCow> | OK, that works. |
| 21:48:10 | <SketchTheCow> | I'm just not on IRC much and my communications are mostly elsewhere. |
| 21:48:35 | <SketchTheCow> | I stay on here to help with major project issues or if someone needs a completely redundant old guy to back up their ridiculous petty shitfight |
| 21:49:12 | <SketchTheCow> | And that is fine. But Archivebot wants specific things and I'm not interested. |
| 21:52:34 | <SketchTheCow> | SPEAKING of raining on everyone's parade, I've been informed FOS has to go down for a hardware upgrade on the 24th |
| 21:54:48 | <@JAA> | Oof |
| 21:55:03 | <SketchTheCow> | Hopefully just a few hours, we'll see |
| 21:55:18 | <SketchTheCow> | I told you! FOS as a pinchpoint is bad! |
| 21:55:24 | <@JAA> | I know! |
| 21:55:28 | <SketchTheCow> | It's not doing anything special, either |
| 21:55:41 | <SketchTheCow> | I mean, in theory, I can teamarchive1 do some of this work |
| 21:56:05 | <SketchTheCow> | I've never, actually, you know, actually evaluated the pipeline I use |
| 21:56:15 | <SketchTheCow> | I bet I could do that experiment. |
| 21:56:28 | <SketchTheCow> | Maybe it'll work better. It has less space than FOS |
| 22:22:34 | <OrIdow6> | (From #archiveteam) |
| 22:23:44 | <OrIdow6> | This is the sort of non-game Flash site that I think is going to be somewhat overlooked |
| 22:25:27 | <@JAA> | Closure announcement is here, by the way: https://forum.lacartoonerie.com/showthread.php?tid=3734&pid=151087#pid151087 |
| 22:29:07 | <SketchTheCow> | I just checked. teamarchive1 has about 3 terabytes. |
| 22:29:25 | <SketchTheCow> | I would be able to get an archivebot pipline to it, but that's not a massive amount of headroom |
| 22:32:51 | <@JAA> | Well, we're more limited by the upload to IA than the pipelines at the moment. |
| 22:33:02 | <@JAA> | Or do you mean for uploads? |
| 22:34:47 | | godane1 quits [Read error: Connection reset by peer] |
| 22:35:26 | | godane1 joins |
| 22:35:32 | | User joins |
| 22:35:56 | | hooway quits [Read error: Connection reset by peer] |
| 22:35:57 | | TheLovinator quits [Quit: Ping timeout (120 seconds)] |
| 22:36:11 | | TheLovinator joins |
| 22:36:28 | | benjinsmith joins |
| 22:37:03 | <SketchTheCow> | So two possibilities. |
| 22:37:25 | <SketchTheCow> | TEAMARCHIVE1 just got mega upgraded, so maybe the network performs better. |
| 22:37:44 | | benjins quits [Ping timeout: 240 seconds] |
| 22:39:51 | | benjinsmith is now known as benjins |
| 22:40:03 | | benjins is now authenticated as benjins |
| 22:42:51 | | @arkiver quits [Excess Flood] |
| 22:43:40 | | arkiver (arkiver) joins |
| 22:43:40 | | @ChanServ sets mode: +o arkiver |
| 22:46:31 | <SketchTheCow> | I am on Off The Hook tonight, but I'll see if I can mirror the archivebot pipeline on teamarchive1, and then we can aim some archivebot output at it. |
| 22:46:44 | <SketchTheCow> | If this works, we can do a swap during crushtimes. |
| 22:46:54 | <SketchTheCow> | Like when FOS has to go down |
| 22:47:44 | <@JAA> | Sounds good. |
| 22:58:41 | <SketchTheCow> | Maybe later tonight we can try it. |
| 23:01:56 | <SketchTheCow> | Actually, let's try it now. Is it hard to swap over? |
| 23:02:38 | <@kiska> | SketchTheCow: Could we get FalconK added to the archivebot collection for uploads? |
| 23:03:18 | <@JAA> | Every pipeline uploader needs to be swapped individually, so kind of but not very. |
| 23:06:24 | <SketchTheCow> | Well, just do one pipeline, we'll see how teamarchive1 keeps up |
| 23:06:43 | <@kiska> | JAA: Could do the instagram pipeline |
| 23:06:44 | <SketchTheCow> | We'll know pretty quickly if this is a good idea. |
| 23:07:01 | <SketchTheCow> | Whether it's a replacement or a spare |
| 23:07:04 | <@JAA> | kiska: Nah, I'd rather not wait a half century before the first file is uploaded. :-P |
| 23:08:06 | <@kiska> | :P |
| 23:08:11 | <@kiska> | Or kiskaJD |
| 23:08:15 | <@kiska> | kiskaJDC* |
| 23:11:40 | | Larsenv quits [Quit: ZNC 1.8.0 - https://znc.in] |
| 23:14:46 | | Arcorann (Arcorann) joins |
| 23:15:36 | | Larsenv (Larsenv) joins |
| 23:51:41 | <SketchTheCow> | Downtime for FOS moved to December 1st. |
| 23:51:52 | <SketchTheCow> | And waiting on network guy to make the backup pipeline work |