| 00:02:06 | <@JAA> | (That's French for PB/TB, for the unaware.) |
| 00:04:06 | <nico_32> | yes octets = bytes |
| 00:06:35 | <@JAA> | I was never worried about the source code. Such a popular project is bound to have hundreds of up-to-date copies everywhere at any time. |
| 00:06:43 | <@JAA> | The issues, pull requests, etc. though... |
| 00:07:38 | <nico_32> | also git lfs |
| 00:07:49 | <nico_32> | i am not sure everyone have it |
| 00:08:09 | <@JAA> | Yeah, quite rare I believe. |
| 02:11:53 | <thuban> | JAA: did you get the google-cache-over-multiple-ab-pipelines working? (is it worth setting up as a warrior job?) i agree, those issues are important |
| 02:12:24 | <@JAA> | thuban: Nope, only two jobs, so ~12 days each expected. |
| 02:12:42 | <thuban> | (and i'm not just saying that because i've had a perfectly good pr languishing open for years) |
| 02:13:50 | <@JAA> | Worth mentioning that the Google Cache certainly won't have all comments on long discussions because GitHub hides them behind a button. I strongly doubt Google fetches that. |
| 02:19:07 | | hurricos joins |
| 02:19:54 | <hurricos> | Hi folks. Is anyone working on recovering the youtube-dl issue tracker? |
| 02:20:11 | <thuban> | yes |
| 02:20:24 | <thuban> | but it's likely not to be complete |
| 02:21:02 | <hurricos> | Is there anything I can do to help? I do database work by day and have a basic understanding of the postgresql db behind the gitlab issue tracker so I may be able to help with the restore process |
| 02:21:57 | <thuban> | the current method is archiving google's cached version of the issue pages, which is slow, likely not to cover every issue, and likely not to show all comments, but it's better than nothing |
| 02:22:46 | <hurricos> | Please also archive the list of watchers of the Github repository. Depending on the exact setup a watcher has a complete recollection of issues and comments. |
| 02:24:48 | <hurricos> | Where is this data being stored right now, if you don't mind me asking? Is it possible to set up an SFTP server for read access to the result? |
| 02:25:11 | <hurricos> | I'd like to take a swing at beginning converting it into a Gitlab-importable dump |
| 02:26:07 | <jodizzle> | hurricos: Our methods generally store content on the Internet Archive |
| 02:27:14 | <jodizzle> | Different topic: has there been any discussion of Twitch streamers deleting their vods? Came across this: https://old.reddit.com/r/LivestreamFail/comments/jgt86c/pokimane_deletes_4_years_worth_of_vods_says_lirik/ |
| 02:27:26 | <hurricos> | jodizzle: how is that usually structured? (do you have a link?) |
| 02:29:17 | <thuban> | it looks like the subscriber list (https://github.com/ytdl-org/youtube-dl/watchers / https://api.github.com/repos/ytdl-org/youtube-dl/subscribers) is blocked along with the rest of the repository. wbm doesn't have the api, but it does have the first page (only) of the webpage |
| 02:29:49 | <thuban> | last grab: https://web.archive.org/web/20201002044053/https://github.com/ytdl-org/youtube-dl/watchers |
| 02:30:45 | <OrIdow6> | hurricos: It will generally go into a warc file (https://www.archiveteam.org/index.php?title=The_WARC_Ecosystem http://fileformats.archiveteam.org/wiki/WARC), which holds copies of webpages |
| 02:30:52 | <OrIdow6> | Or in this case, multiple warc files |
| 02:32:08 | <thuban> | hurricos: what would the "setup" that covers all issues and comments look like? are we talking about hoping for undeleted notification emails, or is there a more structured option? |
| 02:32:17 | <@JAA> | Google doesn't have watchers cached either. |
| 02:32:17 | <icedice> | It might be worth trying to get what you can of youtube-dlc as well: https://github.com/blackjack4494/youtube-dlc |
| 02:32:36 | <icedice> | It was a fork that was more active on fixing pull requests |
| 02:33:09 | <@JAA> | Seems small, will do. |
| 02:33:35 | <hurricos> | `curl https://api.github.com/user/48381040/events | jq -c '.[] | select(.type="IssueCommentEvent")' | wc -l` |
| 02:33:52 | <hurricos> | The user API on Github exposes those issues despite the repo being taken down. |
| 02:33:59 | <@JAA> | Ooh, nice. |
| 02:34:00 | <hurricos> | We need a list of users |
| 02:34:10 | <thuban> | phenomenal! |
| 02:34:51 | <thuban> | i'll go through and pull out the user ids |
| 02:35:19 | <hurricos> | It's only a couple of days worth but we should use it while we can |
| 02:35:31 | <hurricos> | I can set up an SFTP server to upload files to |
| 02:35:35 | <hurricos> | if anyone is interested |
| 02:35:53 | <hurricos> | thuban: how are you doing that |
| 02:35:58 | <hurricos> | (what are you using to grab lists of users) |
| 02:37:53 | <icedice> | https://www.gharchive.org/ |
| 02:39:17 | <hurricos> | https://www.gharchive.org/ |
| 02:39:18 | <hurricos> | yes! |
| 02:39:31 | <hurricos> | I have 13TB free storage, quickly composing a makefile |
| 02:39:57 | <hurricos> | note that if you have individual JSON objects and a lot of cores you can very easily separate them into individual files and use `parallel -P $jobfile jq ...` |
| 02:40:05 | <hurricos> | I use this for data conversions for healthcare data from CCDAs |
| 02:42:34 | <@JAA> | Google cache of youtube-dlc issues and PRs running now in AB. |
| 02:42:55 | <thuban> | https://api.github.com/repos/ytdl-org/youtube-dl/events is blocked, but https://api.github.com/networks/ytdl-org/youtube-dl/events is NOT... and it includes the root. |
| 02:45:03 | <thuban> | pagination with ?page=<i>, last page in "link" header (9), so ~250 events, some of which are issues or prs |
| 02:46:20 | <thuban> | JAA, want me to generate those urls for AB or can you do it |
| 02:47:38 | <hurricos> | thuban: are you suggesting that https://api.github.com/networks/ytdl-org/youtube-dl/events can be descended as a tree somehow? |
| 02:48:04 | <thuban> | it includes events from every repo in the network, and the root is one of the repos in the network |
| 02:48:23 | <thuban> | some of the events in there are from the youtube-dl repo |
| 02:48:26 | <hurricos> | gotcha |
| 02:48:42 | <hurricos> | I've started downloading Github |
| 02:48:52 | <hurricos> | (from gharchive.org) |
| 02:49:02 | <hurricos> | plan is to use some jq once I've got enough to scrape from |
| 02:49:04 | <thuban> | that only goes back up to 90 days or 300 events, though |
| 02:49:08 | <@JAA> | thuban: Done |
| 02:49:26 | <hurricos> | I think gharchive contains all of the open-source events |
| 02:49:41 | <hurricos> | I can bring some rack servers to the lab if I need to distribute the load ultimately but I don't think it's that much data |
| 02:50:13 | <thuban> | i'm looking at gharchive now, tho i'm not about to beg google for bigquery access |
| 02:50:22 | <hurricos> | :P |
| 02:51:37 | <thuban> | ytdl-org/youtube-dl repo id is 1039520 |
| 02:52:15 | <thuban> | hm, or not? |
| 02:52:59 | <thuban> | seem to be multiple repos with different names and that "id" |
| 02:55:07 | <thuban> | lol, no there aren't. my bad |
| 02:56:44 | | hurricos quits [Client Quit] |
| 03:01:00 | | cadence (cadence) joins |
| 03:01:35 | <cadence> | hi o/ i'm told there's youtube-dl things happening here? |
| 03:02:52 | <thuban> | cadence: yes |
| 03:05:02 | | Dallas quits [Client Quit] |
| 03:06:07 | | hurricos joins |
| 03:06:26 | <hurricos> | what was the group ID for youtube-dl's org? |
| 03:06:50 | <cadence> | it was called ytdl-org, I remember that much |
| 03:07:18 | <Ajay> | cadence: they are referring to the results from https://api.github.com/networks/ytdl-org/youtube-dl/events |
| 03:07:54 | <Ajay> | > pagination with ?page=<i>, last page in "link" header (9), so ~250 events, some of which are issues or prs |
| 03:08:00 | | Dallas (Dallas) joins |
| 03:08:08 | <thuban> | hurricos: is group id different from org id? |
| 03:08:41 | <hurricos> | It's the same. I'm just bad at IRC and invariably close pidgin. |
| 03:08:54 | <thuban> | org id is 48381040 |
| 03:09:08 | <hurricos> | thus losing track. I've got my makefile sorted out for pulling down from gharchive, just want to start writing .jq files now |
| 03:09:15 | <hurricos> | thanks |
| 03:09:39 | <thuban> | you said you know something about importing to gitlab? |
| 03:10:19 | <hurricos> | I work with Postgres at work, run a Gitlab server personally and know how to poke at the schemas well enough to figure itout |
| 03:10:48 | <hurricos> | I don't know what import formats it can digest but I can try seeing what my instance can do. Ultimately getting the data is more important right now |
| 03:10:53 | <hurricos> | ingest* |
| 03:11:53 | <hurricos> | getting forms of the data and verifying that an issue history can be constructed from it anyways. |
| 03:11:55 | <hurricos> | `curl data.gharchive.org/2020-10-22-04.json.gz` fails |
| 03:12:10 | <hurricos> | looks like everything from 0 to 9 fails |
| 03:12:13 | <hurricos> | (hours per day) |
| 03:12:23 | <thuban> | odd |
| 03:12:25 | <thuban> | every day? |
| 03:12:39 | <hurricos> | every day |
| 03:12:57 | <thuban> | hm |
| 03:13:31 | <thuban> | ~90k events per hour, so that's a pretty big loss |
| 03:13:40 | <thuban> | wonder if it's intentional |
| 03:14:29 | <hurricos> | base64'd xz of the Makefile I use: /Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4AETAMddADIYSu7u0hpd9dYp9fTAQfIbYxiU1p8TfBs5IdFuAC1VPGqUizyTyQM3kjKkgdZY02qc+hU1hEBGQY9LotHqn75UWyE8SUZ6ocwBYFo6ib3adfUNy2PeI410+x4Sg70/kOi1QRT4jD6JA0iwy3ZKXV8VRWaOaunwT8rJuLh/4buyanFESS59pD26yc56mT+y9i1JhUwRNWvPdC5xWqB+ozXkmCHYl2v2+bDvMo8LwEiqIsJpQAyw5LJvZunaIs6M0KSdOKQRIAAAAAyYFJJrcDR+AAHjAZQCAADcNkNfscRn+wIAAAAABFla |
| 03:14:48 | <hurricos> | (to download) |
| 03:16:02 | <hurricos> | Oh I see what's going on. |
| 03:16:07 | <hurricos> | I'm pulling 09 and not 9 |
| 03:16:22 | <thuban> | lol |
| 03:16:23 | <hurricos> | oh well, I'll fix that in the recipe later |
| 03:16:46 | <hurricos> | I think gharchive should have it, could you pastebin some sample messages I should expect to be in these archives? |
| 03:16:57 | <hurricos> | or I can pull actually from the page I posted wit hcurl |
| 03:19:06 | <thuban> | i just grabbed a random hour and i definitely see ytdl issues/comments complete with bodies |
| 03:19:19 | <thuban> | (event types documentation: https://developer.github.com/v3/activity/event_types/) |
| 03:20:34 | <hurricos> | `find . -type f ! -empty -print0 | parallel -0 -P 20 zgrep -we 13954170828` ... |
| 03:21:24 | <hurricos> | just looking for a specific one I pulled from https://api.github.com/user/48381040/events is in the archives |
| 03:21:48 | <hurricos> | It's not |
| 03:22:04 | <thuban> | event id? |
| 03:22:08 | <hurricos> | so if you curl that you can grep for 13954170828 |
| 03:23:03 | <hurricos> | that id is not in the respective file on 2020-10-23T19:13:57Z |
| 03:23:24 | <hurricos> | Nope, I'm a dumbass. I did not pull that hour |
| 03:23:54 | | qw3rty_ joins |
| 03:24:24 | <hurricos> | it's in 19 :D |
| 03:24:40 | <Ajay> | Would this data include pull request branches? |
| 03:24:41 | <hurricos> | OK, so we get the same data from gharchive as from the api.github.com/user/${id}/events |
| 03:24:57 | <hurricos> | check out the event_types from the developer.github.com documentation |
| 03:25:06 | <hurricos> | I can give a list of what I see in the JSONs though |
| 03:25:17 | <hurricos> | maybe not the content of the PRs but the issue titles and first post, yes |
| 03:25:27 | <Ajay> | yea, I wonder if it is just the posts |
| 03:25:28 | <thuban> | Ajay: gharchive logs _all_ public events, so yes, the data is there (if they're private) |
| 03:25:28 | <hurricos> | perhaps even a transactional history of what and how things were tagged. Let me get a list of event types |
| 03:25:42 | <hurricos> | awesome |
| 03:25:46 | | cadence_ (cadence_) joins |
| 03:25:50 | <Ajay> | ooh, that's great |
| 03:25:54 | <hurricos> | I just pulled october, it's 26g |
| 03:25:59 | <hurricos> | without 0-9AM |
| 03:26:01 | <hurricos> | so |
| 03:26:09 | <hurricos> | I should resize the volume and kick this off for an overnight run |
| 03:26:28 | <hurricos> | seems like this can be processed into a SQLite database with some good parallel code tbf |
| 03:27:24 | | qw3rty__ quits [Ping timeout: 240 seconds] |
| 03:27:44 | <hurricos> | napkin math shows gharchive until 2015 is 3TB |
| 03:27:53 | <hurricos> | sounds small. that can't contain public comment history? |
| 03:28:00 | <hurricos> | are you sure it's *all* public events @ajay? |
| 03:28:15 | <hurricos> | well it is gzipped though. |
| 03:28:18 | <Ajay> | you mean thuban |
| 03:28:22 | <hurricos> | yeah sorry |
| 03:28:24 | <hurricos> | thuban: |
| 03:28:28 | <cadence_> | Git objects are already compressed |
| 03:28:41 | <hurricos> | not git objects, the public comments in issues etc. |
| 03:28:52 | <hurricos> | that's what needs recovered |
| 03:28:58 | <cadence_> | Hmm. Then maybe. Still seems kinda small |
| 03:29:07 | <cadence_> | Sorry for misunderstanding |
| 03:29:30 | <Ajay> | the git object for pull requests would also be important to recover |
| 03:30:00 | <hurricos> | hmm |
| 03:30:33 | <Ajay> | youtube-dl has tons of un-merged pull requests in limbo |
| 03:30:52 | <cadence_> | I think youtube-dlc got a lot of them, but yeah |
| 03:31:11 | <hurricos> | youtube-dlc? we have backups of it though? |
| 03:31:14 | <hurricos> | the PRs |
| 03:31:31 | <Ajay> | oh, we do? |
| 03:31:50 | <cadence_> | youtube-dlc is a fork that merged a bunch of those pull requests |
| 03:31:59 | <Ajay> | youtube-dlc was a fork meant to try to go through the pull request backlog and merge stuff |
| 03:32:05 | <cadence_> | I have it cloned but nothing else from it |
| 03:32:18 | <hurricos> | I wonder if it got pulled down at the same time. If not someone may have a record |
| 03:32:43 | <Ajay> | it seemed to have been pulled at the same time |
| 03:32:47 | <hurricos> | gotcha |
| 03:32:50 | <Ajay> | as well as many other forks |
| 03:32:52 | <cadence_> | Yeah, cause it was a fork. |
| 03:33:18 | <hurricos> | Well, the PRs can be reestablished by users when a new site goes up. |
| 03:33:25 | <cadence_> | I have it cloned which therefore means I probably have a bunch of youtube-dl's unmerged prs. |
| 03:34:04 | <hurricos> | also checking my math again, it's looking closer to 6TB |
| 03:34:04 | <cadence_> | The youtube-dlc people are discussing similar things on gitter, IMO it would be good to organise with them |
| 03:34:12 | <hurricos> | which gitter? |
| 03:34:35 | <cadence_> | The only reason -c existed was because the youtube-dl maintainers were uncooperative. And now that they're out of the way… |
| 03:34:43 | <cadence_> | One sec while I get it |
| 03:34:57 | <thuban> | i think that's a little hasty |
| 03:34:57 | <cadence_> | `youtube-dlc/community` |
| 03:35:22 | <hurricos> | https://gitter.im/youtube-dlc/community |
| 03:35:24 | <hurricos> | I'm there |
| 03:35:31 | <hurricos> | I've been there already :) |
| 03:35:58 | <cadence_> | IMO it would be good to collab & community outreach & all that cause you both have the same goals now |
| 03:36:29 | <hurricos> | I'm literally in that gitter though. The gitter just reflects https://app.element.io/#/room/!xbOjHLEQzPJBXjeTWo:matrix.org |
| 03:36:48 | <cadence_> | Ah, cool ^.^ |
| 03:36:54 | <hurricos> | really I just need a place to coordinate this. I have a homeserver running a Gitlab instance that I'd rather not use for this because I like having home internet and I bet I'd get DDoSed |
| 03:37:00 | <hurricos> | but I *could* do it there. |
| 03:37:22 | <thuban> | aha! ok, so, the PullRequestEvent _does_ have references to the git repository and commit id of the pull request _from the fork_ |
| 03:37:44 | <thuban> | so, still accessible if the repo being pr'd into is hidden |
| 03:39:25 | <thuban> | .payload.pull_request.head.repo.git_url and .payload.pull_request.head.sha |
| 03:39:32 | | cadence_ quits [Read error: Connection reset by peer] |
| 03:40:33 | <thuban> | JAA or anyone else in a position to judge: do we want to mirror GHArchive? |
| 03:40:34 | | cadence_ (cadence_) joins |
| 03:40:37 | <thuban> | as in any/all of (a) downloading its existing backlog, (b) downloading it on an ongoing basis, or (c) doing similar and/or redundant work as part of #gitgud |
| 03:41:30 | | cadence_ quits [Read error: Connection reset by peer] |
| 03:41:40 | <thuban> | (i asked over there whether the web potion of the project included api results, but it's pretty quiet) |
| 03:41:46 | <hurricos> | I can create .torrent files of the whole thing |
| 03:41:58 | | cadence_ (cadence_) joins |
| 03:42:03 | <hurricos> | ... and seed |
| 03:42:39 | <hurricos> | Laboratory B has server infrastructure, that is, we have a single 12-bay R510 with 13.5TB RAID10, 150Mb symmetrical |
| 03:43:18 | <hurricos> | sorry, never mind. I'm not even part of archiveteam. Just gonna focus on the youtube-dl thing. |
| 03:44:40 | <Arcorann> | There's nothing stopping you from helping out with other archiveteam projects after this gets resolved |
| 03:44:55 | <hurricos> | I don't have the time :( I distribute laptops for local community |
| 03:45:47 | <hurricos> | if I could get paid doing that I'd be happy to, I've just already overcommitted and I need to get better at cleaning up house before I start anything like that |
| 03:45:58 | <hurricos> | but I personally rely on youtube-dl, so |
| 03:46:10 | <hurricos> | I just want to see them get the basic stuff to start back up with. |
| 03:46:14 | | cadence_ quits [Ping timeout: 252 seconds] |
| 03:47:30 | <Arcorann> | Fair enough |
| 03:47:44 | <thuban> | hurricos: downloading large amounts of data is a problem we already have a lot of infrastructure for--i think if you really want to see them back up and running soon the hard part is the export. |
| 03:48:28 | | cadence_ (cadence_) joins |
| 03:48:40 | <thuban> | exporting archived data to a gitlab-importable format is something that's been in our long-term plans for a while, but nobody's made a start on it yet--do you want to? |
| 03:49:20 | <hurricos> | I'll take a swing. Not something I was planning to do already but I am planning on doing some work on fresh copies of a Gitlab instance so I might as well try |
| 03:49:53 | <thuban> | great! if you poke us here or in #gitgud i'm sure people will contribute once the ball is rolling |
| 03:50:23 | <hurricos> | I remember now. Gitlab actually lets you import issue lists as CSVs. Comment history perhaps not directly. It would be good to centralize that work around an issue tracker already, does the archiveteam have one? Kanban board of any kind? |
| 03:51:41 | | Wingy quits [Remote host closed the connection] |
| 03:52:08 | | Wingy (Wingy) joins |
| 03:52:20 | <thuban> | generally just the github issue tracker for each code repository |
| 03:52:44 | <thuban> | (which is now seeming like maybe not the best policy ;) but i believe we have copies) |
| 03:53:36 | <thuban> | non-directly-code-related plans and issues are coordinated over irc or on the wiki |
| 03:54:04 | <hurricos> | chicken-and-egg |
| 03:54:32 | <thuban> | ? |
| 03:56:10 | <hurricos> | Lol, just having an issue tracker for things hosted on the place you'd like to make sure is safe from them |
| 03:56:23 | <thuban> | mm |
| 03:57:51 | <hurricos> | not having readline installed (from within a docker container) *sucks* |
| 04:02:30 | | tar-xvf quits [Remote host closed the connection] |
| 04:02:51 | | tar-xvf joins |
| 04:05:53 | | icedice quits [Client Quit] |
| 04:06:23 | <hurricos> | OK, got it. Gitlab uses an `issues` table, comments are in the `notes` table. `notes` points to `issues` by `noteable_type` and `noteable_id` fields. The state of the issue is in the `todos` table; a `todos` can point to an `issues` via the `target_type` and `target_id` fields. |
| 04:06:48 | <hurricos> | everything else, e.g. attachments, links on as you might expect, but it's a fairly loose, object-oriented framework |
| 04:07:03 | <hurricos> | that's 11.3 CE, I haven't updated in a while :upside-down-face: |
| 04:07:33 | <hurricos> | I'll go into gitgud and ask about repositories |
| 04:15:56 | | HP_Archiv quits [Client Quit] |
| 04:16:14 | | HP_Archivist (HP_Archivist) joins |
| 04:23:59 | <mgrandi> | so are you folks just trying to recreate the youtube-dl issues from the gharchive data? |
| 04:24:03 | | quanta314 joins |
| 04:24:25 | <hurricos> | Yes |
| 04:26:29 | <mgrandi> | good luck! |
| 04:26:44 | <hurricos> | I'll need it :( |
| 04:37:44 | | qw3rty_ quits [Ping timeout: 240 seconds] |
| 04:40:16 | | SketchTheCow quits [Read error: Connection reset by peer] |
| 04:40:27 | | quanta314 quits [Client Quit] |
| 04:40:29 | | SketchTheCow joins |
| 04:40:59 | <mgrandi> | @Jean-Fred luckily, PS's store has a all games option 👀 makes this much easier |
| 04:47:11 | | cadence_ quits [Read error: Connection reset by peer] |
| 04:47:31 | | cadence_ (cadence_) joins |
| 05:22:14 | | godane2 joins |
| 05:22:14 | | cadence_ quits [Read error: Connection reset by peer] |
| 05:22:46 | | cadence_ (cadence_) joins |
| 05:24:24 | | godane1 quits [Ping timeout: 240 seconds] |
| 05:29:24 | | cadence_ quits [Ping timeout: 240 seconds] |
| 05:36:02 | | klg quits [Remote host closed the connection] |
| 05:36:34 | | cadence_ (cadence_) joins |
| 05:38:56 | | klg (klg) joins |
| 06:51:27 | | cadence_ quits [Read error: Connection reset by peer] |
| 06:52:28 | | cadence_ (cadence_) joins |
| 08:21:53 | | @arkiver quits [Excess Flood] |
| 08:22:23 | | arkiver (arkiver) joins |
| 08:22:23 | | @ChanServ sets mode: +o arkiver |
| 08:43:16 | <Ryz> | Potential archiving inspiration? https://old.reddit.com/r/AskReddit/comments/jgv6iq/the_internet_is_scheduled_to_go_down_forever_you/ |
| 09:05:13 | <HP_Archivist> | Heh ^^ |
| 09:05:33 | <HP_Archivist> | 'If you see something, save it' |
| 09:39:50 | | cadence_ quits [Client Quit] |
| 09:49:21 | | @arkiver quits [Excess Flood] |
| 09:49:51 | | arkiver (arkiver) joins |
| 09:49:51 | | @ChanServ sets mode: +o arkiver |
| 10:28:15 | | qw3rty joins |
| 10:42:52 | | @arkiver quits [Excess Flood] |
| 10:43:19 | | arkiver (arkiver) joins |
| 10:43:19 | | @ChanServ sets mode: +o arkiver |
| 11:18:48 | | @arkiver quits [Excess Flood] |
| 11:19:16 | | arkiver (arkiver) joins |
| 11:19:16 | | @ChanServ sets mode: +o arkiver |
| 11:28:23 | | @arkiver quits [Excess Flood] |
| 11:28:38 | | arkiver (arkiver) joins |
| 11:28:39 | | @ChanServ sets mode: +o arkiver |
| 11:29:57 | | icedice joins |
| 11:37:19 | <purplebot> | FileFormats created by JesseW (+21, Redirected page to [[Formats]]) just now -- https://www.archiveteam.org/?diff=45703&oldid=0 |
| 12:08:27 | | mutantmonkey quits [Remote host closed the connection] |
| 12:08:45 | | mutantmonkey (mutantmonkey) joins |
| 12:16:57 | | @arkiver quits [Excess Flood] |
| 12:17:25 | | arkiver (arkiver) joins |
| 12:17:25 | | @ChanServ sets mode: +o arkiver |
| 12:49:53 | <wessel1512> | i have found a python script that keyword crawler websites |
| 12:50:56 | <wessel1512> | but it crashes after just a few url captured https://transfer.notkiska.pw/11P1Ep/website_keyword_crawl_error.txt |
| 12:51:21 | <wessel1512> | https://github.com/wessel1512/website-keyword-crawler/blob/master/website_keyword_crawl.py |
| 12:52:03 | <wessel1512> | only i dont know how to fix it |
| 13:00:28 | <wessel1512> | and i like to filter things like: .jpg .png and .js files out |
| 14:15:19 | <purplebot> | File:Albumee-logo.gif uploaded by Arkiver (+0) just now -- https://www.archiveteam.org/?diff=45704&oldid=0 |
| 14:49:23 | | @arkiver quits [Excess Flood] |
| 14:49:26 | | Arcorann quits [Read error: Connection reset by peer] |
| 14:49:52 | | arkiver (arkiver) joins |
| 14:49:52 | | @ChanServ sets mode: +o arkiver |
| 15:23:11 | | hurricos quits [Client Quit] |
| 15:27:08 | | Stiletto quits [Ping timeout: 252 seconds] |
| 15:32:38 | <Jean-Fred> | mgrandi Awesome! Thanks :-) Are you archiving the us-en store only? Asking because there are 5 stores (EMEA and others, America (North and South), Asia, Japan and China), and some stores with many subdomains per country/lang (eg German, French etc). And some information (like local content ratings eg USK or PEGI) might be only in one sub-store. I |
| 15:32:38 | <Jean-Fred> | got that list of domains − , not sure how exhaustive it is but that’s already a start https://justpaste.it/93kgd |
| 15:42:41 | | Stiletto joins |
| 15:54:13 | | @arkiver quits [Excess Flood] |
| 15:54:24 | | arkiver (arkiver) joins |
| 15:54:24 | | @ChanServ sets mode: +o arkiver |
| 16:51:37 | | britm0b quits [Read error: Connection reset by peer] |
| 16:58:52 | | britm0b joins |
| 18:51:19 | | @arkiver quits [Excess Flood] |
| 18:51:49 | | arkiver (arkiver) joins |
| 18:51:49 | | @ChanServ sets mode: +o arkiver |
| 19:25:28 | <icedice> | youtube-dlc is back: https://github.com/blackjack4494/yt-dlc |
| 19:25:43 | <icedice> | It's a more up to date version of youtube-dl |
| 19:26:09 | <icedice> | The dev fixes pull requests a lot faster there |
| 19:28:05 | | thuban quits [Quit: What price Glory ?] |
| 19:30:46 | | thuban joins |
| 19:45:48 | | Smiley (Smiley) joins |
| 19:47:50 | | SmileyG quits [Ping timeout: 252 seconds] |
| 20:30:36 | <mgrandi> | @Jean-Fred: I can see if I can the other language stores too |
| 22:27:17 | <Jean-Fred> | mgrandi Thanks for looking ; I heard from others that the store pages are gone for many/most people now :-( |
| 22:28:45 | <mgrandi> | i think you have to log in? |
| 22:28:59 | <Jean-Fred> | Aaaaah maybe? |
| 22:29:27 | <mgrandi> | https://store.playstation.com/en-us/grid/STORE-MSF77008-ALLGAMES/1?PlatformPrivacyWs1=exempt&direction=asc&psappver=19.15.0&scope=sceapp&smcid=psapp%3Alink%20menu%3Astore&sort=release_date |
| 22:29:33 | <Jean-Fred> | (Also, looking at https://web.archive.org/web/sitemap/https://store.playstation.com/ , things seem to have been well crawled in the past) |
| 22:29:53 | <mgrandi> | going to store.playstation.com seems to be a white page that seems new but going to that link i just posted seems like the old game page |
| 22:30:27 | <mgrandi> | i'll work on this now then |
| 22:30:41 | <Jean-Fred> | Thank you so much, really :) |
| 22:31:09 | <Jean-Fred> | (ok off to sleep here) |
| 22:44:16 | <@JAA> | thuban: Size of GH Archive? |
| 22:53:17 | | MrKen joins |
| 23:00:19 | | Arcorann (Arcorann) joins |
| 23:01:28 | | Arcorann quits [Remote host closed the connection] |
| 23:01:51 | | Arcorann (Arcorann) joins |
| 23:20:14 | | lunik10 joins |
| 23:21:46 | | lunik1 quits [Ping timeout: 252 seconds] |
| 23:21:46 | | lunik10 is now known as lunik1 |