| 00:53:06 | | DopefishJustin quits [Ping timeout: 240 seconds] |
| 02:36:35 | | Stiletto joins |
| 02:47:56 | | TheTechRobo3641 joins |
| 02:48:09 | | TheTechRobo3641 quits [Remote host closed the connection] |
| 02:48:26 | | TheTechRobo3641 joins |
| 02:50:46 | | thetechrobo_ quits [Ping timeout: 265 seconds] |
| 03:06:56 | | TheTechRobo3641 is now known as TheTechRobo |
| 03:07:44 | | TheTechRobo is now authenticated as TheTechRobo |
| 03:16:02 | | DopefishJustin joins |
| 03:16:02 | | DopefishJustin is now authenticated as DopefishJustin |
| 05:28:40 | | qwertyasdfuiopghjkl joins |
| 06:27:41 | | Lord_Nightmare quits [Ping timeout: 252 seconds] |
| 06:31:21 | | Matthww quits [Ping timeout: 252 seconds] |
| 06:34:07 | | Matthww joins |
| 06:49:09 | | sonick quits [Quit: Connection closed for inactivity] |
| 06:55:35 | | monika quits [Ping timeout: 244 seconds] |
| 08:34:52 | | AK (AK) joins |
| 09:47:20 | | qw3rty joins |
| 09:53:54 | | sonick (sonick) joins |
| 12:28:29 | | spirit quits [Ping timeout: 252 seconds] |
| 13:15:28 | | spirit joins |
| 13:34:07 | | Arcorann quits [Ping timeout: 252 seconds] |
| 14:11:03 | | qwertyasdfuiopghjkl quits [Remote host closed the connection] |
| 14:12:22 | | qwertyasdfuiopghjkl joins |
| 14:58:47 | | thetechrobo_ joins |
| 14:59:09 | | thetechrobo_ quits [Remote host closed the connection] |
| 14:59:31 | | thetechrobo_ joins |
| 15:02:03 | | TheTechRobo quits [Ping timeout: 265 seconds] |
| 15:46:12 | | qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds] |
| 15:46:13 | | qwertyasdfuiopghjkl1 joins |
| 16:14:00 | | qwertyasdfuiopghjkl1 is now known as qwertyasdfuiopghjkl |
| 16:34:32 | | TheTechRobo3641 joins |
| 16:38:14 | | thetechrobo_ quits [Ping timeout: 265 seconds] |
| 17:45:24 | | DogsRNice (Webuser299) joins |
| 21:13:05 | | mattwright324|m joins |
| 21:57:22 | <mattwright324|m> | anyone able to help with the cdx api and querying youtube urls? it does work fine for many videos but not for all, for example this video does not return anything from cdx but does have captures |
| 21:57:22 | <mattwright324|m> | https://web.archive.org/cdx/search/cdx?matchType=exact&limit=1&output=json&url=https://www.youtube.com/watch?v=AKmtDSAKR8Y |
| 21:57:22 | <mattwright324|m> | https://web.archive.org/web/*/https://www.youtube.com/watch?v=AKmtDSAKR8Y |
| 22:03:22 | <OrIdow6> | Funny timing, another outsider warc in the wbm |
| 22:03:24 | <OrIdow6> | https://archive.org/details/warc_www_reddit_com-r-TrueFilm-comments-5o4uhg-all_about_lily_chouchou_20170126 |
| 22:04:15 | <OrIdow6> | Interesting |
| 22:06:36 | <OrIdow6> | mattwright324|m: The problem is that the CDXs server is apparently a sorted index in order of SERTs (aka URlkeys, e.g. com,ytimg,i)/vi/akmtdsakr8y/default.jpg) |
| 22:08:04 | <OrIdow6> | I don't entirely understand what's happening here, but it looks like those captures you see aren't the plain URL |
| 22:09:23 | <mattwright324|m> | matchType prefix doesn't work when the url already has parameters? assumed maybe there was other params on the end in those captures but that didn't work either |
| 22:09:28 | <OrIdow6> | The first one shows up in some places, including the source WARC, as https://www.youtube.com/watch?v=AKmtDSAKR8Y&feature=youtu.be&t=301 , but somewhere it's getting displayed with the feature and t params stripped off, I think? |
| 22:11:09 | <OrIdow6> | https://www.youtube.com/watch?v=AKmtDSAKR8Y&feature=youtu.be&t=301 becomes com,youtube)/watch?feature=youtu.be&t=301&v=akmtdsakr8y |
| 22:11:38 | <OrIdow6> | But even on the full URL the CDX server doesn't return it |
| 22:12:22 | <OrIdow6> | Though the availability API does seem to have it |
| 22:12:43 | <OrIdow6> | There are like 3 edge/mystery cases in play here, I don't know which one is causing your problem |
| 22:13:49 | <OrIdow6> | Prefix works, but AFAICT the range query depends on the SURT, which sorts the params by name |
| 22:14:01 | <OrIdow6> | Works some of the time |
| 22:17:51 | <mattwright324|m> | have tried the availability api which can even show availability of fakeurl direct video links but it didn't seem as fast/reliable as the cdx api |
| 22:48:31 | <@JAA> | Parameters get sorted in the surts as well. |
| 22:48:52 | <@JAA> | Oh you already mentioned that. :-) |
| 22:50:12 | <@JAA> | Er, it does show up here: https://web.archive.org/cdx/search/cdx?matchType=exact&limit=1&output=json&url=https://www.youtube.com/watch?feature=youtu.be&t=301&v=AKmtDSAKR8Y |
| 22:50:37 | | Arcorann (Arcorann) joins |
| 22:52:26 | | DogsRNice quits [Read error: Connection reset by peer] |
| 22:53:35 | | DogsRNice (Webuser299) joins |
| 22:53:50 | <mattwright324|m> | that though I think is returning "http://youtube.com/watch?feature=youtu.be" but not for that specific video id, date is 2012 on that first entry. earliest capture from standard wayback search was 2017 |
| 22:54:05 | <@JAA> | Er yes |
| 22:58:20 | <@JAA> | I would suggest checking my complete CDX enumeration from a couple months ago, but that's still access-restricted... https://archive.org/details/youtube_cdx_202111 |
| 22:58:23 | <@JAA> | arkiver: ^ |
| 23:02:32 | <@JAA> | mattwright324|m: Here's the 2021 snapshot: https://web.archive.org/cdx/search/cdx?matchType=exact&limit=1&output=json&url=https://www.youtube.com/watch?spf=navigate%26v=AKmtDSAKR8Y |
| 23:04:49 | <@JAA> | April 2017: https://web.archive.org/cdx/search/cdx?matchType=exact&limit=1&output=json&url=https://www.youtube.com/watch?gl=US%26hl=en%26v=AKmtDSAKR8Y |
| 23:07:02 | <@JAA> | I guess the Jan 2017 one isn't returned because it's not supposed to be in the WBM. But for some reason it's still in the WBM index apparently? No changes to that item since 2018, so something's definitely wrong there. |
| 23:07:27 | <mattwright324|m> | interesting, is there no way to do it currently without those params then? just happens all 3 captures had extra params? |
| 23:07:28 | <@JAA> | arkiver: ^ https://archive.org/details/warc_www_reddit_com-r-TrueFilm-comments-5o4uhg-all_about_lily_chouchou_20170126 https://web.archive.org/web/20170126135640/https://www.youtube.com/watch?v=AKmtDSAKR8Y |
| 23:08:10 | <@JAA> | mattwright324|m: Correct, you have to know all parameters that come before the one you're looking for when alphabetically sorted. Which means it's pretty much useless for YouTube. |
| 23:13:33 | <mattwright324|m> | well that answers it then, thanks! was hoping there was a way since cdx was fast. only other options then is the availability api though that is slow and timesout frequently and doing HEAD https://web.archive.org/web/*/https://www.youtube.com/watch?v={videoId} requests which is similarly slow but at least doesn't timeout |
| 23:21:28 | <@JAA> | If you need to check a large amount of video IDs and don't mind using slightly outdated data, I'd wait for that item I linked above to become available. That's much more efficient once you get it downloaded. |
| 23:23:06 | <@JAA> | Although it's only the unique original URLs, no further details about the snapshots. |
| 23:23:28 | <@JAA> | (The only goal at the time was video ID discovery, and that information is sufficient for that task.) |
| 23:29:37 | <mattwright324|m> | intent is to run availability checks on private and deleted youtube ids in variable amounts in a client side js web app context. depends if any ids come up for a particular channel, typically would be in the range of 1-200 though definitely are channels where there would be many more than that. though given how slow the other options are not likely to implement that kind of checking |
| 23:30:51 | <@JAA> | Hmm, right. |