| 00:27:06 | | Arcorann quits [Ping timeout: 258 seconds] |
| 01:09:05 | | balrog quits [Quit: Bye] |
| 01:32:35 | | balrog (balrog) joins |
| 02:20:27 | | Arcorann (Arcorann) joins |
| 04:15:22 | | qw3rty_ joins |
| 04:19:29 | | qw3rty__ quits [Ping timeout: 265 seconds] |
| 05:09:31 | | DogsRNice quits [Read error: Connection reset by peer] |
| 05:37:47 | | tzt quits [Ping timeout: 265 seconds] |
| 06:00:12 | | tzt (tzt) joins |
| 06:45:55 | | datechnoman (datechnoman) joins |
| 08:29:23 | | Lord_Nightmare quits [Quit: ZNC - http://znc.in] |
| 08:32:42 | | Lord_Nightmare (Lord_Nightmare) joins |
| 09:57:12 | | spirit quits [Client Quit] |
| 13:36:46 | | Arcorann quits [Ping timeout: 258 seconds] |
| 14:03:29 | | spirit joins |
| 15:18:16 | | spirit quits [Client Quit] |
| 17:04:00 | | G4te_Keep3r quits [Quit: The Lounge - https://thelounge.chat] |
| 17:04:32 | | G4te_Keep3r joins |
| 19:30:12 | | Stellarator joins |
| 19:31:37 | <@JAA> | From #down-the-tube: < Stellarator> Not sure if this question is relevant to the channel, but I've noticed that cdx regex doesn't work that well, very often not showing results when those clearly exists. Is there any way to make it search over a larger amount of list in order to make the regex search more effective? |
| 19:31:46 | <@JAA> | Which search are you referring to exactly? |
| 19:32:06 | <@JAA> | There's web.archive.org/web/*/PREFIX/* and the CDX API, probably more as well. |
| 19:32:53 | <Stellarator> | https://web.archive.org/cdx/search/cdx?url=youtube.com/user*&filter=original:.*view=45.*&fl=timestamp,original,mimetype&collapse=urlkey |
| 19:32:57 | <Stellarator> | something like this |
| 19:33:50 | <Stellarator> | So yes, I'm referring to CDX api's regex filter |
| 19:33:54 | <@JAA> | Right |
| 19:34:16 | <@JAA> | That API is a bit weird. |
| 19:34:17 | <Stellarator> | It's a simple query, as you can see |
| 19:34:25 | <@JAA> | It has pagination except it's really odd. |
| 19:34:57 | <@JAA> | With a simple query, it only returns results from the first few million results of the prefix search. |
| 19:35:02 | | britmob|m joins |
| 19:35:42 | <@JAA> | I.e. it does the prefix search with the url parameter and then filters those with the regex. |
| 19:35:58 | <Stellarator> | From what I can tell it applies the filter on the first million (or so) results without the filter |
| 19:36:01 | <Stellarator> | exactly |
| 19:36:11 | <@JAA> | If there are more results from the prefix search than fit into that space, and none of them match, you get zero results. |
| 19:36:45 | <@JAA> | I'm not entirely sure what it actually searches with your query. It automatically enables prefix search mode with a trailing /*, but your url value doesn't have that. |
| 19:36:53 | <Stellarator> | Is there any way to overcome this problem? By abusing the pagination feature, perhaps? |
| 19:37:02 | <@JAA> | Yes, you need to use pagination. |
| 19:37:03 | <@JAA> | BUT |
| 19:37:07 | <@JAA> | There are two different paginations. |
| 19:37:11 | <@JAA> | Ignore the resumeKey one. |
| 19:37:33 | <@JAA> | You want showNumPages + page. |
| 19:38:09 | <@JAA> | There's also weird rate limiting handling. |
| 19:38:26 | <@JAA> | If you want to discover all that yourself, have fun. Or use my script: https://gitea.arpa.li/JustAnotherArchivist/little-things/src/branch/master/ia-cdx-search :-) |
| 19:39:13 | <Stellarator> | What does the script do precisely? |
| 19:40:08 | <@JAA> | It handles pagination and rate limiting with automatic retries and parallel requests. Output is JSONL with one line per match. |
| 19:40:56 | <@JAA> | So it requests showNumPages=true and then iterates over all pages, treating the awkward rate limit redirects (they're 302s to /429.html, which returns HTTP 200...). |
| 19:41:19 | <Stellarator> | Hmm. That might come in handy for my use case I suppose |
| 19:41:47 | <@JAA> | If you specifically want YouTube, I'd recommend using my existing list (unless you need more current data). |
| 19:41:53 | <@JAA> | There are like 200k pages for youtube.com... |
| 19:42:10 | <@JAA> | And it's slow. I don't remember how long it took to run, maybe a week? |
| 19:42:38 | <Stellarator> | Are those videoId, browseId lists? |
| 19:43:05 | <@JAA> | youtube_cdx_202111 is a list of all YouTube URLs known to the CDX API. |
| 19:43:25 | <Stellarator> | Oh that's great |
| 19:43:44 | <@JAA> | And then I attempted to extract video, channel, and playlist IDs from those, although that's not entirely robust because there were so many different URL formats over the past 15 years. |
| 19:44:24 | <Stellarator> | true |
| 19:44:44 | <Stellarator> | I can think of at least 6 url formats for playlists only |
| 19:52:41 | | Jopik (Jopik) joins |
| 20:30:41 | | DogsRNice (Webuser299) joins |
| 20:34:23 | | systwi_ quits [Read error: Connection reset by peer] |
| 20:35:09 | | systwi (systwi) joins |
| 21:06:07 | | Stellarator quits [Client Quit] |
| 21:27:49 | | igneousx (igneousx) joins |
| 21:36:34 | | Arcorann (Arcorann) joins |
| 23:06:35 | | Sluggs quits [Quit: Going offline, see ya! (www.adiirc.com)] |
| 23:40:31 | | qwertyasdfuiopghjkl quits [Ping timeout: 258 seconds] |
| 23:49:38 | | Sluggs joins |