HomeSearchPrevious dayNext day

00:28:27@hook54321 quits [Ping timeout: 622 seconds]
00:28:27mgrandi quits [Ping timeout: 622 seconds]
00:28:27HCross quits [Ping timeout: 622 seconds]
00:28:50justcool393 quits [Ping timeout: 622 seconds]
00:28:50NotEggplant quits [Ping timeout: 622 seconds]
00:30:22aarchi quits [Ping timeout: 622 seconds]
00:32:17tech234a quits [Ping timeout: 622 seconds]
00:34:12sonick quits [Ping timeout: 622 seconds]
00:35:21themadpro quits [Ping timeout: 622 seconds]
00:43:53sonick (sonick) joins
00:44:27themadpro (themadpro) joins
00:49:31sonick quits [Ping timeout: 265 seconds]
00:51:58tech234a (tech234a) joins
00:56:26themadpro quits [Ping timeout: 622 seconds]
01:01:40NotEggplant joins
01:01:45mgrandi (mgrandi) joins
01:21:50justcool393 (justcool393) joins
01:22:36HCross (HCross) joins
01:23:27aarchi (aarchi) joins
01:24:56themadpro (themadpro) joins
01:24:57sonick (sonick) joins
01:35:58hook54321 (hook54321) joins
01:35:58@ChanServ sets mode: +o hook54321
01:39:07murmur quits [Quit: leaving]
02:23:18<@JAA>What am I doing wrong with the CDX API here? https://web.archive.org/cdx/search/cdx?url=reddit.com&matchType=domain&limit=10&filter=original:^https?://(?:[^/]*\.)?reddit\.com(?::[0-9]*)?/r/[^/]*/about
02:24:07<@JAA>My goal is to get all 'about' pages for all subreddits. As far as I can tell, that filter should match those, but no results are returned.
02:33:49<@JAA>filter=original:^https?://(?:[^/]*\.)?reddit\.com(?::[0-9]*)?/ returns com,reddit)/ but no subdomains (except www, but that's special-cased), and I don't understand that one either.
02:35:44<@JAA>Ditto with filter=original:^https?://[^/]*reddit\.com(?::[0-9]*)?/ and I know I've used that exact format before to find subdomains...
02:35:56<@JAA>Maybe things are just fucked still from the earlier downtime?
02:36:12HP_Archivist (HP_Archivist) joins
02:37:01murmur joins
02:39:22<@JAA>My understanding is that both of those should return all known URLs on reddit.com and subdomains. At least the docs mention nothing that the pattern has to match the full URL. But even if so, at least it should return old.reddit.com, np.reddit.com, etc.
02:40:59<@JAA>This different approach doesn't work either: filter=original:.*/r/.*/about.*
02:41:35<@JAA>filter=original:.*/about.* returns a handful of URLs, but none of the ones I'm interested in, even though I know there are some in the WBM, see e.g. https://web.archive.org/web/*/reddit.com/r/pics/about/*
02:51:49HP_Archivist quits [Ping timeout: 258 seconds]
02:54:14HP_Archivist (HP_Archivist) joins
02:55:39HP_Archivist quits [Read error: Connection reset by peer]
03:19:58HP_Archivist (HP_Archivist) joins
03:20:24<OrIdow6>JAA: What are you using to get around the restriction on the number of results it gives you w/o pagination? Or has that been removed?
03:23:47<@JAA>OrIdow6: This is just for testing. I do use the resumeKey pagination.
03:24:45<@JAA>As I understand it, that one can't return empty pages, unlike the page=N one.
03:26:10<@JAA>And well, I don't get a resumeKey anyway if I add showResumeKey=true, so pretty sure that isn't the problem here.
03:32:11<OrIdow6>I don't thinkthat does "real" pagination - https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/src/main/java/org/archive/cdxserver/CDXServer.java#L280
03:32:53<@JAA>I use resumeKey/showResumeKey, not page/showNumPages.
03:33:08<OrIdow6>Try running your query with page=98901, do you get anything?
03:33:31<@JAA>The page param stuff is not true pagination, yes. It goes through different shards of the CDX index, I think?
03:33:58<OrIdow6>Something like that
03:34:33<@JAA>resumeKey is normal pagination, though not with page numbers. You get the first $limit results and, if there are more rows, the resumeKey for the next page.
03:35:31<@JAA>But yeah, https://web.archive.org/cdx/search/cdx?url=reddit.com&matchType=domain&filter=original:^https?://(?:[^/]*\.)?reddit\.com(?::[0-9]*)?/r/[^/]*/about.*&page=98901 returns something. Wut?
03:35:35<OrIdow6>But if I remember correctly resumeKey doesn't get you past the limitation on the initial fetch of results
03:36:23<@JAA>https://web.archive.org/cdx/search/cdx?url=reddit.com&matchType=domain&filter=original:^https?://(?:[^/]*\.)?reddit\.com(?::[0-9]*)?/r/[^/]*/about.*&showNumPages=true → 356731
03:36:27<@JAA>Uhh, yeah...
03:36:28<OrIdow6>It's been a while since I tried using it, but I think that was what I found back then
03:38:02<OrIdow6>So I believe you are fetching the 1.5 million (or 15 million, I can't remember) results it gives you that are at the front of the domain match, then filtering, then applying pagination
03:38:27<@JAA>Hmm
03:39:15<OrIdow6>If you add a page= it will filter on that specific page
03:39:24<OrIdow6>So what I usually do is just iterate through all pages
03:40:31<@JAA>Right, that's what I used to do as well. I did compare the results yesterday though in some tests and got identical lists. Hmm
03:40:37<@JAA>Only small tests, FWIW.
03:41:42<OrIdow6>If it's small enough that it's below the cutoff for non-paginated queries I would expect it to behave identically (except for recently-added entried)
03:41:50<OrIdow6>*entries
03:44:24<OrIdow6>Basically, what I've seen in practice (I suppose much of it is there, but I haven't really read through the whole source) is: first it does a range query on the big CDX index (based on your "match"), then it applies filters/pagination (can't remember which is first, actually)
03:44:37<OrIdow6>"pagination" as in resumeKey
03:46:38HP_Archivist quits [Ping timeout: 258 seconds]
03:48:18<OrIdow6>Without page= and showNumPages= pagination, your "range" includes everything matching the match, but it gets cut off at whatever the limit is; whereas with, it's a fragment that doesn't get cut off
03:49:47<OrIdow6>Also, page= pagination doesn't seem to include as many results as without; what I've seen is that it doesn't include newer ones; my guess, based on something arkiver said at one point, is that page= pagination only gets results from the "'all' index"
03:50:13<OrIdow6>Or at any rate a subset of indexes
04:01:19<@JAA>Hmm
04:01:30<@JAA>I tried poking at the source a bit, but I can't seem to find the relevant parts at all.
04:02:48<@JAA>Doesn't help that some of the stuff are forked repos from iipc, and you can't search in forks on GitHub.
04:13:48qw3rty__ joins
04:16:34<OrIdow6>Yeah, it's not the clearest thing
04:17:16qw3rty_ quits [Ping timeout: 252 seconds]
04:20:57<@JAA>Yeah, I'll stop now, this is too messy.
04:25:33<@JAA>So if I understand you correctly, this means that since https://web.archive.org/cdx/search/cdx?url=reddit.com&collapse=urlkey&matchType=domain&filter=original:^https?://(?:[^/]*\.)?reddit\.com(?::[0-9]*)?/r/.*&limit=100 also returns no results, not a single subreddit URL is in that first block of millions of results from the domain match?
04:27:42<@JAA>I think the index is generally alphabetic, so /user/ pages should come after /r/. I wonder where millions of URLs prior to /r come from.
04:35:11<@JAA>And then I suppose the only way to run my query is to make 357k requests against the API. If I remember correctly from last time I tried to run a significant number of queries, there's quite some rate limiting, so that sucks.
04:35:58<@JAA>Also, I realised what the millions of URLs must be: /api/info.json from #shreddit.
05:12:23<OrIdow6>Looks like it
05:19:14<OrIdow6>And yes
05:32:32<OrIdow6>I did once try to write something to try to cover everything by generating prefixes and running with those as matches, but that didn't work, and I don't remember why
05:35:36<@JAA>Well, you'll always miss some things with such generated prefix searches, namely the shorter matches.
05:37:31<@JAA>I had that issue once on a discovery thing for some site. I think it was Dead Format. It was a search field there, but basically the same thing. When it arrived at searching 'black', there were still too many results, so it started doing 'blacka', 'blackb', etc., obviously missing everything containing the word 'black'.
05:38:30<@JAA>Also, such prefix searches can't work with the matchType=domain obviously, so if subdomains are relevant are not all known, it breaks down.
05:39:28<@JAA>For example, I know of {old,np,i}.reddit.com, but there are probably more. I could catch all of those if the query above worked, but I'd have to know the subdomains first to do a prefix-based search.
05:40:01<@JAA>With that, it'd work well actually in this case since subreddit names also have quite some restrictions. But yeah...
05:40:44<@JAA>In this particular case, I don't need a complete list of all results anyway. I'll probably just sample a bunch of pages and leave it at that.
05:55:52<OrIdow6>What I've seen is that the CDX entries are sorted lexically on urlkey + " " + timestamp (or maybe even the whole line), and so the matches with nothing after what you searched for come first
05:55:58<OrIdow6>But domain etc would be a problem
06:52:30Somebody2 quits [Ping timeout: 265 seconds]
10:12:43Matthww joins
12:00:11HP_Archivist (HP_Archivist) joins
13:04:34HP_Archivist quits [Client Quit]
13:42:07tech234a quits [Ping timeout: 252 seconds]
13:42:22murmur_ joins
13:42:31justcool393_ (justcool393) joins
13:42:33NotEggplant_ joins
13:42:34mgrandi_ (mgrandi) joins
13:42:37HCross_ (HCross) joins
13:42:39tech234a (tech234a) joins
13:42:57hook54321_ (hook54321) joins
13:42:57@ChanServ sets mode: +o hook54321_
13:43:07aarchi_ (aarchi) joins
13:43:34sonick_ (sonick) joins
13:44:15themadpro_ (themadpro) joins
13:44:38justcool393 quits [Ping timeout: 622 seconds]
13:44:38justcool393_ is now known as justcool393
13:45:01@hook54321 quits [Ping timeout: 622 seconds]
13:45:01@hook54321_ is now known as @hook54321
13:45:47themadpro quits [Ping timeout: 622 seconds]
13:45:47themadpro_ is now known as themadpro
13:46:10sonick quits [Ping timeout: 622 seconds]
13:46:10sonick_ is now known as sonick
13:47:19aarchi quits [Ping timeout: 622 seconds]
13:47:19aarchi_ is now known as aarchi
13:47:42murmur quits [Ping timeout: 622 seconds]
13:47:42HCross quits [Ping timeout: 622 seconds]
13:47:42mgrandi quits [Ping timeout: 622 seconds]
13:47:42murmur_ is now known as murmur
13:47:42mgrandi_ is now known as mgrandi
13:47:42HCross_ is now known as HCross
13:48:28NotEggplant quits [Ping timeout: 622 seconds]
13:48:28NotEggplant_ is now known as NotEggplant
13:50:56HP_Archivist (HP_Archivist) joins
14:14:20qwertyasdfuiopghjkl quits [Remote host closed the connection]
14:18:25@hook54321 quits [Ping timeout: 252 seconds]
14:19:06murmur quits [Ping timeout: 265 seconds]
14:19:35NotEggplant quits [Ping timeout: 265 seconds]
14:20:17tech234a quits [Ping timeout: 622 seconds]
14:21:49aarchi quits [Ping timeout: 622 seconds]
14:22:12themadpro quits [Ping timeout: 622 seconds]
14:22:12justcool393 quits [Ping timeout: 622 seconds]
14:22:20sonick_ (sonick) joins
14:22:27themadpro (themadpro) joins
14:22:28aarchi (aarchi) joins
14:22:40HCross_ (HCross) joins
14:22:45tech234a (tech234a) joins
14:22:46hook54321 (hook54321) joins
14:22:46@ChanServ sets mode: +o hook54321
14:23:44sonick quits [Ping timeout: 622 seconds]
14:23:44sonick_ is now known as sonick
14:24:30HCross quits [Ping timeout: 622 seconds]
14:24:30HCross_ is now known as HCross
14:25:16mgrandi quits [Ping timeout: 622 seconds]
14:30:26AlsoHP_Archivist joins
14:33:42HP_Archivist quits [Ping timeout: 258 seconds]
14:34:39NotEggplant joins
14:44:15justcool393 (justcool393) joins
14:46:13HP_Archivist (HP_Archivist) joins
14:49:48AlsoHP_Archivist quits [Ping timeout: 258 seconds]
14:54:06mgrandi (mgrandi) joins
14:54:56HP_Archivist quits [Client Quit]
14:55:13HP_Archivist (HP_Archivist) joins
16:07:17qwertyasdfuiopghjkl joins
18:16:25murmur joins
20:42:20<Ryz>So... was checking https://web.archive.org/web/*/https://specials.ezlinks.com/* as usual on doing ignores on dead websites on ArchiveBot jobs, and I spotted something at the bottom-right, a button that has a magnifying glass and the word 'Help'
20:43:28<Ryz>Here's what it is when not pressed: https://transfer.archivete.am/ZH4Wh/chrome_2021-11-11_12-42-48.png
20:43:47<Ryz>And here it is when pressed: https://transfer.archivete.am/8nY3e/chrome_2021-11-11_12-42-59.png
20:44:37<Ryz>Unsure how helpful this is since I seen a bunch of these but never used 'em
20:45:06<Ryz>Looks like it appeared today since it wasn't at all yesterday or the day before
20:51:05<OrIdow6>???
20:51:10<OrIdow6>There for me too
20:51:33<OrIdow6>It turns from "Help" into "Chat" when you first click on it, then stays as "chat" when minimized, for those wondering
21:43:26<@JAA>Oh fun, the resumeKey pagination is even more broken.
21:43:33<@JAA>Here's a loop: https://web.archive.org/cdx/search/cdx?url=fsc.org&collapse=urlkey&fl=original&matchType=domain&output=json&limit=100&showResumeKey=true&resumeKey=org%2Cfsc%2Cmarketingtoolkit%29%2Fsites%2Fdefault%2Ffiles%2Fs_sdg%2520goals_icons-individual-rgb-05_5.png+20190427122933
21:50:13<@JAA>Anyway, I've rewritten ia-cdx-search to use showNumPages now.
22:12:41qwertyasdfuiopghjkl76 joins
22:15:10qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds]
22:15:24qwertyasdfuiopghjkl76 is now known as qwertyasdfuiopghjkl
22:49:13paul2520 (paul2520) joins
22:52:19paul2520 quits [Remote host closed the connection]
23:46:37qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds]
23:56:03HP_Archivist quits [Ping timeout: 258 seconds]

HomeSearchPrevious dayNext day