| 04:31:05 | | pabs quits [Quit: Don't rest until all the world is paved in moss and greenery.] |
| 04:34:09 | | pabs (pabs) joins |
| 06:30:06 | | Iki quits [Read error: Connection reset by peer] |
| 07:15:24 | | qwertyasdfuiopghjkl quits [Client Quit] |
| 07:17:11 | | qwertyasdfuiopghjkl joins |
| 08:00:18 | | sonick quits [Client Quit] |
| 09:46:08 | | sonick (sonick) joins |
| 11:49:08 | | Iki joins |
| 13:20:27 | | Arcorann quits [Read error: Connection reset by peer] |
| 14:07:21 | <Iki> | I'm trying to find an easy method to list all subdomains of a host using the CDX API |
| 14:08:06 | <Iki> | Example queries: web.archive.org/cdx/search/cdx?url=exampleurl.com/archive&matchType=domain&limit=10000 OR http://web.archive.org/cdx/search/cdx?url=*.exampleurl.com&limit=10000 |
| 14:08:27 | <Iki> | I'm doing this based on the documentation here: https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md |
| 14:08:47 | <Iki> | But I think I'm misunderstanding how to make these queries |
| 14:09:41 | <Iki> | Is there a correct way to phrase these queries? Am I misunderstanding the function of the CDX server? And is there another way to do this that I might consider? |
| 14:09:53 | <Iki> | Thanks :) |
| 14:30:59 | | Sluggs quits [Ping timeout: 244 seconds] |
| 14:32:16 | | Sluggs joins |
| 15:11:02 | | spirit quits [Client Quit] |
| 15:27:53 | <Iki> | On the other hand: I was trying to find a way to do this nicely via the API, but I can just as well simply request all responses and do some post-processing to extract unique subdomains. A little messier, but not a huge problem as long as he CDX server doesn't choke on a large site, I think |
| 15:28:19 | <Iki> | Still curious if there's a less brute-force way, though |
| 16:47:04 | <@JAA> | Iki: matchType=domain (explicit or implied by url=*.x) returns all known URLs under that domain, not just the domains themselves. |
| 16:47:38 | <@JAA> | You could use a filter to only return the homepages of each subdomain, but that then only returns subdomains for which the homepage was grabbed, obviously. |
| 16:47:51 | <@JAA> | Otherwise, you need to list them all and then extract the domains from that. |
| 17:18:36 | | yano quits [Quit: WeeChat, the better IRC client, https://weechat.org/] |
| 17:21:00 | | yano (yano) joins |
| 17:34:34 | | knecht420 quits [Client Quit] |
| 17:35:51 | | knecht420 (knecht420) joins |
| 17:43:14 | <Iki> | Thanks! |
| 21:33:57 | | Larsenv quits [Quit: ZNC 1.8.2+deb1+focal2 - https://znc.in] |
| 21:36:21 | | Larsenv (Larsenv) joins |