| 01:29:22 | | Iki quits [Ping timeout: 265 seconds] |
| 01:33:00 | <balrog> | OrIdow6: usually finding downloads or documentation under a website |
| 01:33:06 | <balrog> | CDX API ... can that get more than 100k? |
| 01:36:14 | <@JAA> | The CDX API will return everything that's in the WBM index. I used it to list all of youtube.com for #down-the-tube a few weeks ago, for example. |
| 01:37:42 | <@JAA> | My script for that (with pagination and rate limit handling etc.) is here: https://gitea.arpa.li/JustAnotherArchivist/little-things/src/branch/master/ia-cdx-search |
| 01:59:18 | | qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds] |
| 02:14:57 | | qwertyasdfuiopghjkl joins |
| 03:17:37 | | Larsenv quits [Quit: ZNC 1.8.2+deb1+focal2 - https://znc.in] |
| 03:31:14 | | Iki joins |
| 04:30:42 | | qw3rty__ joins |
| 04:31:13 | <@JAA> | Huh, new IA crawl project? https://archive.org/details/mega-001 |
| 04:34:29 | | qw3rty_ quits [Ping timeout: 265 seconds] |
| 04:35:07 | <OrIdow6> | Interesting, wonder what the difference is |
| 04:35:33 | <OrIdow6> | https://archive.org/download/MEGA001-20211206145710-crawl504/MEGA001-20211206145710-crawl504_meta.xml suggests they're still using Heretrix |
| 04:35:44 | <OrIdow6> | *Heritrix |
| 04:36:48 | <@JAA> | I'm guessing it's a different config or seed list. |
| 04:36:51 | <OrIdow6> | Oh, it's on the details page too |
| 04:36:53 | <OrIdow6> | Yeah |
| 04:39:22 | <OrIdow6> | But how so |
| 05:40:26 | | DogsRNice quits [Read error: Connection reset by peer] |
| 09:36:55 | | knecht420 quits [Client Quit] |
| 09:37:32 | | knecht420 (knecht420) joins |
| 11:23:30 | | qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds] |
| 13:21:21 | | Arcorann__ quits [Ping timeout: 258 seconds] |
| 15:19:33 | | s-crypt1 (s-crypt) joins |
| 15:21:41 | | s-crypt quits [Ping timeout: 244 seconds] |
| 15:21:42 | | s-crypt1 is now known as s-crypt |
| 15:26:37 | | s-crypt2 (s-crypt) joins |
| 15:29:00 | | s-crypt quits [Ping timeout: 258 seconds] |
| 15:29:00 | | s-crypt2 is now known as s-crypt |
| 15:41:58 | | s-crypt quits [Ping timeout: 265 seconds] |
| 15:42:27 | | kiska quits [Ping timeout: 265 seconds] |
| 15:44:23 | | flashfire42 quits [Ping timeout: 265 seconds] |
| 15:46:07 | | s-crypt (s-crypt) joins |
| 15:56:13 | | s-crypt quits [Ping timeout: 258 seconds] |
| 15:57:02 | | flashfire42 (flashfire42) joins |
| 15:58:21 | | s-crypt (s-crypt) joins |
| 16:00:21 | | kiska (kiska) joins |
| 19:19:22 | | Stiletto quits [Remote host closed the connection] |
| 19:43:51 | | Stiletto joins |
| 22:06:07 | | Stiletto quits [Remote host closed the connection] |
| 22:08:50 | | Arcorann__ joins |
| 22:15:42 | | Stiletto joins |