| 00:08:53 | | marked1957 quits [Client Quit] |
| 00:09:27 | | marked1957 joins |
| 00:18:13 | | Hackerpcs (Hackerpcs) joins |
| 00:24:35 | | Arcorann_ joins |
| 00:50:33 | | britmob joins |
| 00:59:21 | <Dallas> | Fanfic.net has moved behind cloudflare anti scrape, you can get round it with the original ip atm, how quickly do you reckon it'd be before I get the ip nulled or an angry email?? |
| 01:04:16 | <Ryz> | Uhh, that website doesn't exist...? |
| 01:06:56 | <@JAA> | Yeah, fanfic.net is a parked domain. |
| 01:15:01 | <Dallas> | sorry sorry my bad |
| 01:15:02 | <Dallas> | https://www.fanfiction.net/ |
| 01:15:18 | <Dallas> | I text type to easily, apologies |
| 01:18:38 | <@JAA> | No Cloudflare here. |
| 01:21:27 | <Dallas> | Oh I hit this on every page :/ https://up.dallasthe.dev/WindowsTerminal_F9zXNkR4ji.png |
| 01:29:00 | <@JAA> | Try a browser UA? |
| 01:34:34 | | second (second) joins |
| 01:36:57 | | sec^nd quits [Ping timeout: 240 seconds] |
| 01:36:57 | | second is now known as sec^nd |
| 01:51:35 | | wickedplayer494 quits [Read error: Connection reset by peer] |
| 01:51:54 | | wickedplayer494 joins |
| 02:00:02 | | Dallas quits [Client Quit] |
| 02:03:53 | | Dallas (Dallas) joins |
| 02:08:43 | | ave quits [Remote host closed the connection] |
| 02:09:17 | | ave (ave) joins |
| 02:19:20 | <Ryz> | JAA, Dallas, checking with https://subdomainfinder.c99.nl/scans/2020-12-22/fanfiction.net - there is Cloudflare |
| 02:19:35 | <Ryz> | ...And it didn't before in the past: https://subdomainfinder.c99.nl/scans/2020-11-22/fanfiction.net |
| 02:23:04 | <@JAA> | Lord_Nightmare: https://twitter.com/AOC/status/1341145260014100480 |
| 02:24:09 | <@JAA> | Ryz: Well, yes, it uses Cloudflare, but it doesn't have Under Attack mode enabled generally. |
| 02:25:11 | <@JAA> | I hate Cloudflare in general, but only the Under Attack and CAPTCHA modes are the problem archival-wise. |
| 02:46:28 | <hexa-> | JAA: same expectations with the EU-Brexit Deal |
| 03:23:04 | | Arcorann_ quits [Ping timeout: 240 seconds] |
| 04:06:09 | | DogsRNice quits [Read error: Connection reset by peer] |
| 04:46:12 | | qw3rty_ joins |
| 04:50:00 | | qw3rty__ quits [Ping timeout: 252 seconds] |
| 06:42:29 | | Stilett0 joins |
| 06:44:14 | | Stiletto quits [Ping timeout: 252 seconds] |
| 07:50:15 | | Arcorann_ joins |
| 08:29:52 | | godane quits [Client Quit] |
| 08:51:39 | | spirit joins |
| 09:10:31 | | HP_Archivist quits [Client Quit] |
| 09:10:48 | | HP_Archivist (HP_Archivist) joins |
| 10:06:37 | | hooway joins |
| 10:29:58 | | Fusl_ (Fusl) joins |
| 10:29:58 | | @ChanServ sets mode: +o Fusl_ |
| 10:30:21 | | pnJay (pnJay) joins |
| 10:36:05 | | ftl joins |
| 11:26:21 | | wickedplayer494 is now authenticated as wickedplayer494 |
| 12:22:40 | <grawity> | Dallas: I have a bunch of Python-based scrapers for various sites (all using python-requests as the HTTP library), so the last time I had to deal with CF's "Authentication needed", I just changed requests.Session() to cloudscraper.CloudScraper() and it handled all the ugly stuff automatically |
| 14:01:10 | | vitzli (vitzli) joins |
| 14:03:36 | | vitzli quits [Client Quit] |
| 14:50:36 | | Arcorann_ quits [Ping timeout: 252 seconds] |
| 16:02:48 | <Dallas> | Ah I see, I'll try some user agents and see if it gets me around the captcha if not I'd love to give that a go grawity |
| 16:08:13 | <Dallas> | Ryz that site's been running on a pretty basic setup no cdn or anything for years and dropped a CF update a couple day's ago breaking a load of peoples bots :/ |
| 16:10:56 | <@JAA> | grawity: Interesting, and they have a pure-Python parser as well. As far as I can see though, that's also for the old challenges like my tool, not the updated ones. Haven't tried it though. |
| 16:26:07 | <Dallas> | I hit captcha wit a UA as well, I may just try running an archive now using the pinned host and hope it finishes before they null route it lol |
| 16:28:38 | <Dallas> | The save page now button on https://web.archive.org/ still can grab a page perfectly, do they have some sorta whitelist deal with CF? |
| 16:30:29 | <@JAA> | Maybe there is some IP-based blocking involved. As I said last night, I'm not getting the Attack mode challenge from my residential connection. |
| 16:30:49 | <@JAA> | I am getting it from an OVH IP with Firefox UA though. |
| 16:34:30 | | tapos quits [Client Quit] |
| 16:35:21 | | tapos joins |
| 16:35:40 | <Dallas> | I'm blocked on residential and my hetzner ip ffs |
| 16:35:59 | <Dallas> | Glad CF hate me in particular lol |
| 16:52:40 | | Hecz quits [Remote host closed the connection] |
| 16:52:49 | | artman joins |
| 16:53:57 | | sec^nd quits [Ping timeout: 240 seconds] |
| 16:54:10 | | cadence quits [Ping timeout: 252 seconds] |
| 16:54:11 | | Terbium quits [Ping timeout: 252 seconds] |
| 16:54:40 | | Terbium joins |
| 16:56:06 | | cadence (cadence) joins |
| 16:56:13 | | sec^nd (second) joins |
| 16:58:00 | | Hecz joins |
| 16:58:01 | | Hecz is now authenticated as Hecz |
| 16:58:01 | | Hecz quits [Changing host] |
| 16:58:01 | | Hecz (Hecz) joins |
| 17:05:10 | | cadence quits [Ping timeout: 252 seconds] |
| 17:05:32 | | cadence (cadence) joins |
| 17:13:13 | | thuban quits [Quit: What price Glory ?] |
| 17:16:13 | | thuban joins |
| 17:49:00 | <Dallas> | Is wpull the best way to try grabbing a site atm ?? |
| 18:03:44 | | VADemon joins |
| 18:12:49 | | tapos quits [Client Quit] |
| 18:36:22 | <@JAA> | Depends on your definitions of 'best' and 'grabbing'. |
| 18:37:00 | | tapos joins |
| 18:37:01 | <@JAA> | Ignoring the bugs, it's probably the best way to do recursive crawls. |
| 19:41:26 | | tapos quits [Client Quit] |
| 20:10:08 | | artman quits [Ping timeout: 244 seconds] |
| 21:48:37 | <atphoenix> | High stability, high capacity storage, in quartz glass. Expected to reach to 360 TB per 12 cm disc. https://www.5dmemorycrystal.com/technology/ |
| 21:49:05 | <atphoenix> | Can't happen soon enough |
| 21:50:32 | | lunik1 quits [Quit: :x] |
| 21:52:45 | | lunik1 joins |
| 22:50:06 | <@JAA> | ETA: 2050 |
| 22:53:43 | | Arcorann_ joins |
| 23:16:14 | | Doranwen quits [Quit: bbl] |
| 23:45:00 | | VADemon quits [Read error: Connection reset by peer] |