HomeSearchPrevious dayNext day

00:06:00geezabiscuit quits [Read error: Connection reset by peer]
00:06:08geezabiscuit (geezabiscuit) joins
00:07:48wessel1512 quits [Remote host closed the connection]
00:08:11wessel1512 joins
01:16:26Arcorann quits [Ping timeout: 240 seconds]
02:03:06dm4v quits [Ping timeout: 240 seconds]
02:04:28dm4v joins
02:04:30dm4v quits [Changing host]
02:04:30dm4v (dm4v) joins
02:11:17HP_Archivist (HP_Archivist) joins
02:11:22HP_Archivist quits [Client Quit]
02:48:49DogsRNice (Webuser299) joins
03:38:02DogsRNice quits [Read error: Connection reset by peer]
03:57:38sonick quits [Client Quit]
04:19:18Arcorann (Arcorann) joins
04:20:24<@JAA>lol, s.mil.ru has a revoked TLS certificate.
04:22:00<@JAA>Revoked a little over 5 hours ago, apparently.
05:26:50Arcorann quits [Ping timeout: 265 seconds]
07:43:16Niklink quits [Ping timeout: 265 seconds]
08:07:20Arcorann (Arcorann) joins
08:46:27wessel1512 quits [Ping timeout: 265 seconds]
09:05:29sonick (sonick) joins
09:15:17<T31M>probably related the the new .ru root cert that they are pushing for "local users" ? https://www.bleepingcomputer.com/news/security/russia-creates-its-own-tls-certificate-authority-to-bypass-sanctions/
09:15:35<T31M>"Currently, the only web browsers that recognize Russia’s new CA as trustworthy are the Russia-based Yandex browser and Atom products, so Russian users are told to use these instead of Chrome, Firefox, Edge, etc."
09:30:49driib (driib) joins
09:57:06lunik1 quits [Client Quit]
09:57:55lunik1 joins
10:02:25lunik1 quits [Client Quit]
10:04:47lunik1 joins
10:08:14syntaxx quits [Quit: The Lounge - https://thelounge.chat]
10:08:25syntaxx (syntaxx) joins
10:17:56lunik1 quits [Ping timeout: 265 seconds]
10:26:36lunik1 joins
10:42:18monoxane4 quits [Client Quit]
10:55:57lunik1 quits [Client Quit]
11:05:39lunik1 joins
11:15:17sonick quits [Client Quit]
11:46:01sonick (sonick) joins
12:28:21chrismeller (chrismeller) joins
13:38:24mjh quits [Read error: Connection reset by peer]
13:49:30Arcorann quits [Ping timeout: 265 seconds]
13:54:29mjh (mjh) joins
14:15:58qw3rty quits [Quit: Leaving]
14:22:38Niklink joins
14:32:31Ruthalas quits [Ping timeout: 265 seconds]
14:34:50Ruthalas (Ruthalas) joins
14:47:09mjh quits [Ping timeout: 265 seconds]
14:51:00mjh (mjh) joins
15:04:55benjinsmith joins
15:07:10benjinsmith is now known as benjinis
15:07:19benjins quits [Ping timeout: 265 seconds]
15:07:27benjinis is now known as benjins
16:02:33qwertyasdfuiopghjkl quits [Ping timeout: 265 seconds]
16:03:10<@JAA>T31M: No, it's a Let's Encrypt certificate, and that individual cert got revoked. The domain is a Bitly alias, FWIW.
16:33:24<audrooku|m>is it A. possible to use grab-site with a proxy (I tried passing args to wpull and it gave me an operation not permitted error) and B. possible to resume a grab that finished with a bunch of errors because I got IP-banned
16:33:55sec^nd quits [Remote host closed the connection]
16:35:03sec^nd (second) joins
16:35:31<audrooku|m>E.G. retry and recrawl all the non-200 page grabs
16:41:24<IDK>is #robloxd the place for roblox discussion
16:41:31<IDK>I think I might know some information
16:41:42chrismeller quits [Ping timeout: 265 seconds]
16:58:56Jack_Thompson joins
17:37:29LeGoupil joins
17:37:33<@JAA>audrooku|m: Proxies are generally a bad idea for archival. Resumption isn't implemented as far as I know.
18:12:26benjinsmith joins
18:14:51benjins quits [Ping timeout: 265 seconds]
18:16:30<audrooku|m>thanks for the reply, JAA why are they? would a VPN be bad for the same reason? I'd at consider it better than every page being "sorry, {IP} is banned..."
18:16:43<audrooku|m>* I'd at least consider it
18:17:10<audrooku|m>s/thanks for the reply, JAA why are they? would a VPN be bad for the same reason? I'd at consider it better than every page being "sorry, {IP} is banned..."/thanks for the reply JAA, why are they? would a VPN be bad for the same reason? I'd at least consider it better than every page being "sorry, {IP} is banned..."/
18:17:45<@JAA>Great work on that edit, Matrix bridge. lol
18:18:24<@JAA>Well, you want a clean internet connection for archival. No NXDOMAIN hijacking, no MITM manipulation of data, etc. Proxies frequently do that shit.
18:20:33<audrooku|m>Ah I see, that's not great. What about VPNs?
18:24:40<audrooku|m>(and I assume the solution for ip-bans for most people here is to just spin up another server)
18:27:06<@JAA>Same deal with VPNs generally. Without knowing their network etc. or very extensive testing, it's not really possible to know whether it's clean.
18:30:22benjinsmith quits [Remote host closed the connection]
18:31:56benjinsmith joins
18:32:23<audrooku|m>Got it, in that case I'll just spin up a server. Maybe it'll be faster than my home connection and I'll get a full rip before the moderators notice.
18:35:36benjinsmith quits [Read error: Connection reset by peer]
18:36:38benjinsmith joins
18:38:19wessel1512 joins
19:31:12benjinss joins
19:32:40benjinsmith quits [Ping timeout: 265 seconds]
19:58:13qwertyasdfuiopghjkl joins
20:18:58<masterX244>had that fun once where enumeration of a site got me IP-banned. their POST-based pagination was inefficient
20:19:28<masterX244>ran that enumeration via TOR to get a URL list which i ran from a hetzner VM
20:21:40LeGoupil quits [Client Quit]
20:21:53LeGoupil joins
20:22:03JackThompson joins
20:23:09benjinss quits [Read error: Connection reset by peer]
20:25:11benjinss joins
20:25:34benjinss quits [Remote host closed the connection]
20:27:11benjinss joins
20:28:40Jack_Thompson quits [Client Quit]
20:53:41<audrooku|m>my server is low spec and gets stuck running setup.py for lxml and I'm not quite sure what to do about it
20:54:08nirv quits [Read error: Connection reset by peer]
20:56:31<@JAA>Is it stuck or just busy compiling the C stuff in the background?
20:59:09<immibis>audrooku|m: IP bans are more likely automatic. Only the most obsessive admin is scrolling through logs to find high usage IPs
20:59:23<immibis>it's more likely that you should crawl *slower* and not get banned at all
20:59:53<immibis>an admin will only look at it if you cause a problem or (less likely) if you stick out on a graph
21:03:14<audrooku|m>nope, maybe I could've stood to go slower but this forum is very old and they definitely do it manually, they have a couple dozen mods
21:03:34<audrooku|m>I crawled it for a few hours and didn't get banned until sometime when I went to bed
21:03:41<audrooku|m>I added a decent delay between requests
21:04:10<audrooku|m>it's a forum
21:04:59<audrooku|m>people who run them just seem to do stuff like that, maybe i'm wrong but it looks like it's from the early 2000s
21:06:24<audrooku|m>doesn't respond to ctrl c or further ssh stuff so don't know, I made a large swapfile so hopefully that helps
21:06:50<audrooku|m>s/stuff/sessions/, s/don't/probably/, s/know/stuck/
21:07:22<appledash>Forum software is often pretty bad and inefficient so it is very possible you were impacting their site
21:08:45<immibis>well that sounds like a problem with the server you are sshing to, not the forum that server is scraping
21:11:34<appledash>(I was referring to the reason they got IP banned)
21:34:18<immibis>and I was referring to them not being able to make any more ssh sessions
21:36:47<audrooku|m>correct, it was an issue with my server, but I was responding with that to the person who asked about whether it was stuck or busy compiling C code. I used Element's reply feature but I assume you can't see that if you're using IRC directly
21:39:14<@JAA>Yup, that's not relayed at all, so best to mention the nick normal IRC style instead.
21:43:40benjinss is now known as benjins
21:45:15LeGoupil quits [Client Quit]
21:46:59<audrooku|m>I try to.
21:46:59<audrooku|m>appledash: I can see that being a possibility, though they were also using local hosting and not something like AWS, so I could also see insufficient hosting being an issue. I'll increase the time between requests. (20ms to 65ms, with my 5 concurrent connections).
21:46:59<audrooku|m>In regards to the lxml issue the swapdisk allowed it to finish the install, now I just need to figure out why it's trying to use the wrong version of python and erroring out when it hits a line of code using asyncio...
21:49:37<@JAA>ludios_wpull works on newer Python versions, doesn't it?
21:50:15<@JAA>And that's what grab-site uses.
21:50:23<@JAA>wpull proper requires Python 3.6 or lower.
21:50:33<@JAA>It's in dire need of some love.
21:51:47<audrooku|m>Hmm, I followed the instructions as per the repo and allegedly it works if you do that.
21:51:47<audrooku|m>grab-site worked that on my WSL install when I installed it yesterday, I'll try that wpull fork if I can't get it to run by changing the python version, thanks JAA.
21:51:57<audrooku|m>the grab-site repo*
22:05:20tbc1887 (tbc1887) joins
22:07:33immibis quits [Remote host closed the connection]
22:08:43immibis joins
22:21:33<TheTechRobo>audrooku|m: grab-site should automatically use ludios, though
22:21:40<TheTechRobo>ludios_wpull was made specifically for grab-site
22:22:32<audrooku|m>Huh... don't know what it is then... this is annoying..
22:22:43<audrooku|m>I read that just now
22:23:09<TheTechRobo>What was the problem? Invalid syntax thanks to @asyncio.async?
22:23:25<TheTechRobo>If so, ?????????
22:23:29<audrooku|m>yep
22:23:44tbc1887 quits [Read error: Connection reset by peer]
22:24:09<audrooku|m>I followed the instructions exactly in the repo, it's done this on all my ubuntu 20 LTS AWS installs. it worked fine on wsl2 though.. really odd
22:24:19<TheTechRobo>That's...Strange
22:25:14qw3rty joins
22:28:20<@JAA>Uh yeah, grab-site's setup.py directly references ludios_wpull.
22:30:20<audrooku|m>TheTechRobo: it might be because they both froze when installing lxml, then I had to hard reboot and try again in those cases.
22:30:20<audrooku|m>I reinstalled the OS and preemptively created a swapfile and this install is working.
22:30:52<audrooku|m>created a swapfile before following the instructions in the repo so I wouldn't have to kill this during setup as well*
22:30:59<TheTechRobo>audrooku|m: Still don't know why that would cause -this-
22:31:12<TheTechRobo>Do you have vanilla wpull installed too or something?
22:31:25<audrooku|m>Nothing of the sort
22:31:31<audrooku|m>it must've just butchered the install when it did that
22:31:33<audrooku|m>not sure why
22:31:41<audrooku|m>but I've got it running now. it was on fresh ubuntu installs
22:32:13<@JAA>Should be an ancient pip version either then. I think the syntax used should be supported for ages already anyway.
22:32:24<@JAA>The syntax in setup.py, to be clear.
22:32:49<@JAA>Shouldn't*
22:34:12<audrooku|m>if you're referring to the async.io error, that wasn't during setup and was when I ran grab-site afterwards (after running the setup *again*), if you're referring to the install breaking if it hung and I rebooted then I don't have anything to add to that conversation.
22:35:37BlueMaxima joins
22:38:52<@JAA>Well, both, kind of. Specifically why you got the normal wpull (which has the asyncio error under 3.7+) from the installation of g-s.
22:40:22<@JAA>Also, I wrote that without fully reading what you wrote. Just ignore me. :-)
22:48:32syntaxx quits [Client Quit]
22:57:31Arcorann (Arcorann) joins

HomeSearchPrevious dayNext day