HomeSearchPrevious dayNext day

00:35:58HP_Archivist (HP_Archivist) joins
01:00:25hooway quits [Client Quit]
01:47:30qw3rty_ quits [Ping timeout: 264 seconds]
01:47:39qw3rty joins
01:48:31<@JAA>https://forums.tesla.com/ will become read-only on the 15th and probably disappear at some point.
01:50:49<@JAA>Looks like they run Vanilla. AB has its issues with that, mainly due to the JS extraction.
01:59:52ddd joins
02:14:05<purplebot>Deathwatch edited by Switchnode (+255, /* 2021 */ add forums.tesla.com …) just now -- https://www.archiveteam.org/?diff=46396&oldid=46390
02:24:10<Megame>Ge.tt is deleting all content March 10
02:24:12<Megame>http://ge.tt/
02:26:42<@JAA>Yes, we've been working on that one already.
02:28:09<Megame>(y)
02:49:21Arcorann joins
02:52:54Arcorann_ quits [Ping timeout: 264 seconds]
03:08:01DogsRNice quits [Read error: Connection reset by peer]
03:15:38lennier2 joins
03:19:55lennier1 quits [Ping timeout: 264 seconds]
03:19:58lennier2 is now known as lennier1
03:44:14Mineroboter joins
03:44:54Minerobo- joins
03:45:08Mineroboter_ quits [Ping timeout: 250 seconds]
03:49:18Mineroboter quits [Ping timeout: 264 seconds]
04:35:30qw3rty_ joins
04:39:42qw3rty quits [Ping timeout: 264 seconds]
04:40:49ddd quits [Remote host closed the connection]
04:47:54<@JAA>I'm grabbing the Tesla forums now (thread page HTML only). ETA is ~16 hours. I wrote a generic Vanilla qwarc spec file for this which can be reused for other such forums in the future (provided it isn't an ancient version).
05:22:30Ryz quits [Remote host closed the connection]
05:23:06Ryz (Ryz) joins
05:25:18etnguyen03 quits [Client Quit]
05:40:07postpwnk (postpwnk) joins
05:49:43<thuban>JAA: cool!
05:51:24<thuban>does that actually avoid js issues? i'm not really familiar with the differences between qwarc/aiohttp and archivebot/wpull
05:52:23<@JAA>Yeah, qwarc works completely differently than wpull. It's not a recursive crawler at all. Rather, it's a framework for grabbing *stuff*, and you need to specify exactly what (which URLs) you want to grab, how to process their content to discover more stuff, etc.
05:53:07<@JAA>In this case, I'm just using JSON-LD and the <link rel="next" /> element to grab all thread pages.
05:54:05<thuban>ahh, i see. neato
05:54:35<@JAA>So you can't just throw qwarc at a random site but rather need to write the code specifically for that site (or its software). But then it's much, *much* more efficient than a dumb recursive crawl.
05:54:42<thuban>do you have spec files uploaded anywhere? i'm curious what that looks like
05:54:55<@JAA>I've done 2k requests per second across 3 machines with qwarc on a project before.
05:55:45<@JAA>https://transfer.notkiska.pw/1pAVX/vanilla-forums.py
05:56:01<@JAA>(It's also self-documenting and will add that file to the -meta.warc.gz for future reference.)
05:57:01<@JAA>It's really just Python code that has a particular structure (qwarc.Item subclasses and an optional specDependencies global variable), which then gets imported by the qwarc CLI and processed accordingly.
05:58:21<Jake>2k requests/s is pretty impressive!
05:58:55<@JAA>Yeah, that one was fun. Only works if the target site has no rate limits, obviously.
06:00:52<tech234a>Might be worth archiving (or at least keeping an eye on) some of the Adobe product support forums at https://community.adobe.com/ ... they still have forums for discontinued products such as Flash, Flash Builder, Air, etc.
06:01:21<@JAA>I'm doing about 7 req/s on the Tesla forums now as they do have rate limiting and immediately blocked me when I tried with 10 connections earlier.
06:02:21<@JAA>tech234a: Agreed. Also, fuck Lithium...
06:03:14<tech234a>Is Lithium the forum software?
06:03:58<@JAA>Yeah
06:04:30<@JAA>Causes even more issues than Vanilla with recursive crawls.
06:07:31<@JAA>Looks like threads can be enumerated though (forum and thread slugs can be anything, redirects to the right ones), and it also has <link rel="next" /> for multi-page threads. So should be easy enough.
06:07:57<@JAA>At least for a basic content archive, not necessarily browsable though.
06:14:30katocala quits [Ping timeout: 264 seconds]
06:14:55katocala joins
06:28:15<@JAA>Jake: The key to achieving that, by the way, is high concurrency via asyncio and zero parsing. I do all my processing with bytes/string methods (mostly find), very rarely with regex, which is already slow but still much better than any HTML parser. qwarc itself only does minimal processing, basically just writing to WARC and some basic HTTP status code stuff for following redirects or retrying errors
06:28:21<@JAA>(which is also fully customisable). That drives down CPU usage massively and allows for 600 req/s on a shitty ancient i3-2130 that I mainly use for this.
06:30:36<tech234a>JAA: cool, thanks for looking into it!
06:45:17Megame quits [Remote host closed the connection]
06:49:54Doranwen quits [Ping timeout: 264 seconds]
06:54:09Doranwen (Doranwen) joins
07:03:16<mgrandi>It's possible to use asyncio with a process worker or whatever it's called to farm out parse calls, but ideally someone should make an asyncio interface to lxml / bs4...
07:04:58<mgrandi>Also we might want to grab the tweets from https://twitter.com/officialmcafee , see https://arstechnica.com/tech-policy/2021/03/feds-indict-john-mcafee-for-cryptocurrency-pump-and-dump-fraud/
07:05:39<mgrandi>I checked the logs and we grabbed his tweets on October of last year , however I don't know how to do a snscrape with a time range
07:09:22<@JAA>mgrandi: asyncio doesn't make sense with HTML parsing. asyncio solves the case where you need to wait for a response from an I/O device (most commonly network, could also be disk). You can use the time spent there to do other work instead of just sitting around. But parsing's a CPU-heavy task. Short of adding more CPUs, you can't accelerate it really.
07:09:46<@JAA>(I mean, apart from using more efficient parser code, obviously.)
07:11:10<mgrandi>Well, asyncio works with multiple processes too, but I meant in terms of not blocking the entire event loop waiting for lxml to finish parsing
07:11:50<@JAA>Yeah, but that only matters if your program is single-threaded. qwarc supports running multiple processes in parallel, so you can easily saturate all cores.
07:12:39<@JAA>And even if it didn't support that, you could run multiple independent projects at the same time and saturate it that way.
07:13:02<@JAA>(Which is exactly what ArchiveBot pipelines do because wpull is effectively single-threaded.)
07:13:12<mgrandi>What exactly is qwarc? Or a github link?
07:13:38<@JAA>See above for a broad explanation. https://git.kiska.pw/JustAnotherArchivist/qwarc
07:14:57godane1 joins
07:15:40<mgrandi>Ok just making sure
07:17:17<mgrandi>And wpull is single threaded but attempts to (and is broken atm) to use multiple concurrent connections stuff though right
07:17:28godane quits [Ping timeout: 250 seconds]
07:18:21wickedplayer494 quits [Remote host closed the connection]
07:18:34<@JAA>Yeah, through asyncio, which spawns threads in the background for the network stuff but is still effectively single-threaded.
07:18:47<@JAA>And it isn't broken, only the --concurrency option on the CLI is.
07:18:56<@JAA>(You can also set it via a plugin.)
07:20:32<mgrandi>Have you tried using aiohttp?
07:20:53<mgrandi>Oh it's not broken, ah ok
07:21:56<@JAA>qwarc is using aiohttp.
07:22:13<@JAA>But aiohttp doesn't expose the raw data stream, so qwarc uses awful hacks to get its hands on it.
07:23:46<@JAA>mgrandi: Done re https://twitter.com/officialmcafee
07:23:58Arcorann quits [Ping timeout: 250 seconds]
07:25:19<@JAA>Again, the network part is a tiny part of wpull's CPU usage (exceptions confirm the rule). Mostly it's the HTML parsing, poor DB performance/design, and on large crawls Python's horrible cookie jar implementation.
07:25:55<@JAA>The latter two can be solved, the first one is much harder.
07:26:53<@JAA>But I plan to revisit all of that again soon™.
07:32:23<mgrandi>Yeah, I thought the network part was broken, not just the cli
07:35:16wickedplayer494 joins
07:41:54Arcorann joins
07:48:23BlueMaxima quits [Read error: Connection reset by peer]
08:00:53ragu__ joins
08:04:54ragu_ quits [Ping timeout: 264 seconds]
08:19:08<Jake>(sorry a bit late, but I checked out the repo, super cool stuff!!!)
08:23:23ragu__ quits [Read error: Connection reset by peer]
09:07:24hooway joins
09:14:06spirit2 joins
09:15:46Arcorann quits [Ping timeout: 250 seconds]
09:24:05spirit2 quits [Client Quit]
09:26:05<purplebot>Webzdarma edited by Sanqui (+75, webzdarma_searchenginescraper1_05 …) just now -- https://www.archiveteam.org/?diff=46397&oldid=46392
09:29:10OrIdow6 quits [Quit: Quitting.]
09:30:29OrIdow6 (OrIdow6) joins
09:43:19yawkat` quits [Ping timeout: 264 seconds]
09:46:38yawkat (yawkat) joins
10:18:52Arcorann joins
10:53:48sliccricc quits [Remote host closed the connection]
10:54:07sliccricc (sliccricc) joins
11:02:55Arcorann quits [Remote host closed the connection]
11:03:23Arcorann joins
11:05:29sliccricc quits [Remote host closed the connection]
11:05:30godane1 quits [Ping timeout: 264 seconds]
11:05:47sliccricc (sliccricc) joins
12:10:43Arcorann_ joins
12:14:30Arcorann quits [Ping timeout: 264 seconds]
12:23:17Arcorann_ quits [Read error: Connection reset by peer]
12:23:40Arcorann_ joins
14:06:57groente_1 quits [Quit: leaving]
14:12:27sliccricc quits [Remote host closed the connection]
14:12:44sliccricc (sliccricc) joins
14:23:31pilgrim quits [Ping timeout: 252 seconds]
14:27:27sliccricc quits [Remote host closed the connection]
14:27:41sliccricc (sliccricc) joins
14:33:50Arcorann_ quits [Ping timeout: 250 seconds]
14:44:47Justin[home] joins
14:49:19DopefishJustin quits [Ping timeout: 264 seconds]
15:28:21onetruth quits [Client Quit]
15:51:14Trieste joins
16:10:53ddd joins
16:13:04ddd quits [Remote host closed the connection]
16:36:46Megame joins
19:05:13AlsoHP_Archivist joins
19:06:07HP_Archivist quits [Ping timeout: 264 seconds]
19:06:12AlsoHP_Archivist quits [Client Quit]
19:06:28HP_Archivist (HP_Archivist) joins
19:06:54<@JAA>My Tesla forums grab finished about half an hour ago without any issues. :-) 155k thread pages from 113k threads containing just under 2 million comments, which seems to match the numbers on the homepage pretty well.
19:09:06AlsoHP_Archivist joins
19:12:06HP_Archivist quits [Ping timeout: 264 seconds]
19:18:42AlsoHP_Archivist quits [Ping timeout: 264 seconds]
19:19:10AlsoHP_Archivist joins
19:22:36AlsoHP_Archivist quits [Client Quit]
19:31:24HP_Archivist (HP_Archivist) joins
19:54:51ddd joins
20:02:57HP_Archivist quits [Client Quit]
20:47:52Ruthalas quits [Quit: END OF LINE]
20:53:20HackMii_ (hacktheplanet) joins
20:54:07HackMii quits [Ping timeout: 264 seconds]
20:54:07HackMii_ is now known as HackMii
21:17:50DogsRNice (Webuser299) joins
21:27:49<atphoenix>thanks for grabbing that
21:35:48spirit quits [Client Quit]
21:37:29HP_Archivist (HP_Archivist) joins
21:48:43tzt quits [Ping timeout: 264 seconds]
22:01:59onetruth joins
22:10:09onetruth quits [Client Quit]
22:11:14tzt joins
22:15:55HackMii quits [Remote host closed the connection]
22:16:02etnguyen03 (etnguyen03) joins
22:17:06HackMii (hacktheplanet) joins
22:37:51ddd quits [Client Quit]
22:39:24BlueMaxima joins
22:47:27Ruthalas (Ruthalas) joins
23:11:25Arcorann_ joins
23:40:05sonick joins
23:41:24hooway quits [Client Quit]
23:41:57<sonick>Can you archive Mastodon instances that are scheduled to be shut down?
23:42:56<AK>https://wiki.archiveteam.org/index.php/Mastodon
23:43:06<AK>"Since May 2019, ArchiveTeam no longer archives content from Mastodon instances or accounts unless requested explicitly by the owner(s)."
23:44:02<sonick>Oh, I didn't know that...Thanks.
23:48:58sonick leaves
23:57:45Doran (Doranwen) joins
23:58:18Doranwen quits [Ping timeout: 264 seconds]

HomeSearchPrevious dayNext day