HomeSearchPrevious dayNext day

00:42:24AlsoHP_Archivist joins
00:42:32AlsoHP_Archivist quits [Remote host closed the connection]
00:43:44AlsoHP_Archivist joins
00:44:02AlsoHP_Archivist quits [Remote host closed the connection]
00:45:06HP_Archivist quits [Ping timeout: 240 seconds]
00:45:14AlsoHP_Archivist joins
00:45:32AlsoHP_Archivist quits [Remote host closed the connection]
00:46:44AlsoHP_Archivist joins
00:47:02AlsoHP_Archivist quits [Remote host closed the connection]
00:48:15AlsoHP_Archivist joins
00:48:32AlsoHP_Archivist quits [Remote host closed the connection]
00:49:44AlsoHP_Archivist joins
00:50:02AlsoHP_Archivist quits [Remote host closed the connection]
00:51:14AlsoHP_Archivist joins
00:51:32AlsoHP_Archivist quits [Remote host closed the connection]
00:52:44AlsoHP_Archivist joins
00:53:02AlsoHP_Archivist quits [Remote host closed the connection]
00:54:15AlsoHP_Archivist joins
00:54:32AlsoHP_Archivist quits [Remote host closed the connection]
00:55:44AlsoHP_Archivist joins
00:56:02AlsoHP_Archivist quits [Remote host closed the connection]
00:57:14AlsoHP_Archivist joins
00:57:32AlsoHP_Archivist quits [Remote host closed the connection]
01:10:26h3ndr1k quits [Ping timeout: 240 seconds]
01:14:46h3ndr1k (h3ndr1k) joins
01:17:59qwertyasdfuiopghjkl quits [Client Quit]
01:22:57yawkat quits [Ping timeout: 252 seconds]
01:34:11Hackerpcs quits [Quit: Hackerpcs]
01:36:21yawkat (yawkat) joins
01:36:23Hackerpcs (Hackerpcs) joins
02:07:30tzt (tzt) joins
02:21:12<VerifiedJ>Vimeo is being shitty about people using their service and bandwidth again. https://twitter.com/JennyENicholson/status/1493406251408969728
02:32:20tbc1887 (tbc1887) joins
02:54:19shoghicp quits [Ping timeout: 265 seconds]
03:13:56shoghicp joins
03:13:56shoghicp quits [Changing host]
03:13:56shoghicp (shoghicp) joins
03:14:30tbc1887 quits [Read error: Connection reset by peer]
04:20:29Arcorann quits [Read error: Connection reset by peer]
04:26:42Arcorann (Arcorann) joins
04:59:15<@JAA>The 'Free Christian Fundraising Platform' GiveSendGo got hacked, and the data was made available at givesendgone.wtf. I think this gets the AT channel naming seal of approval. :-)
05:01:36<Jake>Hahaha
05:03:22<ivan>:)
05:14:19mutantmonkey quits [Ping timeout: 252 seconds]
05:27:00mutantmonkey (mutantmonkey) joins
05:52:03datechnoman (datechnoman) joins
06:46:57<thuban>JAA, some questions about snscrape again:
06:47:09<thuban>- lj is like reddit in that (a) comments on posts can be hidden in the initial page and expanded via ajax and (b) sometimes we care more about the comments than the posts.
06:47:20<thuban>i've reverse-engineered the api used for retrieving comments, and each comment does have a corresponding permalink. for socialbot usage, would it be appropriate to use a --format parameter that included comment permalinks?
06:47:25<thuban>most journals would have relatively few comments, but popular communities might have several million in total.
06:47:52<thuban>- if so: is https://github.com/JustAnotherArchivist/socialscrape-bot the correct repo? and what's the correct way to actually structure such a --format parameter, given that all the similar ones socialbot currently uses are marked as deprecated by snscrape?
06:48:44<thuban>- the lj api is such that just generating the url for each post would take fewer requests than actually scraping the details of that post (would take fewer requests than scraping the comments).
06:48:59<thuban>would it make sense for the module to have a 'no details' flag (presumably default False for library use but set by the cli if not --format and not --jsonl)?
07:19:25BlueMaxima quits [Read error: Connection reset by peer]
08:42:56@arkiver quits [Remote host closed the connection]
08:44:48arkiver (arkiver) joins
08:44:48@ChanServ sets mode: +o arkiver
09:08:07Sluggs joins
13:52:08Arcorann quits [Ping timeout: 265 seconds]
15:37:18Iki joins
15:39:26AnotherIki quits [Ping timeout: 240 seconds]
15:39:58LeGoupil joins
16:07:35qwertyasdfuiopghjkl joins
16:08:49LeGoupil quits [Ping timeout: 252 seconds]
16:41:18seednode49 quits [Ping timeout: 265 seconds]
16:43:10seednode49 (seednode) joins
16:55:55spirit joins
17:06:56randomusernamegg7 joins
17:08:31LeGoupil joins
19:10:38LeGoupil quits [Client Quit]
19:20:18Craigle quits [Quit: The Lounge - https://thelounge.chat]
19:20:40Craigle (Craigle) joins
19:27:53lunik1 quits [Remote host closed the connection]
19:28:13lunik1 joins
19:42:56lunik1 quits [Client Quit]
19:47:31lunik1 joins
20:05:00Barto quits [Quit: WeeChat 3.4]
20:07:04Barto (Barto) joins
21:30:58BlueMaxima joins
22:09:08<@JAA>thuban: Let's not worry about socialbot and AB (for now). The comments thing is really a general issue. With the existing modules, I went with separate scrapers, although the line is somewhat blurry so far; e.g. TwitterUserScraper for a user's tweets and TwitterTweetScraper for getting replies, but the user scraper also returns self-replies etc. For LJ, that'd mean one scraper that scrapes all posts
22:09:14<@JAA>in a community/user and one scraper that scrapes all comments for a particular post. It's not particularly convenient when you want to chain them though.
22:43:49archiver223 joins
22:44:09archiver223 quits [Remote host closed the connection]
22:48:51<thuban>JAA: i was already planning on implementing separate scrapers in that way, but intended to have the posts scraper run a comments scraper for each post (so that the jsonl output for the post would include all the comments).
22:49:27<thuban>are you suggesting i shouldn't? that indeed seems inconvenient...
22:52:14<@JAA>I don't like the thought of one gigantic output line for a post with thousands of comments, but I'm not generally opposed to an option that would `yield from` a comments scraper and return them that way, I think.
22:53:53<thuban>as in, an option for the posts scraper that would cause it to emit mixed output of posts and comments?
22:54:09<@JAA>Yeah, somewhat similar to the subreddit scraper.
22:54:26<@JAA>Although actually, what might be even better here is something different.
22:55:54<@JAA>Yield only `Post` items from the community scraper, and the items have a way to call the comments scraper. One way would be to have `Post.comments` be an iterator that then yields the comments. Would need some care with the JSON serialisation though.
22:57:43<@JAA>(The Reddit scraper is actually not a good comparison because it works very differently. Pushshift directly has feeds of all comments in a subreddit, for example.)
22:58:33<thuban>that sounds interesting, but you're right about the serialization. that's handled implicitly by the dataclasses python module, right?
23:00:25<@JAA>No, it's a custom implementation.
23:00:46<@JAA>`snscrape.base._JSONDataclass`
23:00:54BlueMaxima quits [Read error: Connection reset by peer]
23:00:59<thuban>that would explain why i couldn't find it in the dataclasses docs, lol.
23:01:46<thuban>ok, i'll have a look at it.
23:02:01<@JAA>I'll spend some time pondering how to best design this without it becoming a huge mess of deeply interacting code.
23:03:21Arcorann (Arcorann) joins
23:05:50<thuban>speaking of the base scraper, have you considered implementing browser user-agent generation and/or rate-limited requests there, for utility purposes? (i see several of the modules implement them independently.)
23:39:25<@JAA>It's definitely crossed my mind before. I think there was even a TODO in the code at some point. But it wasn't on my radar anymore, so thanks.
23:45:41<thuban>cool cool. want me to open an issue?
23:51:23<s-crypt>is there an area/site to view when the last socialscrape was done for a user like there is the archivebot viewer for sites?
23:58:37<thuban>s-crypt: socialbot just creates a text file of urls and then starts an archivebot job for it, so you should be able to find its results in the general archivebot viewer (text-file jobs have the filename appended to the domain, so it's searchable)
23:59:37<@JAA>The viewer doesn't index files that contain an @, so all of those are missing.
23:59:56<@JAA>IRC log grepping is the only reasonable way currently.

HomeSearchPrevious dayNext day