| 00:18:40 | <@JAA> | Yeah, I hate the .jsonlz4 format. There's really no reason why they couldn't have gone with a normal standardised compression format. |
| 00:22:01 | | pabs (pabs) joins |
| 00:23:24 | <@JAA> | TheTechRobo: As I understand it, the X.Org server is (effectively) single-threaded while clients can be multi-threaded depending on their implementation with some restrictions. |
| 00:25:06 | <@JAA> | What do you mean by 'discarding' non-200 responses? Not writing to WARC? Not retry them? |
| 00:25:26 | <TheTechRobo> | JAA: Not write to WARC |
| 00:25:38 | | pabs quits [Remote host closed the connection] |
| 00:26:04 | <TheTechRobo> | In this case, it's not downloading for archival, but for data-miners |
| 00:27:00 | <@JAA> | Not possible (except with monkeypatching, which lets you do anything). |
| 00:27:08 | <TheTechRobo> | Ok, thanks |
| 00:29:09 | <ivan> | I have a script to remove them afterwards |
| 00:29:29 | <@JAA> | I hope that script doesn't use warcio. |
| 00:29:44 | <TheTechRobo> | Why not warcio, out of curiosity |
| 00:30:37 | <@JAA> | It has plenty of issues of not complying with the WARC spec, mangling data, etc. |
| 00:30:57 | <@JAA> | Here's a selection of them: https://github.com/webrecorder/warcio/issues/created_by/JustAnotherArchivist |
| 00:31:20 | <ivan> | https://github.com/ArchiveTeam/greader-grab/blob/master/warc2warc_greader.py it uses Python 2 and hanzo.warctools |
| 00:31:24 | <ivan> | it was very reliable! |
| 00:32:33 | <ivan> | it looks like warc2warc is in https://github.com/internetarchive/warctools |
| 00:34:50 | <@JAA> | ivan: That seems to strip transfer encoding if I'm reading it correctly. It also assumes that the request record comes immediately before the response record, which isn't the case with wpull for example. |
| 00:35:01 | <ivan> | ah, unfortunate |
| 00:35:13 | <ivan> | you're probably the leading WARC scientist now |
| 00:36:48 | <@JAA> | Just a grumpy guy who hates how many people violate the WARC standard. :-) |
| 00:37:20 | <ivan> | someone recently taught me the value of saving data in formats (or at least having available derivations) that easily shared with and viewed by non-technical people |
| 00:37:48 | <ivan> | I was singing the praises of SQL and I came to understand that a SingleFile capture may be more valuable sometimes |
| 00:37:50 | <TheTechRobo> | WARC can derive to that, though |
| 00:37:57 | <ivan> | right, sure |
| 00:38:01 | <TheTechRobo> | You can't derive back to warc unless you have the request stuff |
| 00:38:05 | <@JAA> | Oh, for sure. WARCs are horrible to work with. |
| 00:38:15 | <TheTechRobo> | That too |
| 00:38:18 | <ivan> | well, unless you wanted some post-JS execution state and don't want to re-run the JS with that exact old browser |
| 00:40:01 | <ivan> | Wayback Machine is just like 'surely this JavaScript from 2005 will do the correct thing when run in your latest browser in a shared context at web.archive.org' |
| 00:40:39 | <@JAA> | Yeah, scripts are a mess. And WASM will make it worse. |
| 00:41:03 | <@JAA> | But there isn't really a good solution to that since browsers change all the time. |
| 00:41:51 | <ivan> | if a webpage delivers a 0day for a particular old Chrome, can we archive the experience of getting owned? |
| 00:41:59 | <TheTechRobo> | BTW, why do web-apps not work in the WBM? Is it because resource fetching is wonky because the WBM cant rewrite its links? or is it because the wbm isn't post-capable? |
| 00:42:05 | <TheTechRobo> | ivan: lol |
| 00:42:08 | <@JAA> | :-) |
| 00:42:19 | <@JAA> | Lots of reasons for that. |
| 00:42:40 | <@JAA> | URL rewriting is one of them. I'm still not sure whether the WBM uses ServiceWorkers yet. (It ought to.) |
| 00:42:49 | <@JAA> | And yeah, lack of POST/PUT/DELETE support is another big one. |
| 00:42:52 | <TheTechRobo> | wdym by service workers? |
| 00:43:01 | <TheTechRobo> | whgy would thos ehelp |
| 00:43:07 | <@JAA> | Also random query parameters in URLs generated by JS etc. |
| 00:43:21 | <@JAA> | E.g. current timestamp, canvas fingerprint, whatever. |
| 00:43:53 | <@JAA> | Service workers can intercept requests by anything in the page, including random obfuscated JS and whatnot, which the WBM would be unable to statically rewrite. |
| 00:44:10 | <TheTechRobo> | Ohhh |
| 00:44:26 | <TheTechRobo> | Also, I had a great idea a few months ago, but it won't work because security reasons |
| 00:44:26 | <@JAA> | I think webrecorder has been using them for a while now. |
| 00:44:49 | <TheTechRobo> | The WBM could have used HTTP 10whatever Use Proxy for its responses if it wasn't canned |
| 00:45:17 | <TheTechRobo> | which would help |
| 00:45:24 | <@JAA> | Probably the WBM has concerns about keeping support with Netscape Communicator or something. |
| 00:46:22 | <TheTechRobo> | Maybe then they should just document if_, im_, etc and they won't have to worry about that :-) |
| 00:47:10 | <TheTechRobo> | "If you're using Netscape Communicator 1.0 or some other web-browser from the Stone Age, try this link instead" |
| 00:47:15 | <TheTechRobo> | Maybe they're worried abt hassle, tho |
| 00:47:45 | <TheTechRobo> | Although considering how hard it is to get the WBM functioning without JS (without knowing its url structure at least) it's moot :P |
| 00:48:08 | <@JAA> | It took me over two years after Python 2 EOL to convince Jake that ia no longer needs to support it, so yeah... |
| 00:48:47 | <TheTechRobo> | youtube-dl is currently facing the same issue, apparently they're going to recommend yt-dlp if you have a new python |
| 00:49:02 | <TheTechRobo> | and youtube-dl will stay as a "compatibility version" or whatever |
| 00:49:25 | <@JAA> | Isn't that already what it is? Where 'compatibility' means 'works with the YouTube from a year ago but not the current one'? |
| 00:49:43 | <TheTechRobo> | It recently got a new maintainer |
| 00:49:52 | <TheTechRobo> | It's in a pinned issue |
| 00:50:04 | <TheTechRobo> | (and a new release) |
| 00:50:19 | <@JAA> | Python 2 should've been thrown into the abyss at least 8 years ago or whenever 3.3 came out. |
| 00:50:23 | <@JAA> | Huh |
| 00:50:28 | <@JAA> | Well, we'll see how that goes... lol |
| 00:51:05 | <@JAA> | youtube-dl still supports Python 2.6. Wow... |
| 00:51:10 | <TheTechRobo> | Yup |
| 00:51:26 | <TheTechRobo> | At least if you're going to support 2.x, make it 2.7+... |
| 00:51:35 | <TheTechRobo> | 2.7's been around for years |
| 00:53:10 | <@JAA> | Happy to see that the vast majority of people in the comments of that issue are sane and dislike the Python 2 support. |
| 00:53:24 | <@JAA> | It's been a long journey to get to that point... |
| 00:53:49 | <TheTechRobo> | I am currently mentally yelling at my script |
| 00:54:08 | <TheTechRobo> | For some reason, deleting ONE URL ENTRY from the session makes the file size drop from 4mb to 400kb... |
| 00:54:15 | <@JAA> | I had to defend too many times that I wrote Python-3-only code years after it first came out 'because Python 2 is still there and perfectly works!!1!'. |
| 00:54:18 | <TheTechRobo> | and there are like 181 tabs inside the first window lol |
| 00:54:43 | <TheTechRobo> | JAA: Supporting python 2 is actually a chore |
| 00:54:52 | <TheTechRobo> | Ppl need to realise that |
| 00:55:00 | <TheTechRobo> | /needed, its better now |
| 00:55:01 | <@JAA> | Tell me about it. I just yanked the remains of that support out of ia last week. |
| 00:55:24 | <@JAA> | The stdlib packages lack so many features it isn't even funny. |
| 00:55:35 | <TheTechRobo> | wdym stdlib? |
| 00:55:53 | <@JAA> | standard library |
| 00:56:03 | <TheTechRobo> | but like do you mean in general or python 2 |
| 00:56:11 | <@JAA> | The Python 2 versions specifically. |
| 00:57:25 | <@JAA> | Silly `try: from pkg import X except ImportError: # Python 2 emulation code` statements all over the place. |
| 00:57:36 | <TheTechRobo> | Oh yep, I love those |
| 00:57:48 | <TheTechRobo> | Even supporting pre-3.4 was annoying |
| 00:57:50 | <TheTechRobo> | Or was it pre-3.6? |
| 00:57:53 | <TheTechRobo> | I believe pre-3.6 |
| 00:58:17 | <TheTechRobo> | Anyway, I had a `try: ModuleNotFoundError except NameError: ModuleNotFoundError = ImportError` |
| 00:59:01 | <@JAA> | Ah yes, the great exception hierarchy overhaul. |
| 00:59:20 | <TheTechRobo> | I'm still not sure why I did that, as ModuleNotFoundError derives from ImportError, but |
| 00:59:35 | <@JAA> | Yeah, was just about to mention that, at least it was all backward-compatible. |
| 00:59:42 | <@JAA> | Also OSError and its numerous subclasses. |
| 00:59:44 | <TheTechRobo> | (I needed the code since I was doing an `except (ModuleNotFoundError, ImportError` for some reason) |
| 01:00:10 | <TheTechRobo> | Also back when I first started programming I had a bug where elif would act as an else |
| 01:00:12 | <TheTechRobo> | Guess why |
| 01:00:32 | <TheTechRobo> | `elif thing == "Hi" or "Bye":` |
| 01:00:38 | | chrismeller (chrismeller) joins |
| 01:00:38 | <@JAA> | Hell, IOError still exists in Python 3.10 even though it was deprecated in 3.3. |
| 01:01:00 | | chrismeller quits [Remote host closed the connection] |
| 01:02:08 | | chrismeller (chrismeller) joins |
| 01:02:30 | <TheTechRobo> | So apparently my script is dleeting all th e tabs from a window |
| 01:02:30 | | chrismeller quits [Remote host closed the connection] |
| 01:02:31 | <TheTechRobo> | ...Nice |
| 01:03:38 | | chrismeller (chrismeller) joins |
| 01:03:43 | <TheTechRobo> | Now it doesn't delete anything |
| 01:04:00 | | chrismeller quits [Remote host closed the connection] |
| 01:05:08 | | chrismeller (chrismeller) joins |
| 01:05:30 | | chrismeller quits [Remote host closed the connection] |
| 01:05:53 | | chrismeller (chrismeller) joins |
| 01:14:31 | <TheTechRobo> | Figured it out...sorta |
| 01:15:16 | <TheTechRobo> | Yup, figured it out |
| 01:18:21 | <TheTechRobo> | JAA: Have you ever tried Rust? (the programming language) |
| 01:27:25 | | pabs (pabs) joins |
| 01:29:34 | <@JAA> | TheTechRobo: Not really, no. I've read code in it before but never actually worked with it. |
| 01:39:55 | <TheTechRobo> | I...think my script worked? |
| 01:40:14 | <TheTechRobo> | Unless, of course, I've been modifying the wrong file, that would be nice, and fun, and INFURIATING |
| 01:43:12 | <TheTechRobo> | oh shit |
| 01:43:26 | <TheTechRobo> | "Firefox Developer Edition closed unexpectedly while starting." |
| 01:45:29 | <TheTechRobo> | Apparently Safe Mode just uses an older session |
| 01:45:31 | <TheTechRobo> | ...So that's fun |
| 01:45:49 | <TheTechRobo> | In other words, I bricked my session, and FF doesn't even give an error why! :D |
| 01:47:48 | <TheTechRobo> | In other news, my Hong Kong crawl is _still_ going on, and I think Twitter banned me! Even better! |
| 02:02:01 | <@JAA> | How did you manage that? |
| 02:35:46 | | Meli quits [Ping timeout: 240 seconds] |
| 02:42:49 | <TheTechRobo> | JAA: Manage what? |
| 02:43:18 | <@JAA> | TheTechRobo: Getting banned from Twitter. |
| 02:44:05 | | Meli (Meli) joins |
| 02:44:32 | <TheTechRobo> | Well, this specific crawl has been running since the 3rd, and I've been struggling to finish the crawl (i SO want resuming to be added to grab-site) since January |
| 02:44:56 | <TheTechRobo> | And it's only going to get worse... :-( |
| 02:45:07 | <TheTechRobo> | > on 02-03; 143,140.3 MB in 1,238,405 resp. at 2.0/s, 11,726,001 in q.; 2 con. w/ 0-233 ms delay; igon |
| 02:45:27 | <TheTechRobo> | A bunch of the URLs are Twitter status URLs |
| 02:45:48 | <@JAA> | Even 2/s shouldn't get you banned I think. |
| 02:46:00 | <TheTechRobo> | Should I just add an ignore for the status URLs anyway? |
| 02:46:03 | <TheTechRobo> | JAA: well |
| 02:46:12 | <@JAA> | Also, make sure they're actually useful content. The redesign is JS rubbish and won't contain anything useful. |
| 02:46:16 | <TheTechRobo> | incognito mode gives me "Something went wrong" when loading tweet |
| 02:46:34 | <TheTechRobo> | JAA: I thought so but AB does it and I assumed it didn't require setup beforehand because of tha |
| 02:46:41 | <TheTechRobo> | *did it |
| 02:47:10 | <@JAA> | I don't want to discuss the details in a publicly logged channel. |
| 02:47:22 | <@JAA> | But probably you're getting the JS crap. |
| 02:47:44 | <TheTechRobo> | Ah well, all the tweets are archived anyway with snscrape's jsonl param |
| 02:47:53 | <TheTechRobo> | The URLs are actually extracted from the jsonl |
| 02:49:08 | <thuban> | snscrape has a plugin system, right? i've been thinking of writing one for livejournal |
| 02:50:03 | <@JAA> | They're called modules, but more or less, yes. |
| 02:50:20 | <@JAA> | LJ is more like a blog than social media though, isn't it? |
| 02:50:40 | <thuban> | does it matter? |
| 02:50:40 | <TheTechRobo> | I've been writing one for parler but I'm stuck on https://github.com/JustAnotherArchivist/snscrape/issues/379 |
| 02:51:19 | <@JAA> | Not from a technical perspective, but snscrape is about social networking services, so I generally don't want to merge things that are out of scope. |
| 02:52:07 | <TheTechRobo> | And it's probably best if I don't use requests.post directly |
| 02:54:03 | <thuban> | then i guess it depends how you define 'social media'. livejournal predates the prototypical examples of the term, but it has significant community-oriented features (as contrasted with eg blogspot) and once served a social role similar to that now held by eg tumblr |
| 02:56:56 | <@JAA> | What social interactions are there apart from blog post comments? |
| 02:58:37 | <thuban> | off the top of my head, friends, friends-locked posts, 'memories', communities (like multi-contributor blogs, which (depending on settings iirc?) people could freely join) |
| 03:03:17 | <@JAA> | Hmm, fair enough. Communities sound similar enough to subreddits. |
| 03:07:14 | <@OrIdow6> | i v a n complained here about being banned from Twitter recently, could be that they're getting more strict |
| 03:20:16 | <Doranwen> | having used LJ extensively in the mid-2000s, I can confirm the community aspects of it all |
| 03:20:35 | <Doranwen> | I often described it to people as a cross between a blog and a social network |
| 03:21:24 | <Doranwen> | some people only posted comments or to communities, and used their own personal LJ very minimally |
| 03:21:40 | | Doranwen is *definitely* interested in methods to archive LJ |
| 03:21:58 | <thuban> | TheTechRobo: i can't reproduce your parler issue; `snscrape.base.Scraper._post("https://parler.com/open-api/profile-feed.php", data={'user': 'NewtGingrich'})` returns good data for me. |
| 03:22:33 | <Doranwen> | there's a ton of fandom communities with valuable data to fandom, that we don't know how to save other than manually c/p-ing entries or saving full pages (the issue that some posts are only accessible while logged in makes things more complicated) |
| 03:22:36 | <thuban> | are you doing something weird with the headers, or perhaps incorrectly rejecting responses in _check_api_response? |
| 03:23:56 | <Doranwen> | and more recently, DW (Dreamwidth) made a post saying that LJ seemed to have closed off all 3rd-party access (used to be one could crosspost to both), which makes things tough for anyone who didn't import their entries already (so glad I did several years ago) |
| 03:24:03 | <TheTechRobo> | Maybe I am, that might make sense |
| 03:24:17 | <TheTechRobo> | Unfortunately, with the latest dev version, I am now running into AttributeError: 'str' object has no attribute '_request' |
| 03:29:38 | <thuban> | idk without context, post code |
| 03:32:40 | <TheTechRobo> | Using your code |
| 03:32:43 | <TheTechRobo> | thuban: |
| 03:32:47 | <TheTechRobo> | ^ |
| 03:33:01 | <TheTechRobo> | >>> snscrape.base.Scraper._post("https://parler.com/open-api/profile-feed.php", data={'user': 'NewtGingrich'}) |
| 03:33:01 | <TheTechRobo> | Traceback (most recent call last): |
| 03:33:01 | <TheTechRobo> | File "<stdin>", line 1, in <module> |
| 03:33:01 | <TheTechRobo> | File "/home/thetechrobo/.local/lib/python3.9/site-packages/snscrape/base.py", line 221, in _post |
| 03:33:01 | <TheTechRobo> | return self._request('POST', *args, **kwargs) |
| 03:33:02 | <TheTechRobo> | AttributeError: 'str' object has no attribute '_request' |
| 03:33:21 | <thuban> | er. Scraper() |
| 03:33:37 | <TheTechRobo> | That's what I get for blindly copypasting LMAO |
| 03:33:55 | <TheTechRobo> | Huh. It works. Yep, my check_api_responses is probably wrong. |
| 03:34:32 | <TheTechRobo> | ... |
| 03:34:34 | <TheTechRobo> | .... |
| 03:34:38 | <TheTechRobo> | .................... |
| 03:34:50 | <thuban> | nah my bad, shouldn't have sent something different from what i actually ran without checking it |
| 03:35:02 | <TheTechRobo> | - r != 200 |
| 03:35:06 | <TheTechRobo> | + r.status != 200 |
| 03:35:21 | <TheTechRobo> | actually no |
| 03:35:25 | <TheTechRobo> | + r.status_code != 200 |
| 03:35:27 | <TheTechRobo> | i am smart |
| 03:35:32 | <thuban> | (surprising that the class obj is quietly coerced to a string though) |
| 03:35:34 | <thuban> | nice |
| 03:35:49 | <@JAA> | thuban: It isn't. The string becomes the `self` argument. |
| 03:36:06 | <thuban> | oh! |
| 03:36:49 | <thuban> | (well... it is still surprising.) |
| 03:37:05 | <TheTechRobo> | Now the real problem is |
| 03:37:19 | <TheTechRobo> | It only seems to give the first 2 pages, after that it continues with page 2... |
| 03:37:20 | <@JAA> | I'm slightly surprised as well though that `cls.method(...)` is valid if that isn't a staticmethod. |
| 03:37:30 | <thuban> | Doranwen: ouch, that's tough. not at all surprising given their general trajectory since the russian buyout, but a pity seeing as they were originally behind openid |
| 03:38:14 | <@JAA> | TheTechRobo: I remember now that's what I observed some time ago as well when I briefly looked at implementing it. |
| 03:38:26 | <TheTechRobo> | ...Nevermind it seems to be fixed |
| 03:38:38 | <TheTechRobo> | JAA: Yeah I saw that when I first started but assumed i'd been banned or smth |
| 03:38:43 | <TheTechRobo> | Seems to work now? |
| 03:40:20 | <TheTechRobo> | Created a PR |
| 03:40:42 | <TheTechRobo> | Where's the Draft button |
| 03:41:53 | <TheTechRobo> | Anyway, do not merge it yet |
| 03:42:00 | <TheTechRobo> | I still haven't used the dataclasses |
| 03:42:06 | <TheTechRobo> | Going to bed now, cya |
| 03:48:01 | | Stiletto quits [Remote host closed the connection] |
| 03:52:19 | | Stiletto joins |
| 03:54:11 | <Doranwen> | thuban: yeah, if you ever work out a way for automated scraping of journals, communities, etc.… I'd *definitely* make use of it |
| 03:56:05 | <Doranwen> | the good news is that it doesn't *seem* like it's going to go down immediately - so a slow crawl of the stuff we really want is possible without running out of time, hopefully (slow enough not to trigger IP blocking) |
| 03:56:46 | <Doranwen> | but figuring out how to deal with the posts that aren't public is another challenge, and that is also a critical piece |
| 03:59:09 | <Doranwen> | years ago I used to save the occasional post using the mht format, and with ones I had to save while logged-in I'd go into the html and edit bits till it removed my logged-in info in the corner, lol, but that's very primitive compared to what would be necessary here |
| 04:01:00 | <thuban> | i think that's out of scope for snscrape |
| 04:01:04 | <@JAA> | Fun, the Debian unstable Docker image appear to be fucked. Package signing keys are missing, so `apt update` fails. gnupg isn't installed properly, so you can't add keys. And installing gnupg isn't possible because the keys are missing. (╯°□°)╯︵ ┻━┻ |
| 04:02:06 | | sonick quits [Client Quit] |
| 04:02:07 | <ivan> | I used LXD to put a Debian in my NixOS recently and that worked |
| 04:02:43 | <@JAA> | Which Debian? |
| 04:02:53 | <@JAA> | Stable works fine. |
| 04:03:24 | <@JAA> | I haven't looked deeper into it, just using that instead now for a brief test. |
| 04:04:52 | <thuban> | (partly because it's oriented toward public posts--there's no technical reason login info couldn't be added to the options for a particular module, but i suspect that's not intended--and partly because it retrieves message urls rather than the messages themselves) |
| 04:05:09 | <ivan> | JAA: it was bullseye |
| 04:05:46 | <@JAA> | Yeah, makes sense, if that broke, it would get noticed within minutes. |
| 04:06:43 | | Stiletto quits [Remote host closed the connection] |
| 04:07:36 | <@JAA> | lolwut? https://github.com/debuerreotype/docker-debian-artifacts/issues/122 |
| 04:10:30 | | Stiletto joins |
| 04:12:12 | <@JAA> | Well, not the issue I'm seeing, but hilarious nonetheless. |
| 04:12:57 | <thuban> | Doranwen: it _might_ make sense to write an lj snscrape module, then an 'unofficial' wrapper script that would log in and invoke it as a library. but you'd still have to decide how you wanted to archive the presumably-private result urls; grab-site with cookies is the obvious way, but the warc output isn't suited to redacting login information if that's really what you want |
| 04:12:59 | <thuban> | to do |
| 04:23:00 | <Doranwen> | well, for fandom, the warc format in general is very user-unfriendly |
| 04:23:02 | <Doranwen> | even if it's authentic |
| 04:23:58 | <Doranwen> | I've never used it myself - the likelihood of the average non-techie fandom person having any idea what they're doing to try to look at an old community… |
| 04:24:08 | <Doranwen> | so that'll be an interesting thing to figure out |
| 04:24:48 | <Doranwen> | and yeah, I'd have to figure out how to strip my personal login info from any pages saved that way, if I'm going to share with anyone and not just view privately |
| 04:24:51 | | Doranwen ponders |
| 04:25:04 | <thuban> | could do as much extraction as possible within the snscrape module and then just use the jsonl output. |
| 04:25:35 | <Doranwen> | I'm willing to give something a try, that's for sure |
| 04:25:49 | <thuban> | i forget, is livejournal theming templated html (like tumblr) or just css? if the former, reliable scraping is much more difficult. |
| 04:26:11 | <Doranwen> | the former, I believe |
| 04:26:15 | <Doranwen> | I know there are community themes |
| 04:26:42 | <Doranwen> | pretty sure it's not just css |
| 04:26:57 | <Doranwen> | but I don't know that I know enough to answer with 100% accuracy |
| 04:29:25 | <Doranwen> | this looks templated to me: https://comment-fic.livejournal.com/ |
| 04:39:08 | <thuban> | if you could dig up any documentation on the theme system _or_ info on forcing an unstyled/consistently-styled view (there's the mobile site but is it fully-featured? is there an equivalent to dw's '?style=light'?) and drop me links, i'd really appreciate it |
| 04:39:17 | <Doranwen> | I'll do my best! |
| 04:40:47 | <thuban> | thanks! i have a big pile of projects and i can't promise i'll get to this one any time soon, but i definitely want to give it a go |
| 04:43:54 | <Doranwen> | yeah, I'm still working on Yahoo Groups in all my spare time, lol - I just outsourced your question to the others on the one Discord server, we'll see if someone can come up with the info |
| 04:44:14 | <Doranwen> | if not, I'll try to dig it up - but I suspect someone out there would know more about the answer to that question than I would |
| 04:44:39 | <Doranwen> | I never mucked with the theme stuff or paid much attention to it, other than selecting the one I liked best |
| 04:46:13 | <thuban> | oh, info on the api would also be good, if any's out there. (it appears to be confusing and terrible) |
| 04:50:50 | <Doranwen> | lol, I'm not surprised |
| 04:51:11 | <Doranwen> | hopefully someone out there will know before I get to the point where I have the time to look for that info myself |
| 05:01:11 | <Doranwen> | OK, I've asked on four servers (two with mostly overlapping users), we'll see what results - I'll let you know if we're able to turn up something |
| 05:01:36 | | fuzzy8021 quits [Read error: Connection reset by peer] |
| 05:02:02 | | systwi quits [Read error: Connection reset by peer] |
| 05:02:17 | | fuzzy8021 (fuzzy8021) joins |
| 05:02:37 | | systwi (systwi) joins |
| 05:07:12 | | DogsRNice quits [Read error: Connection reset by peer] |
| 05:10:28 | <Doranwen> | if nothing else, I might send a message to one of the DW people who knows the LJ codebase well and see if they'd be at all willing to help, but pm-ing them out of the blue feels a bit weird, so I'm hoping someone will find something more visible than that |
| 05:13:02 | <@OrIdow6> | What is DW? |
| 05:13:27 | <Doranwen> | Dreamwidth |
| 05:13:41 | <Doranwen> | founded by a bunch of people who left LJ, more or less, and wanted away from the Russian overlords |
| 05:13:56 | <thuban> | ty! (if i can't find a decent api for pagination/post content, it looks like it'll be mobile site for those and __rpc_get_thread for the comments.) |
| 05:14:07 | <Doranwen> | someone did say "?format=light" works for LJ, and I tested it |
| 05:14:14 | <Doranwen> | it does work on the individual posts, but not the main LJ pages |
| 05:14:33 | <Doranwen> | I tried it on comment-fic to see, no effect on the main page, but applying it to a specific post stripped formatting back to the basic version |
| 05:14:55 | <Doranwen> | the easiest way to find all posts is to use a journal's calendar |
| 05:15:04 | <Doranwen> | as far as user browsing goes |
| 05:15:12 | <Doranwen> | not sure if there's a way to use that in the automation method |
| 05:15:53 | <Doranwen> | you can see a little of how that works here: https://comment-fic.livejournal.com/2021/ |
| 05:17:05 | <thuban> | nb, i expect the _current_ lj codebase is very, possibly entirely, different than the foss perl one that became dreamwidth (and friends) |
| 05:17:56 | <Doranwen> | yeah, there is that |
| 05:22:03 | <thuban> | unfortunately calendars appear to be themed, and i haven't found a way to get that view in a consistent format. good to know about ?format=light, though; i _think_ the mobile site doesn't munge post content at all (in which case it's probably better to target it anyway since it's lighter-weight and less likely to change), but i'm glad to ahve a fallback |
| 05:22:39 | <thuban> | *have |
| 05:22:42 | <Doranwen> | though the importer *was* working up until recently, if intermittently (LJ had a nasty problem of banning the DW IPs temporarily due to people putting in the wrong password when trying to crosspost, and claiming they couldn't whitelist it) |
| 05:22:55 | <Doranwen> | which means whoever kept it working did know something of how the api would've worked, I think? |
| 05:24:10 | <thuban> | hm, possibly! |
| 05:24:51 | <thuban> | i've got to get to bed; will come back to this later. |
| 05:26:01 | <Doranwen> | yeah, I better get back to work myself |
| 05:26:05 | <Doranwen> | will keep an eye out here :) |
| 05:26:17 | <Doranwen> | I check on this channel at least once a day |
| 05:34:02 | | Doranwen sighs |
| 05:34:20 | <Doranwen> | with a paid account one can force all journals to display in the style one chooses |
| 05:34:26 | <Doranwen> | but I don't have that |
| 05:54:57 | | wickedplayer494 quits [Remote host closed the connection] |
| 06:03:09 | | BlueMaxima quits [Read error: Connection reset by peer] |
| 06:05:07 | | wickedplayer494 joins |
| 06:05:17 | | wickedplayer494 is now authenticated as wickedplayer494 |
| 06:38:04 | | benjins quits [Read error: Connection reset by peer] |
| 06:56:42 | | sonick (sonick) joins |
| 08:35:46 | | Meli quits [Ping timeout: 240 seconds] |
| 08:39:23 | | Meli (Meli) joins |
| 10:02:07 | | @EggplantN quits [Quit: Ping timeout (120 seconds)] |
| 10:03:13 | | EggplantN (EggplantN) joins |
| 10:03:13 | | @ChanServ sets mode: +o EggplantN |
| 12:16:11 | | benjins joins |
| 13:07:24 | | qwertyasdfuiopghjkl quits [Client Quit] |
| 13:11:48 | | qwertyasdfuiopghjkl joins |
| 13:37:32 | <@rewby> | JAA: I just had a cursed idea for "authentic" warc replay with support for JS and webworkers and stuff. Find a smallish browser, compile it to wasm, configure it to use a http(s)-proxy instead of direct connect. Point it at something like pywb3 in proxy-replay mode. |
| 13:41:37 | | Arcorann quits [Ping timeout: 265 seconds] |
| 13:53:25 | <chrismeller> | is there a docker image of wget-lua hosted somewhere? found a reference to atdr.meo.ws/archiveteam/wget-lua, but it doesn't seem to exist. |
| 14:04:11 | | le0n quits [Ping timeout: 252 seconds] |
| 14:04:17 | | le0n_ (le0n) joins |
| 14:08:23 | | benjins is now authenticated as benjins |
| 14:30:21 | | knecht420 quits [Client Quit] |
| 14:33:45 | | knecht420 (knecht420) joins |
| 14:56:22 | <Doranwen> | thuban: I can confirm that *while logged in*, ANY journal can be forced to display in the same style that one's own is - just have to add `?style=mine` to the end of the URL - and *this works for calendar too* |
| 14:56:29 | <Doranwen> | one has to manually keep adding that, but it *does* work |
| 14:57:00 | <Doranwen> | I tested by logging in and and adding it to the end of the URL for that comm and even the calendar is being displayed in the format mine is |
| 14:58:31 | <thuban> | Doranwen: fortunately i don't think that will be necessary |
| 14:58:35 | <thuban> | https://transfer.archivete.am/12j1lN/lj_notes.txt |
| 14:59:37 | <TheTechRobo> | I did some benchmarks on lrzip vs zstd: https://gist.github.com/TheTechRobo/091acf60f80d007557ad35821fdb3a6d |
| 15:00:03 | <TheTechRobo> | result: zstd is FAAAAAAAAAAAAAAST |
| 15:00:12 | <thuban> | i _think_ this covers everything (though i would feel better about the friends lists if you could point me to examples of people with huge numbers of friends/in huge numbers of communities, etc) |
| 15:00:43 | <Doranwen> | I'm sure I can find someone with enormous numbers of friends or comms, lol, just have to ask for suggestions. |
| 15:01:55 | <Doranwen> | I'll get back to you if I find something, may be a while since I'll be busy for a few hours |
| 15:04:21 | <Doranwen> | subscribers, btw, are people who've subscribed to one's LJ (like an RSS feed), and so the entries of the LJ in question will appear on *their* reading pages |
| 15:04:58 | <Doranwen> | separate from friends, which is the equivalent of DW's having "granted access" - friends can view protected entries, but don't necessarily have to subscribe, they can just do it by "I'll go look at that user's LJ now" |
| 15:05:09 | <Doranwen> | so the two aspects are separate |
| 15:05:44 | | DogsRNice (Webuser299) joins |
| 15:05:53 | <thuban> | (it might be nice to _generate urls for_ calendars, to archive for discoverability in the wbm or whatever, but this is trivially done without actually scraping anything) |
| 15:06:06 | <Doranwen> | ok |
| 15:06:07 | <thuban> | gotcha, thanks |
| 15:06:16 | <Doranwen> | my profile's only got 50 friends, probably not the numbers you were wanting |
| 15:06:25 | <Doranwen> | member of 108 comms, it looks like |
| 15:16:02 | <thuban> | huh: the member-of list is _not_ truncated in the page js. funny, but i'll take it |
| 15:17:53 | <thuban> | (and profile.get_friends goes over 100 without paginating) |
| 15:27:20 | | qwertyasdfuiopghjkl quits [Remote host closed the connection] |
| 15:39:27 | | qwertyasdfuiopghjkl joins |
| 16:27:59 | | thetechrobo_ joins |
| 16:31:26 | | TheTechRobo quits [Ping timeout: 240 seconds] |
| 16:40:13 | | thetechrobo_ is now known as TheTechRobo |
| 16:47:45 | | TheTechRobo is now authenticated as TheTechRobo |
| 16:50:14 | | Iki1 joins |
| 16:50:21 | | rellem (chrismeller) joins |
| 16:52:46 | | AnotherIki quits [Ping timeout: 240 seconds] |
| 16:53:30 | | chrismeller quits [Ping timeout: 265 seconds] |
| 16:57:59 | | lunik1 quits [Ping timeout: 252 seconds] |
| 16:58:28 | <@JAA> | rewby: Yeah, that's disgusting. lol Also, you'll still have to make sure that the non-deterministic JS stuff (e.g. `Date.now()`) returns the same value on replay as on capture. I've toyed with the idea of a patched browser that records every call to a JS function with a non-deterministic result. Then somehow store that in the WARC and use it again on playback. |
| 17:16:49 | <Sanqui> | JAA: I <3 that idea |
| 17:17:27 | <@JAA> | Sanqui: As do I. I hate the idea of actually implementing it though. lol |
| 17:18:09 | | mutantmonkey quits [Ping timeout: 252 seconds] |
| 17:19:21 | <@JAA> | The least horrible approach would probably be to use Firefox ESR since that stays on the same version for ~10 months at least. |
| 17:21:55 | <Sanqui> | I'll keep the idea in mind as I wade through the more advanced Puppeteer APIs |
| 17:25:12 | <@JAA> | :-) |
| 17:26:16 | <TheTechRobo> | @TeamVanillo twitter account is up, so guess what I'm currently grabbing :-) |
| 17:28:42 | <TheTechRobo> | https://archive.is/BTwcp/41f1795f6815607a8091ba9a55cc49cfae209700.png |
| 17:31:55 | | mutantmonkey (mutantmonkey) joins |
| 17:56:27 | | sonick quits [Client Quit] |
| 18:01:06 | | rellem quits [Ping timeout: 240 seconds] |
| 19:06:14 | <@OrIdow6> | rewby: I've thought you might even be able to go further, have a full VM so you can run Flash and Silverlight and whatnot |
| 19:09:01 | <thuban> | JAA, request for comment, if you don't mind me picking your brain: |
| 19:09:18 | <thuban> | - am i correct in thinking snscrape modules are intended to mimic browser requests, rather than use a dedicated api even if one is available? |
| 19:09:29 | <thuban> | - should dataclasses match or closely mimic structures served by the target site (including fields that appear to be useless or redundant), or should they be rationalized? |
| 19:09:31 | <@OrIdow6> | The JS thing might work a lot better if you had a more flexible way of doing user interactions |
| 19:09:40 | <thuban> | (related: jsonl means no fancy attempts at reference structures, right? you stuff the whole user struct into every tweet even if you're only searching for tweets by one user?) |
| 19:10:06 | <@OrIdow6> | Race conditions (or rather, innocuous situations that become race conditions under replay) could be a problem |
| 19:18:51 | | useretail quits [Remote host closed the connection] |
| 19:20:09 | | useretail joins |
| 19:20:17 | | useretail quits [Remote host closed the connection] |
| 19:20:29 | | useretail joins |
| 19:21:34 | <@JAA> | thuban: If the API is open and decent, it's fine to use that. Just nothing that requires auth. Regarding item structure, I've been including only reasonably meaningful and useful fields. But I'd have to see what you have in mind for more details. Yes, references between items should be avoided; it'd be fine to reuse the same `User` object if it's identical anyway, but it should be included properly |
| 19:21:40 | <@JAA> | (when the relevant data is available, cf. `TweetRef` introduced recently). |
| 19:22:32 | | lunik1 joins |
| 19:24:05 | <thuban> | oh hm, that changes things. (lj has a time capsule from 2006 in the form of an _xml_ rpc api.) |
| 19:24:19 | <@JAA> | Eww |
| 19:28:17 | <thuban> | as for questionable fields, i was thinking of multiple forms of the same timestamp, stuff related to the current user, tracking junk, and weird ones of unidentifiable meaning that have the same value for every item i've tried |
| 19:28:52 | <@JAA> | Yeah, no, that can all go to /dev/null. |
| 19:29:06 | | useretail quits [Client Quit] |
| 19:29:24 | <@JAA> | Also, native Python objects, no timestamp ints or such mess. |
| 19:29:53 | <thuban> | right. if you want a warc you can run it through warcprox |
| 19:29:57 | <thuban> | mm, yes |
| 19:33:28 | <thuban> | "*Warning:* The `xmlrpc.client` module is not secure against maliciously constructed data. If you need to parse untrusted or unauthenticated data see _XML vulnerabilities_." well there's a blast from the past :| |
| 20:13:24 | | useretail joins |
| 20:59:06 | <thuban> | ok, i have concluded that this thing is so janky there is literally nothing i would be better off using it for |
| 21:01:58 | <thuban> | i held out hope for some time of at least using it to get post contents, since you can get up to 50 entries at once (as opposed to the one at a time i would have to use otherwise). but after figuring out which of their wacky pagination schemes would actually get me all the way through a blog, i realized that embeds are just completely broken |
| 21:03:24 | <thuban> | like, there's a parseljtags parameter that's supposed to convert them (among other things), but it doesn't work properly |
| 21:03:47 | <thuban> | i guess that was... informative...? |
| 21:09:42 | | chrismeller (chrismeller) joins |
| 21:40:29 | | BlueMaxima joins |
| 21:45:55 | | fuzzy8021 quits [Ping timeout: 265 seconds] |
| 22:23:09 | | Hackerpcs quits [Quit: Hackerpcs] |
| 22:24:31 | | Hackerpcs (Hackerpcs) joins |
| 22:38:55 | | chrismeller quits [Read error: Connection reset by peer] |
| 23:24:06 | | BlueMaxima quits [Ping timeout: 240 seconds] |
| 23:28:08 | | Arcorann (Arcorann) joins |
| 23:39:43 | | BlueMaxima joins |
| 23:48:12 | | Iki joins |
| 23:50:51 | | Iki1 quits [Ping timeout: 252 seconds] |
| 23:51:19 | | fuzzy8021 (fuzzy8021) joins |
| 23:53:59 | <Doranwen> | thuban: do you still need a user with a ton of friends or whatever? because apparently there's a user ranking thing and the 3rd highest one is in English (first two being in Russian) and they're friended by like 500+ users, lol |
| 23:58:11 | <Doranwen> | see what I mean: https://www.livejournal.com/ratings/users?country=noncyr |