HomeSearchPrevious dayNext day

00:08:14Terbium quits [Quit: No Ping reply in 180 seconds.]
00:09:26Terbium joins
00:23:43@dxrt quits [Quit: ZNC - http://znc.sourceforge.net]
00:25:12dxrt joins
00:25:14dxrt quits [Changing host]
00:25:14dxrt (dxrt) joins
00:25:14@ChanServ sets mode: +o dxrt
00:30:53Jake5 (Jake) joins
00:32:32Jake quits [Ping timeout: 252 seconds]
00:32:32Jake5 is now known as Jake
00:32:42@arkiver quits [Excess Flood]
00:33:29arkiver (arkiver) joins
00:33:29@ChanServ sets mode: +o arkiver
04:18:44benjins quits [Read error: Connection reset by peer]
04:19:34godane2 joins
04:20:16benjins joins
04:21:53godane1 quits [Ping timeout: 252 seconds]
04:49:16qw3rty__ joins
04:53:14qw3rty_ quits [Ping timeout: 252 seconds]
06:39:06Ryz quits [Remote host closed the connection]
06:40:31Ryz (Ryz) joins
07:23:04lukash quits [Ping timeout: 240 seconds]
07:25:02lukash (lukash) joins
07:51:09VADemon joins
08:03:03godane1 joins
08:05:10Doranwen quits [Ping timeout: 252 seconds]
08:05:32godane2 quits [Ping timeout: 252 seconds]
08:06:24Doranwen joins
09:00:18britmob2 quits [Client Quit]
10:07:02Arcorann quits [Read error: Connection reset by peer]
10:34:56Arcorann (Arcorann) joins
10:36:02Arcorann quits [Read error: Connection reset by peer]
10:39:45@arkiver quits [Excess Flood]
10:40:41arkiver (arkiver) joins
10:40:41@ChanServ sets mode: +o arkiver
10:51:11Arcorann (Arcorann) joins
11:00:12<Arcorann>https://delistedgames.com/sony-basically-removed-25-years-of-history-from-playstation-com/
11:26:30hooway joins
11:30:16VADemon quits [Read error: Connection reset by peer]
13:51:20britmob2 joins
14:00:13mutantmonkey quits [Remote host closed the connection]
14:00:37mutantmonkey (mutantmonkey) joins
14:00:52<yano>yikes
14:20:16Doranwen quits [Ping timeout: 252 seconds]
14:20:47Doranwen joins
14:25:14Arcorann quits [Ping timeout: 252 seconds]
14:32:44Doranwen quits [Ping timeout: 252 seconds]
14:32:57Doran joins
14:46:05mutantmonkey quits [Remote host closed the connection]
14:46:26mutantmonkey (mutantmonkey) joins
14:49:09<Dallas>I'm using snscrape on an instagram user, is there a way to extract all the media urls for each post? The DisplayUrl only points to the first image if a post has multiple images
14:52:24mutantmonkey quits [Ping timeout: 240 seconds]
15:05:37mutantmonkey (mutantmonkey) joins
15:47:25<@JAA>Dallas: I don't think so. Not without making one request per post with multiple images, at least, and that'll get you banned even quicker than normal.
15:48:38<Dallas>ah crap that's annoying, well I can do it slowly, I'm adding the field `mediaUrls` to the snscrape insta module so I can at least dump all the direct urls out
15:49:38<@JAA>Yeah, I basically gave up on developing the Instagram module because it's basically useless with the bans. :-/
15:53:07<Dallas>Do we know the rate limit? do they send a 429 or is it just a straight up ban outta no where?
16:00:17<@JAA>Nope. And it's a 302 to the login.
16:04:00<@JAA>All I know about the rate limit is that it's ridiculously low and can easily be triggered just by browsing manually. Also, there's a strong dependence on your IP address.
16:04:57<@JAA>Residential IPs are less likely to get banned, as are IPs from blocks not allocated to major server providers.
16:54:03marked19578 joins
16:55:56marked1957 quits [Ping timeout: 252 seconds]
16:55:56marked19578 is now known as marked1957
17:07:01Lord_Nightmare quits [Quit: ZNC - http://znc.in]
17:09:51Lord_Nightmare (Lord_Nightmare) joins
17:55:40Doran is now known as Doranwen
18:54:54Doranwen quits [Ping timeout: 252 seconds]
18:55:06Doran joins
18:59:09Doran quits [Remote host closed the connection]
18:59:41Doran joins
19:00:16Doran is now known as Doranwen
19:12:30Doranwen quits [Ping timeout: 252 seconds]
19:12:36Doranwen joins
19:19:43VerifiedJ quits [Quit: The Lounge - https://thelounge.chat]
19:44:49VerifiedJ (VerifiedJ) joins
20:58:24@arkiver quits [Excess Flood]
20:59:19arkiver (arkiver) joins
20:59:19@ChanServ sets mode: +o arkiver
21:48:45<systwi>Does anyone have any suggestions for creating an .md5 file from IA's _files.xml files?
21:49:00<systwi>As in each file listed in it.
21:49:44<@JAA>https://git.kiska.pw/JustAnotherArchivist/little-things/src/branch/master/iasha1check should be useful for that.
21:49:47<systwi>I've tried with `sed' and even keypress automation to try to create it, and it usually works (albeit *very* clunky)
21:49:48<@JAA>It's nasty but it works.
21:50:01<systwi>Thanks I'll check it out
21:56:30<systwi>`readarray' doesn't exist on my copy of macOS :-/
21:56:50<@JAA>> macOS
21:56:52<@JAA>Found the problem.
21:57:42<@JAA>If you don't need the fancy check, just direct the output to a file instead of reading it into an array and then doing all that stuff below with it.
21:58:12<@JAA>The script is for checking an IA item against a local directory to verify that everything's uploaded correctly.
21:58:24<@JAA>That's why it ignores derives and the item metadata files.
21:58:34<systwi>I was trying to do that, near the end I changed one of the lines to "{ (printf "%s\n" "${iamd5sums[@]}") > >(sed 's,^, ,') 2>&1; } > md5sums.txt" but haven't tried it yet.
21:59:36<systwi>Yep, I was looking to ignore derives.
22:01:03<@JAA>What Bash version are you using by the way that doesn't have readarray? That was added ages ago.
22:01:13<systwi>Yeah, macOS'
22:01:18<systwi>Woops
22:01:41<systwi>Yeah, macOS's bash is quite old due to licensing issues I think, it's 3.2.57
22:01:55<@JAA>lol wtf
22:02:11<systwi>Granted, I'm not running the newest OS
22:02:34<systwi>I'm 3 major versions behind
22:02:56<systwi>Newer macOS versions use zsh
22:03:48<@JAA>Ah
22:04:47<@JAA>Yeah, apparently it's because Bash 4+ uses GPLv3 instead of GPLv2. Apparently Apple hates GPLv3.
22:05:32<systwi>I could install bash 4 with brew, but I'll stick with stock.
22:05:38<@JAA>Anyway, the older version should probably work for you: https://git.kiska.pw/JustAnotherArchivist/little-things/src/commit/d10a1d3675dfc1091ee9988a49093c3685751e6d/iasha1check
22:06:06<@JAA>However, it doesn't verify that the file lists match as well, so if you have files in the local directory that aren't in the item, it won't catch that.
22:07:06<systwi>That's fine, I'm actually doing the opposite. I'm making sure the download didn't fail. Missing files in that regard would make sense.
22:07:31<@JAA>Ah
22:10:47<systwi>Is there any need to keep -P in the grep query? macOS doesn't have it :-/
22:11:09<systwi>*macOS's `grep' doesn't support -P
22:11:41<ivan>brew install grep
22:11:53<@JAA>Yes, it's necessary for non-greedy matching.
22:12:11<@JAA>The non-crazy way would be to actually parse the XML, of course.
22:12:32<@JAA>But where's the fun in that?
22:12:34<ivan>but that would unnecessarily insist on valid XML
22:12:56<systwi>Hah, yeah, I was kinda doing that before all of this. I even tried converting the XML to JSON and parsing that, which was a bit more tolerable, but yikes.
22:13:21<systwi>Can't believe I haven't run brew install grep before, thanks
22:17:21<systwi>Ahhh, I'll bring this over to my Linux box. It spit out the file like I changed it to do, but it's, let's just say, a little broken.
22:25:28<systwi>Hmm, never mind, I still have the same issue. The filename gets interpreted as part of the hash
22:26:06<systwi>And the output is, for example, "ipsum dolor Lorem"
22:26:56<systwi>The only change I'm making is to save the hashes as a file
22:34:47godane1 quits [Read error: Connection reset by peer]
22:35:25godane1 joins
22:35:32User joins
22:35:56hooway quits [Read error: Connection reset by peer]
22:35:57TheLovinator quits [Quit: Ping timeout (120 seconds)]
22:36:12TheLovinator joins
22:36:28benjinsmith joins
22:36:41<tech234a>New YouTube Terms of Service allows YouTube to show ads on non-Partner Program videos; Payments to partners treated as royalties. https://www.youtube.com/t/terms
22:37:44benjins quits [Ping timeout: 240 seconds]
22:39:51benjinsmith is now known as benjins
22:42:51@arkiver quits [Excess Flood]
22:43:40arkiver (arkiver) joins
22:43:40@ChanServ sets mode: +o arkiver
22:57:32<systwi>JAA: Figured it out, `awk' was causing the issues. The files I was checking have spaces in their names.
22:58:25<@JAA>Ah
22:58:28<systwi>Here's a hacky workaround: https://transfer.notkiska.pw/GOTg9/iasha1check.txt
22:58:53<systwi>Take note that most of my GNU commands have "g" prepended to them
22:59:03<systwi>But it works!
22:59:14<@JAA>Yeah, unless there are 2+ consecutive spaces in a filename.
22:59:51<systwi>Like this?
22:59:55<@JAA>Yeah
23:00:03<systwi>Uh oh
23:01:06<systwi>Oh, I see, it removes the extra space
23:01:28<@JAA>sed 's,^\(.*\) \([0-9a-f]\+\)$,\2 \1,' instead of the awk command should work (assuming a reasonable version of sed).
23:02:18<@JAA>There are probably various more edge cases my command doesn't cover properly though, like stuff that's escaped as XML entities in the filename.
23:02:43<@JAA>In any case, you'll get warnings/errors about that, so it won't fail silently.
23:02:55<systwi>Ok, cool, thank you.
23:03:10<systwi>That worked with gsed (GNU sed). Didn't try it with BSD sed
23:11:40Larsenv quits [Quit: ZNC 1.8.0 - https://znc.in]
23:14:46Arcorann (Arcorann) joins
23:15:36Larsenv (Larsenv) joins
23:15:38<systwi>Would you happen to know if there's a way to get direct links to original items only?
23:16:05<systwi>I'm looking into it now, but I just thought I'd ask just in case.
23:19:34<@JAA>Nope
23:21:12<@JAA>Looks like `ia download` has no option to skip derives either. Interesting.
23:23:00<systwi>Huh. There must be a way, somehow. How else would IA's zip-on-the-fly work?
23:23:30<@JAA>Oh, it's certainly possible, just not implemented.
23:23:35<systwi>I see
23:24:32<@JAA>I bet it would be pretty trivial to implement there as well.
23:24:41<@JAA>The CLI should already have that information.

HomeSearchPrevious dayNext day