HomeSearchPrevious dayNext day

00:01:22dm4v quits [Read error: Connection reset by peer]
00:02:09dm4v joins
00:02:11dm4v quits [Changing host]
00:02:11dm4v (dm4v) joins
00:24:42Ryz quits [Remote host closed the connection]
00:26:29Ryz (Ryz) joins
00:32:04Ryz quits [Remote host closed the connection]
00:33:51Ryz (Ryz) joins
00:48:44tzt quits [Read error: Connection reset by peer]
01:17:13<useretail>do we have some tools for scraping DKAN Open Data Platform websites? https://getdkan.org/
01:31:55OrIdow6 (OrIdow6) joins
01:31:56@ChanServ sets mode: +o OrIdow6
01:35:36Meli quits [Ping timeout: 244 seconds]
01:36:42Meli (Meli) joins
01:43:34tzt (tzt) joins
01:45:55crumbling_heart joins
01:46:04crumbling_heart quits [Client Quit]
01:47:04heart joins
01:47:29wickedplayer494 quits [Ping timeout: 244 seconds]
01:52:13wickedplayer494 joins
02:12:42heart_ joins
02:13:10heart leaves
02:14:16heart_ quits [Changing host]
02:14:16heart_ (heart) joins
02:58:36britmob joins
03:55:13qw3rty__ joins
03:59:02qw3rty_ quits [Ping timeout: 250 seconds]
04:07:03sec^nd quits [Ping timeout: 258 seconds]
04:11:49sec^nd (second) joins
04:18:06DogsRNice quits [Read error: Connection reset by peer]
04:56:40nicolas17 quits [Ping timeout: 250 seconds]
05:34:35<pabs>https://attivissimo.blogspot.com/2021/08/expert-help-needed-to-decode-rare-piece.html https://news.ycombinator.com/item?id=28260568
05:34:44<pabs>" Expert help needed to decode a rare piece of space history: the Lunokhod data tapes "
05:53:59Viniter69 (Viniter) joins
07:51:57<pabs>https://news.opensuse.org/2021/07/26/deavtivating_connect_o_o/
07:52:02<pabs>(already shutdown)
08:18:39BlueMaxima quits [Read error: Connection reset by peer]
08:18:53BlueMaxima joins
09:29:04BlueMaxima quits [Client Quit]
09:43:51bitbit (bitbit) joins
10:01:35rsn joins
10:03:37rsn_ quits [Ping timeout: 252 seconds]
10:08:42Terbium quits [Quit: http://quassel-irc.org - Chat comfortably. Anywhere.]
10:09:03Terbium joins
10:40:24rsn_ joins
10:42:14rsn quits [Ping timeout: 244 seconds]
10:48:00<bitbit>Hi. is wpull being maintained? I get the following error when I try to run it https://bpa.st/62LA
10:51:16<bitbit>the documentation says support is only up to python3.4. when I try it with python3.4, I get the following error https://bpa.st/62LA
10:53:25<@JAA>bitbit: wpull 1.2.3 will work with 3.4. wpull 2.0.3 (you'll need to install from GitHub) works up to 3.6.
10:53:41<@JAA>Also, you pasted the same URL twice.
10:55:01<bitbit>ah sorry, the 2nd URL is https://bpa.st/VEAA
10:56:25<@JAA>It's not really maintained in the last couple years, no. Nobody's found the time for that, sadly.
11:04:24qwertyasdfuiopghjkl joins
11:05:00<IDK>JAA: should I keep doing the deactivating twitter account thing in AB
11:05:39<IDK>I felt like 50% of them did it for attention
11:10:25<bitbit>JAA: thanks. I tried 1.2.3 on py3.4 and 2.0.3 on py3.6 ... didn't work :(
11:11:49<bitbit>is there any tool that's commonly used as an alternative to wpull? (not wget)
11:12:22<@JAA>Can't help with 'didn't work'. I've used 1.2.3 on 3.4 for years and am running multiple machines (AB pipelines) with 2.0.3 on 3.6...
11:12:59<@JAA>IDK: Seems like a mostly useless grind to me, but your call.
11:13:32<IDK>Alright last one then
11:14:12<IDK>I will just focus on searching proactive sites
11:15:00<@JAA>bitbit: Oh, maybe you got the wrong versions of html5lib or Tornado.
11:15:28<IDK>New chinese law are killing 90% of the education idustry
11:16:58<@JAA>bitbit: Try `pip install Tornado==4.5.3 html5lib==0.9999999` to install the right ones.
11:17:23<@JAA>Er, tornado, not Tornado, I think.
11:25:22<bitbit>JAA: that worked! thanks
11:34:18<bitbit>JAA: there's a new error. perhaps I need a specific version of sqlalchemy? https://bpa.st/VEAA
11:35:58<@JAA>bitbit: Wrong link again, but yeah, I forgot about that. `pip install SQLAlchemy<1.4`
11:36:51<@JAA>You may also want to force `dnspython==1.16.0` because the older versions have bugs that can lead to crashes if you have e.g. Logitech mice anywhere in the same network. (Yes, really.)
11:36:55rsn joins
11:38:13<bitbit>https://bpa.st/FUKA
11:38:46rsn_ quits [Ping timeout: 252 seconds]
11:39:35<bitbit>alright now it seems to work OK!
11:39:47<@JAA>:-)
11:43:21<[42]>JAA: wat
11:43:31<[42]>got a link to the logitech issue?
11:50:07<@JAA>Nope, it was discussed in #archivebot at the time.
11:51:55<@JAA>Basically, Logitech's control software thingy sends UDP broadcasts on some high port. Not sure about the reason, maybe device discovery? In any case, if that port happens to coincide with the outgoing port for a DNS query and that broadcast packet arrives at just the right time, dnspython raises an exception because it can't parse it as a DNS reply, and that crashes wpull.
11:52:27<@JAA>Lots of other things can cause the same thing, obviously. Logitech was just a specific one I found interfering with one of the AB pipelines back then.
11:55:04<@JAA>The wpull crash is https://github.com/ArchiveTeam/wpull/issues/365 (but that doesn't mention Logitech etc.).
12:25:34bitbit quits [Client Quit]
12:32:46<[42]>lol
12:40:12bitbit (bitbit) joins
12:42:57<bitbit>JAA: another wpull error. it seems to happen at the end after it finished, I think? do you know what may be the issue this time? https://bpa.st/EGUA
12:48:02<@JAA>bitbit: I've seen that before but never figured out what caused it. Haven't seen any issues from it either though. If you have a minimal reproducible example (i.e. as little options as necessary, ideally just one request), please file an issue on GitHub.
12:48:32<bitbit>JAA: ok.thanks again :)
13:14:48rsn_ joins
13:17:13rsn quits [Ping timeout: 252 seconds]
13:25:32LeGoupil joins
14:08:38<bitbit>when using 'wpull --table', say I come across a response that I didn't anticipate (e.g. 403 error) and stop the crawl, change my IP, and resume the crawl -- do I have to use a separate database tool to delete the 403'd requests from the table? if so, would deleting from the 'queued_urls' table suffice?
14:15:19<@JAA>wpull doesn't have a --table option.
14:21:49qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds]
14:21:58qwertyasdfuiopghjkl joins
14:26:04<@JAA>bitbit: Handling crap like that is tricky and not really well-supported. You would have to mark the corresponding DB row as todo again, but you'd still have the 403 in the WARC anyway. You could make wpull retry 403s through a hook, but you can't prevent the WARC records being written. One strategy is to vary the delay settings (--concurrency is broken when using plain wpull) until you find one that
14:26:10<@JAA>works reliably, then start from scratch with that to get one clean crawl.
14:26:32nertzy (nertzy) joins
14:27:43<bitbit>JAA: right. meant to say --database
14:29:32<@JAA>I should say: manipulating the DB is not supported at all, and doing it wrong can easily crash wpull in an entertaining variety of ways.
14:30:39<bitbit>what did you mean about --concurrency being broken?
14:30:56<@JAA>https://github.com/ArchiveTeam/wpull/issues/339
14:31:13<@JAA>It's simply a no-op and runs at one concurrent request anyway.
14:32:31<bitbit>thats too bad. do you instead use wpull with parallel then?
14:33:03<@JAA>I can all but guarantee you that that won't work.
14:34:06<@JAA>I rarely use wpull outside of ArchiveBot these days. Previously, I used 1.2.3, and this bug was one of the reasons. You can work around it with a plugin, but meh...
14:34:22<@JAA>Maybe grab-site is a better option for you.
14:42:10nertzy quits [Client Quit]
14:44:50<bitbit>JAA: nice!
14:48:48nertzy (nertzy) joins
14:55:06yawkat` quits [Ping timeout: 250 seconds]
15:04:11Arcorann quits [Ping timeout: 244 seconds]
15:58:01nertzy quits [Client Quit]
15:59:58yawkat (yawkat) joins
16:37:16nertzy (nertzy) joins
16:47:53<bitbit>is there anything like https://github.com/alard/warc-proxy that supports https?
16:50:21<bitbit>oh I guess its pywb
16:50:38<@JAA>warcprox
16:51:29<@JAA>pywb also has a proxy mode, but it uses warcio which has serious issues with accuracy on writing WARCs.
16:54:12<bitbit>I'm looking to replay saved warcs via proxy in a browser https://github.com/internetarchive/warcprox says "warcprox stores warcs to disk which may be accessed with openwayback and pywb"
16:55:51<bitbit>what I mean is that it seems warcprox is only for writing warcs not viewing them? and they point you to using pywb for viewing
16:56:40<@JAA>Correct.
16:56:54<@JAA>pywb is fine for viewing, just not for writing.
16:57:16<bitbit>ok :)
16:57:20<bitbit>thanks
16:57:50<@JAA>Oh right, sorry, alard/warc-proxy also does the serving part. There are so many 'WARC proxy' projects that all do different things...
16:57:54<@JAA>Yeah, pywb's the right one then.
17:02:47<bitbit>it sure can get quite confusing with all the different tools and that half of them are deprecated
17:27:18qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds]
17:28:39qwertyasdfuiopghjkl joins
17:30:29nicolas17 joins
17:52:46Ruthalas quits [Ping timeout: 252 seconds]
18:03:59qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds]
18:06:31<nicolas17>hmph looks like old versions of Xcode were signed with a now-expired certificate, and at some point Apple re-signed them (so if you download them now, hashes don't match what is published elsewhere), and some specific versions were deleted altogether :/
18:10:25<[42]>isn't the way code signing works that it doesn't matter if the signing cert is expired as long as it wasn't expired on the timestamped signature?
18:10:46<[42]>as in it shouldn't be needed to publish with a new sig
18:12:41qwertyasdfuiopghjkl joins
18:22:13bitbit quits [Client Quit]
18:27:28Ruthalas (Ruthalas) joins
18:30:51qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds]
18:34:07Ruthalas quits [Client Quit]
18:34:30Ruthalas (Ruthalas) joins
18:40:00nicolas17 quits [Ping timeout: 250 seconds]
18:40:07useretail quits [Quit: Leaving]
18:42:49nicolas17 joins
18:47:32DogsRNice (Webuser299) joins
18:49:28qwertyasdfuiopghjkl joins
18:58:45qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds]
19:10:36nertzy quits [Client Quit]
19:17:06qwertyasdfuiopghjkl joins
19:20:50Iki1 joins
19:24:04Iki quits [Ping timeout: 244 seconds]
19:24:17spirit quits [Client Quit]
19:28:43Iki1 quits [Ping timeout: 244 seconds]
20:07:06Gereon6 quits [Ping timeout: 250 seconds]
20:20:05jamesp (jamesp) joins
20:20:20<jamesp>What are you guys archiving that's related to Afghanistan?
20:42:53<nicolas17>[42]: that's how it should work but https://hammen.medium.com/the-apple-packagepocalypse-2019-edition-6e2d4bc0aa90
20:43:18<nicolas17>(that's for macOS updaters, but Xcode had the same issue I think)
20:44:40<[42]> :(
20:46:41LeGoupil quits [Client Quit]
20:48:08<nicolas17>so if I download Xcode_8.3.xip from Apple now, the hash doesn't match https://github.com/notpeter/apple-installer-checksums/blob/master/xcode.md
20:49:14<nicolas17>but more annoyingly, if I download 8.3.1 I get an error... makes me think that I better archive the ones I can download before they delete more
20:53:54sec^nd quits [Remote host closed the connection]
20:54:20sec^nd (second) joins
21:05:04Iki joins
21:43:38<Jake>jamesp: everything we can?
21:50:55knecht420 quits [Read error: Connection reset by peer]
21:51:04knecht420 (knecht420) joins
21:51:36<Ryz>Heheh, watched an oldie but a goodie; casting an ant colony with molten aluminum: https://www.youtube.com/watch?v=IGJ2jMZ-gaI
21:55:31xit quits [Quit: Ping timeout (120 seconds)]
21:55:52xit joins
22:22:40<jamesp>Ryz I got that video saved.
22:24:39<Ryz>Could grab the whole YouTube channel if needed~
22:24:45<Ryz>*You could
22:27:33nertzy (nertzy) joins
22:32:38Jake0 (Jake) joins
22:33:09useretail joins
22:34:22Jake quits [Ping timeout: 252 seconds]
22:34:22Jake0 is now known as Jake
22:34:44<jamesp>I'll try
22:42:28nertzy quits [Client Quit]
22:54:55Ruthalas quits [Client Quit]
22:55:15Ruthalas (Ruthalas) joins
22:57:13<nicolas17>sooo can/should I upload these xcode installers to archive.org before apple deletes more of them? would apple send me/IA a legal nastygram? idk how this works
22:57:14<nicolas17>I guess "it's probably fine" given there's macOS installers there already >.>
23:02:57<@JAA>nicolas17: Does this answer your question? https://transfer.archivete.am/inline/bG4mu/aatt.png
23:03:28<nicolas17>I know someone who has 22TB of iOS firmware images in a NAS
23:03:31<@JAA>It could happen that the items get darked (= made inaccessible), but even then at least the data would still exist.
23:04:00<@JAA>exist on systems unrelated to Apple, Inc.*
23:06:51jamesp leaves
23:15:10Arcorann (Arcorann) joins
23:20:48<nicolas17>https://archive.org/details/SCOOTCD oh wow I had forgotten I uploaded this... some day I should scan the paper manual it came with
23:50:07nertzy (nertzy) joins

HomeSearchPrevious dayNext day