HomeSearchPrevious dayNext day

00:07:27<mgrandi>http://www.trumptwitterarchive.com/
00:08:21<@JAA>Yes, there are various efforts to archive them. Videos also get uploaded to IA automatically etc.
00:09:02<mgrandi>That site didn't get videos, but yeah
00:29:13<purplebot>Current Projects edited by Wickedplayer494 (+37, reddit to scripts-only) just now -- https://www.archiveteam.org/?diff=46096&oldid=46083
00:33:53<@arkiver>wessel1512: you're now an automoderated user on the wiki
00:34:13<purplebot>Coronavirus edited by Wessel1512 (+851, /* Netherlands */) just now -- https://www.archiveteam.org/?diff=46097&oldid=46092
00:34:13<purplebot>Disestablishments in 2014 edited by Nemo bis (+235, LSI example) just now -- https://www.archiveteam.org/?diff=46098&oldid=42820
00:34:28<wessel1512>Tanks
00:34:45<@arkiver>welkom
00:35:14<purplebot>GameMaker Sandbox edited by Childishbeat (+269, /* Archives */ A new archive, derived …) just now -- https://www.archiveteam.org/?diff=46099&oldid=31814
00:49:04godane quits [Ping timeout: 240 seconds]
00:57:41<mgrandi>Do we want to make sure we have POTUS and all that? Make sure we have his followers too?
00:59:13<purplebot>Coronavirus edited by Justcool393 (+8, Linking to the proper IRC channel …) 24 minutes ago -- https://www.archiveteam.org/?diff=46100&oldid=46097
01:02:13<purplebot>Elections/2021 Tweede Kamerver election edited by Wessel1512 (+277) just now -- https://www.archiveteam.org/?diff=46101&oldid=45822
01:08:11<mgrandi>https://www.theverge.com/2021/1/8/22221648/google-suspends-bans-parler-play-store
01:08:30<mgrandi>I guess they have a website view of it
01:08:37<@arkiver>#neparlepas
01:09:55<mgrandi>We should have a channel of new channel names
01:10:03<mgrandi>Rooms*
01:11:31<@Kaz>we do, except we call it a wiki
01:11:46<@arkiver>how about a bot where you can go !parler and it'll give #neparlepas
01:11:51<@arkiver>actually
01:11:54<@arkiver>!channel parler
01:12:26<@Kaz>would require you to know a project exists, which relies on you seeing the site (or something), at which point it's redundant
01:13:05<@arkiver>true
02:00:02Dallas quits [Client Quit]
02:03:57Dallas (Dallas) joins
02:59:47Mateon1 quits [Remote host closed the connection]
02:59:50Mateon1 joins
03:11:42<tech234a>Trump posted to another account, @TeamTrump, which has since been suspended. https://twitter.com/sarahfrier/status/1347732318383849473
03:23:46<OrIdow6>Do we want to start a crawl of whitehouse.gov &c, with this talk of impeachment or use of the 25th? I think IA does end-of-term stuff anyhow - does that cover it?
03:26:51<maxfan8>That sounds good, just in case
03:27:07<maxfan8>(it's better to have dupes of those kinds of things anyways)
03:27:42<OrIdow6>I'm more or less asking whether someone knows something about what IA does
03:28:48Jonboy345 joins
03:29:03<@JAA>I think they just run recursive crawls with a hand-crafted list of URLs. Not entirely sure though.
03:32:05<tech234a>Also: “Appears the Trump campaign’s digital director tried to give Trump his account. Twitter promptly suspended him” https://twitter.com/JDiamond1/status/1347741119434649603
04:05:13<mgrandi>All government websites should probably be backed up yeah
04:06:15<mgrandi>Also, apparently the twitter API still works for suspended accounts (like using embeds for websites)
04:06:23<mgrandi>Text only though
04:19:01DogsRNice quits [Read error: Connection reset by peer]
04:25:52nerdguy1138 quits [Ping timeout: 252 seconds]
04:26:56qw3rty__ joins
04:30:24qw3rty_ quits [Ping timeout: 240 seconds]
04:41:45nerdguy1138 (nerdguy1138) joins
04:51:31<Jonboy345>I have a site that I'd like to backup using archivebot, but the content of the posts is behind a login... Is is possible to provide a login cookie to archivebot to get past the login requirement?
04:57:15<OrIdow6>No
04:57:58<OrIdow6>But if it's at risk, there are other methods to archive it, that can use login cookies
04:58:09<OrIdow6>Jonboy345:
05:00:45<Jonboy345>I don't think it's in immediate risk, but it's an automotive enthusiast forum with LOADS of information that I think is worth saving.
05:01:20<Jonboy345>OrIdow6:
05:02:39<Jonboy345><OrIdow6>
05:03:00etnguyen03 quits [Client Quit]
05:03:00<Jonboy345>Well, I'm dumb and can't seem to get @mentions right.. lol.
05:04:03<OrIdow6>Jonboy345: If you want to make a copy yourself, wget can take cookies
05:04:33<OrIdow6>This is IRC, there's no special syntax for pings; you just say the person's name
05:04:34<Craigle>Not sure what client you're using, but IRC doesn't require @ mentions. Usually people have it set to highlight on their username or certain keywords
05:05:27<Jonboy345>OrIdow6: Would it be in a form that the internet archive could use so the data would be available to be searched?
05:05:54<Jonboy345>I'm using hexchat on Server 2019
05:06:01<Jonboy345>craigle
05:06:36<justcool393>https://www.archiveteam.org/index.php?title=Government_Backup
05:06:43<justcool393>There's this page that talks more about it
05:07:12<OrIdow6>Jonboy345: You're free to upload it to the Internet Archive, but it won't get into the WayBack Machine
05:08:00<Jonboy345>Yeah... That's what I'm after... Throw the URL of the bookmarked thread in the wayback machine and see the info...
05:08:16<Craigle>That paged me
05:08:18<OrIdow6>For that to happen, you would need to be on a fairly short list of people with upload permissions (I am *not* one of them), who would also probably have to do the crawl
05:08:35<Craigle>Should be the same if I do Jonboy345
05:09:36<Jonboy345>Yup.. Got that one, Craigle... Just didn't see any difference in the text on my side so assumed it wasn't alerting who I was mentioning..
05:09:46<OrIdow6>Thanks justcool393 - looks like IA is doing their own thing - https://archive.org/details/EndOfTerm2020WebCrawls?sort=-publicdate
05:10:34<Craigle>Nah, working as intended. Just a system designed decades before @ mentions became a thing, lol
05:11:00<Jonboy345>I'm old enough to remember matrix dot printers, but not old enough to be an irc user. LOL
05:11:39<Jonboy345>or an experienced irc user. lol.
05:14:09<Jonboy345>OrIdow6 anyone you know of on that short list?
05:15:46<atphoenix>Jonboy345, you can use the WBM extension to submit lists of URLs to IA for saving. https://github.com/internetarchive/wayback-machine-webextension
05:16:00<atphoenix>requires an IA account (free)
05:16:33<atphoenix>if you have a smallish list of specific forum URLs you want to save, at least
05:17:03<Jonboy345>I've been using that for the threads I come across that are especially dense with information, thanks atphoenix.
05:18:39<Jonboy345>Was just curious if there was a better way. The manufacturer isn't even selling the cars in the US anymore, so in the next couple decades I highly doubt there will be more than a couple hundred of their vehicles on the road.
05:18:45<OrIdow6>Jonboy345: People read this channel - if they want to help, they will, but again, it's some work, and you said the site doesn't look liek it's at risk
05:20:45<OrIdow6>atphoenix: Looks like that just runs SPN on the URL you give it? Don't think it'd capture anything behind a login wall
05:20:49<Craigle>You may also want to post the site here. If anyone is interested in scoping it out, they can do that without you having to be around
05:21:18<Craigle>We have a variety of timezones in here, so there's people coming and going all the time
05:22:51HP_Archivist (HP_Archivist) joins
05:23:51<Jonboy345>https://www.planetisuzoo.com/ is the site I have in mind.
05:24:23<atphoenix>oh, right, it's basically easy SPN
05:25:13<Craigle>So... I can view post contents without being logged in. Am I special?
05:25:40<Jonboy345>Can you see attachments such as images?
05:26:08<Craigle>Oh, nevermind. Looks like I can't see attachments unless they're externally hosted
05:26:46<Craigle>No. First couple posts I randomly hit, I could. But the ones hosted on the site I can't
05:27:18<Jonboy345>And loads of them are on photobucket so it's a crapshoot if they even work or are watermarked. lol
05:28:29<Jonboy345>I messaged the admin of the site earlier this year to ask if they could temporarily lift the login requirement for attachments, but they never responded to me
05:31:31godane (godane) joins
05:32:06<jodizzle>Jonboy345: You could try using grab-site like this with your forum: https://github.com/archiveteam/grab-site#website-requiring-login--cookies
05:35:13<Jonboy345>Thanks, jodizzle. I may use that to at least get a copy of the site.
05:44:13<purplebot>Running Archive Team Projects with Docker edited by Tech234a (-1, Remove extra blank line) just now -- https://www.archiveteam.org/?diff=46102&oldid=46057
06:21:29HP_Archivist quits [Client Quit]
06:37:49icedice quits [Client Quit]
07:14:54<mgrandi>Do we have a channel for getting the Donald.win?
07:15:16<mgrandi>I remember @jaa saying they use cloudflare but I just saw on twitter that someone found their raw IP
07:24:26<OrIdow6>#archivebot?
07:24:49<mgrandi>https://twitter.com/SoatokDhole/status/1347769797036146688
07:25:05<OrIdow6>Seeing as this is Twitter, I can only assume they're going to be DoSed now
07:25:12<mgrandi>Apparently it only accepts connections from CF ips tho
07:28:09<mgrandi>The problem is they are leaving on cloudflare attack mode so it's hindering efforts
07:30:15<OrIdow6>Twitter page SPNd, by tje wau
07:30:19<OrIdow6>*the way
07:31:46<Ajay>how good is that cloudflare + wayback machine collaboration, does it only save popular stuff?
07:34:07<mgrandi>i dunno what that is
07:34:29<Ajay>https://blog.cloudflare.com/cloudflares-always-online-and-the-internet-archive-team-up-to-fight-origin-errors/
07:34:31<mgrandi>but that page that someone linked that apparently uses cloudflare workers, gets the login page for thedonald.win so that is promising
07:34:31<OrIdow6>That's my understanding
07:35:09<OrIdow6>https://archive.fart.website/bin/irclogger_log/archiveteam-ot?date=2020-09-18,Fri&sel=24#l20 - discussion of the CF/IA thing
07:35:18<Ajay>looks like it's an explicit setting you have to enable
07:35:24<mgrandi>https://web.scraper.workers.dev/?url=thedonald.win&selector=html&scrape=text&pretty=true
07:36:02<OrIdow6>More discussion 2 days from then
07:36:15<mgrandi>vs doing a curl where i just get a "cloudflare redirect" page
07:37:05<Ajay>I see, so we shouldn't trust always online
07:37:09<mgrandi>i don't know anything about cloudflare so i dunno if we can somehow get a wpull or wget running behind it
08:33:42Jonboy345 quits [Read error: Connection reset by peer]
08:36:53Jonboy345 joins
08:37:36Jonboy345 quits [Read error: Connection reset by peer]
08:40:00Jonboy345 joins
08:45:59Jonboy345 quits [Read error: Connection reset by peer]
08:48:34Jonboy345 joins
09:06:56burnmjd joins
09:06:57burnmjd quits [Client Quit]
09:18:35hooway joins
10:00:17purplebot quits [Client Quit]
10:00:41purplebot joins
10:27:28spirit joins
11:00:28<justcool393>Is there a channel for the Capitol Hill stuff
11:02:54<mgrandi>probably just putting stuff in archivebot
11:03:09<justcool393>okay
11:49:10LutherZane quits [Quit: Connection closed for inactivity]
12:21:11spirit quits [Client Quit]
12:55:04Jonboy345 quits [Ping timeout: 240 seconds]
13:10:36Ryz quits [Remote host closed the connection]
13:11:20Ryz (Ryz) joins
14:27:04Arcorann_ quits [Ping timeout: 240 seconds]
14:32:41etnguyen03 (etnguyen03) joins
15:13:44Barto quits [Ping timeout: 240 seconds]
15:14:06spirit joins
15:19:10Barto (Barto) joins
15:29:43Barto quits [Ping timeout: 252 seconds]
15:43:36Barto (Barto) joins
15:56:07Barto quits [Ping timeout: 252 seconds]
16:03:44Barto (Barto) joins
16:18:34Jonboy345 joins
16:21:58Barto quits [Ping timeout: 252 seconds]
16:25:20HP_Archivist (HP_Archivist) joins
16:30:12Barto (Barto) joins
17:07:03pew joins
17:17:26<Dallas>Does anyone know if someone’s archiving that gab website ? I’m not super familiar with it but I’m seeing a lot of screenshots of it on Twitter lately, if not I might see about grabbing the top posts every few mins
17:19:14<Dallas>Oh it’s a mastodon instance that no one else peers with
17:24:18<@JAA>I looked into it shortly after they switched to Mastodon, but they disabled all the fun API stuff, so I ignored it after. Note that it requires JS, so you'll need some browser-based archival thingy.
17:28:11<Craigle>Just to add to that discussion, I found a site I had never heard of also being used. https://wimkin.com
17:28:22<Craigle>Here is an example page that I grabbed yesterday: https://wimkin.com/1r3
17:36:06<Dallas>Ah okay, I have a decent machine lowing about I use for media so I should be able to handle a couple instances of headless chrome
17:36:22<Dallas>its behind cf but not anti-scrape phew
17:50:14godane1 joins
17:52:24godane quits [Ping timeout: 240 seconds]
18:10:06godane1 quits [Client Quit]
18:15:15Jonboy3451 joins
18:18:44Jonboy345 quits [Ping timeout: 240 seconds]
19:07:58etnguyen03 quits [Read error: Connection reset by peer]
19:38:34<purplebot>GTF Képhost edited by Bzc6p (+885, /* Shutdown */ update) just now -- https://www.archiveteam.org/?diff=46103&oldid=46066
20:32:45<brad>Hmm. CloudFlare does have “workers”. I wonder if you could use that to run a regular container?
20:34:00<brad>Also, I wonder how you might be able to have a containerized grab-site running but reporting back to a different server? One where you could require authentication, so that you can control who can report status?
20:36:32<brad>No, it looks like they use “isolates”, which is an isolated JavaScript process, as opposed to the heavier containers that others use. See https://developers.cloudflare.com/workers/learning/how-workers-works
20:37:34<brad>Does anyone want to write a version of grab-site that runs as a CloudFlare Workers “Isolate”?
20:37:59<Dallas>They basically run the chrome v8 runtime, anything that compiles to wasm will run in an isolate
20:40:06<Dallas>They also have an api to take a page of html and run stuff against it, I think you could extract all the page links with it https://developers.cloudflare.com/workers/runtime-apis/html-rewriter
20:40:22<Dallas>Then have the worker call itself with the links from the page
20:41:10<Dallas>hmm does anyone run grab-site in docker ??
20:46:13Zerote_ joins
20:46:16fuzzy802 joins
20:46:16fuzzy8021 quits [Killed (NickServ (GHOST command used by fuzzy802!~fuzzy8021@173-224-26-244.ptcnet.net))]
20:47:04ragu quits [Read error: Connection reset by peer]
20:47:17rsn_ joins
20:47:21atphoenix quits [Read error: Connection reset by peer]
20:47:21jonboy3452 joins
20:47:24HackMii quits [Ping timeout: 240 seconds]
20:47:27AlsoHP_Archivist joins
20:47:28ragu joins
20:47:31Jonboy3451 quits [Remote host closed the connection]
20:47:44katocala quits [Ping timeout: 240 seconds]
20:47:51atphoenix (atphoenix) joins
20:48:09katocala joins
20:48:37endrift quits [Quit: +++CARRIER LOST+++]
20:48:45endrift joins
20:49:23HP_Archivist quits [Ping timeout: 240 seconds]
20:49:24britm0b quits [Ping timeout: 240 seconds]
20:49:41britm0b joins
20:49:43Zerote quits [Ping timeout: 240 seconds]
20:50:43rsn quits [Ping timeout: 240 seconds]
21:14:17HackMii (hacktheplanet) joins
21:22:43<@JAA>Dallas: https://github.com/ArchiveTeam/grab-site/pull/149
21:23:24<Dallas>Sweet ty :)
21:31:33lukash7 quits [Client Quit]
21:40:36lukash7 joins
21:58:45<Dallas>crap, gab turned on some cf protection that's stopped me being able to crawl it :(
22:05:58<brad>JAA & Dallas — as soon as that PR is merged, I’ll create a PR with some documentation on how I’ve used grab-site in a Docker container, and lessons I’ve learned.
22:07:08<brad>I wonder — could we compile that entire container to WASM so that it could be run inside of CloudFlare Workers?
22:10:01<@JAA>That sounds like a 'they were so preoccupied with the question of whether they could that they never stopped to ask whether they should' situation.
22:13:48<@JAA>I'm not sure you can get proper WARC recording in Wasm though since you'll want the raw HTTP stream before transfer encoding etc. I don't think you can create sockets in Wasm, but I might be wrong about that. Browsers typically don't give you the raw data through their APIs.
22:13:52<brad>Well, if CF makes it hard to archive sites unless the archives is running in CFW, then I think they did this to themselves.
22:14:09<brad>Archiver, that is.
22:14:54<brad>Man, I didn’t think I had to worry about autocorrect here in irc. 😔
22:16:04<@JAA>You don't get file access either in Wasm, so writing WARCs is tricky.
22:16:08<@JAA>The list probably goes on.
22:18:25@kiska Nom nom nom
22:18:28<@kiska>On memory
22:19:54<brad>So, s scrape in CF?
22:20:18<brad>Snscrape that is?
22:20:39<brad>Or chromebot?
22:20:49<@JAA>'Up to 10ms CPU time per request' might be another little issue.
22:21:01<brad>Oh.
22:21:38<brad>Kinda hard to fit much archiving in 10ms.
22:27:14<@JAA>I wonder if you could simply build an HTTP CONNECT proxy.
22:30:56<Dallas>I'm trying to use wpull with phantomjs on to grab https://gab.com/ but It exits straight away and the warc has a 503 error in it I assume that's the CF protection and I'm screwed ?
22:31:15<@JAA>Yep
22:31:38<@JAA>I've been meaning to revisit my Cloudflare challenge breaker for a while.
22:31:59<Dallas>@JAA> 'Up to 10ms CPU time per request' might be another little issue.
22:31:59<Dallas>That's execution time, if you fire a request the thread your worker runs on is halted and the timer stops until the request comes back
22:32:26<@JAA>Yeah, I know, but HTML parsing and WARC writing will eat up those 10 ms already.
22:32:46<@JAA>If you actually want to run that stuff on the server side.
22:33:50<Dallas>Grab page > Extract links > Store html and request details in KV Store > fire next request based on extracted links
22:33:51<Dallas>Then have a local server thing that grabs the whole KV store (they have an api for doing that) and gen the warc from it
22:33:51<Dallas>I'd say that's possible but god knows if it'll work and scale or how long it'd take to build
22:33:55<brad>A CFW http/https proxy that would bypass CF over-aggressive CF “protection”, I think that could be a good thing. But someone would have to be willing to commit to running and paying for that proxy, and risk losing their account.
22:34:20<Dallas>I've tried grabbing via worker and I think the CF protection still applies to their own ips
22:35:19<Dallas>Although the link mgrandi sent earlier does bypass the gab 503 so I may be wrong about that https://web.scraper.workers.dev/?url=gab.com&selector=html&scrape=text&pretty=true
22:36:49<Dallas>I know a lot of stuff was grabbed in #archivebot but there are a load of calls for parler to be kicked off aws so is it worth trying to grab everything ?
22:37:07<mgrandi>i don't know if the server will go down but they use AWS for dns
22:37:18<@JAA>You need the raw HTTP stream for WARCs, not just the decoded body. And I don't think you can get that with Wasm because it has the same limitations as all JS APIs.
22:37:39<@JAA>I never tried it though, so happy to stand corrected on that.
22:38:31AlsoHP_Archivist quits [Client Quit]
22:50:46<Dallas>yeah it uses the standard fetch api so doubtful you could pull the stream from it
22:59:47<mgrandi>yeah, if we could somehow use a cloudflare worker to proxy traffic that would bypass it, but i dunno if that is against TOS and would get shut down fast or what
22:59:58<mgrandi>might just have to use a phantomjs or whatever based scraper
23:17:13nerdguy1138 quits [Ping timeout: 252 seconds]
23:31:57nerdguy1138 (nerdguy1138) joins
23:43:34<purplebot>Elections/2021 Tweede Kamerver election edited by Wessel1512 (+250, /* Political party and campaign …) just now -- https://www.archiveteam.org/?diff=46104&oldid=46101
23:56:19Ruthalas quits [Quit: END OF LINE]

HomeSearchPrevious dayNext day