HomeSearchPrevious dayNext day

00:03:26hooway quits [Client Quit]
00:06:34<purplebot>File:ThinkGeek July 2010.png uploaded by Wickedplayer494 (+0) just now -- https://www.archiveteam.org/?diff=46137&oldid=0
00:07:34<purplebot>File:Thinkgeeklogo.gif uploaded by Wickedplayer494 (+0) just now -- https://www.archiveteam.org/?diff=46138&oldid=0
00:08:35pew joins
00:09:34<purplebot>ThinkGeek created by Wickedplayer494 (+2241, This died (and apparently we ArchiveBotted …) just now -- https://www.archiveteam.org/?diff=46139&oldid=0
00:14:43random2362473473457 joins
00:20:22<wickedplayer494>bonus points if you get the "Florida area" reference
00:25:59random2362473473457 is now known as DoesThisWorkYet
00:28:57<wessel1512>The tracker seems to be down
00:29:02<@kiska>We know
00:36:34<purplebot>Template:Navigation box edited by Wickedplayer494 (+0, Relocate Tencent Weibo) 15 minutes ago -- https://www.archiveteam.org/?diff=46140&oldid=46134
00:41:22cspotcode joins
00:41:57Mineroboter joins
00:42:25Web-user joins
00:55:42Web-user quits [Remote host closed the connection]
00:59:48Web-user joins
01:15:05HP_Archivist (HP_Archivist) joins
01:21:57Web-user quits [Remote host closed the connection]
01:22:59Mineroboter quits [Ping timeout: 244 seconds]
01:32:01<OrIdow6>Wow, Parler brought a lot of people
01:32:32godane quits [Client Quit]
01:33:23Web-user joins
01:35:02<OrIdow6>If that continues to occur at a high rate, may want to split it off into subchannels (though perhaps I'm just fatigued at reading logs)
01:35:25<OrIdow6>By the way, I did some looking into that Japanese homepage thing, wrote a small grab script
01:35:51<OrIdow6>Very simple, only interesting thing is some very lenient URL extraction
01:35:56<OrIdow6>But obviously that can wait
01:36:11<OrIdow6>So-Net U+ Pages, shuts down the 28th, should be a few TB at most
01:53:09DoesThisWorkYet quits [Remote host closed the connection]
01:59:48tzt quits [Read error: Connection reset by peer]
02:00:03Dallas quits [Client Quit]
02:06:39Dallas (Dallas) joins
02:07:35DoesThisWorkYet joins
02:16:06DoesThisWorkYet quits [Client Quit]
02:16:58DoesThisWorkYet joins
02:19:01cspotcode quits [Remote host closed the connection]
02:40:26<Craigle>OrIdow6: Based on Amazon's statement, less than 6 hours left for it to be up I believe
02:40:41<OrIdow6>Yes, 11:59 PM PST
02:40:50<OrIdow6>SO about 5 hours 20m
02:41:17<Craigle>But yeah, scrollback is crazy in there if you step away for any length of time
02:41:38<OrIdow6>If they don't switch off AWS before then for whatever reason
02:42:22<Craigle>Someone posted a statement earlier from Parler making it seem like there would be "downtime" once AWS shuts them off.
02:42:34<Craigle>They don't seem ready to pick up and migrate
02:42:43<OrIdow6>Hopefully
02:43:31<etnguyen03>https://parler.com/post/009ba435b68c46d4955e1cd7737fa27c?
02:43:37<@JAA>Can we please not spread the Parler stuff across two channels?
02:43:50<@kiska>-> #neparlepas
03:01:35Mateon2 joins
03:03:44Mateon1 quits [Ping timeout: 265 seconds]
03:03:44Mateon2 is now known as Mateon1
03:08:25Web-user35 joins
03:12:00Web-user quits [Ping timeout: 244 seconds]
03:19:50<HP_Archivist>Hey Craigle - quick question. So I've been using the GH repo for the WBM extension all week, pretty good/definitely better
03:20:10<HP_Archivist>But how to stay signed in? I'm continuously asked to sign in if I want to crawl a page
03:22:25<Craigle>To be honest, I haven't had that issue. Are you still signed in if you go to the IA homepage? I assume the extension is using your cookie
03:24:44taka joins
03:29:07britmob quits [Read error: Connection reset by peer]
03:29:07britm0b quits [Read error: Connection reset by peer]
03:41:24<taka>I had the same problem as he did, "You are not logged in. Please log in and try again." It seems that the save is not proceeding.However, in IA, I am still logged in.
03:50:06<HP_Archivist>taka - same here. I'm signed into my IA account all the time, but the extension still prompts to sign in, etc
03:50:30<HP_Archivist>Craigle - are we missing an option/setting? I looked at the settings but didn't see a 'stay logged in'
03:51:24<HP_Archivist>Because being prompted each time to save a web page makes a great tool nearly unusable, not worth it
03:55:19<Craigle>Are you seeing it on a bulk save, or just a regular SPN click?
03:55:27<Craigle>Bulk Save is a known issue: https://github.com/internetarchive/wayback-machine-webextension/issues/690
03:55:40<Craigle>Also, we should probably take this to #internetarchive
03:58:54<taka>Craigle Oh, it was a known problem.In my case it occurs in bulk saves.
03:59:25<Craigle>Awesome, looks like they are at least aware of that issue
04:11:34jlu5 (jlu5) joins
04:12:41nnnoice joins
04:13:14taka quits [Remote host closed the connection]
04:18:12fuzzy802 quits [Remote host closed the connection]
04:18:30fuzzy8021 joins
04:18:30fuzzy8021 quits [Changing host]
04:18:30fuzzy8021 (fuzzy8021) joins
04:20:30OrIdow6 is now known as Eb7WFWNM
04:20:49Eb7WFWNM is now known as OrIdow6
04:21:44OrIdow6 is now known as UmVeri
04:21:54Web-user35 leaves
04:24:35qw3rty__ joins
04:28:19qw3rty_ quits [Ping timeout: 265 seconds]
04:39:34britmob joins
04:54:04DogsRNice quits [Read error: Connection reset by peer]
05:00:01treora quits [Client Quit]
05:00:52etnguyen03 quits [Client Quit]
05:01:15treora joins
05:20:27DoesThisWorkYet quits [Remote host closed the connection]
05:31:14UmVeri is now known as OrIdow6
06:39:07ivan (ivan) joins
06:39:31<ivan>might be the wayback machine javascript that erases your cookies if you have too many?
07:48:16atomicthumbs joins
07:49:10<atomicthumbs>just saw the thing about the parler mirroring effort. y'all are heroes of online
07:55:29lennier2 joins
07:55:39<flashfire42>Well propaganda of crazy people is always valuable for future historians to study
07:55:58<atomicthumbs>there's a lot more than that in there
07:56:14<atomicthumbs>it's valuable for antifascists to study right now
07:58:52LeGoupil joins
07:59:04lennier1 quits [Ping timeout: 240 seconds]
07:59:09lennier2 is now known as lennier1
08:11:27<mgrandi>I will be shocked if it doesn't come back in some form
08:11:36<mgrandi>its apparently bankrolled by some rich people so they will probably host it themselves or something
08:12:24<OrIdow6>Not like AWS is free in the first place
08:12:31<OrIdow6>#neparlepas
08:13:36<purplebot>Parler edited by Wickedplayer494 (+0, Amazon took it to the back of the …) just now -- https://www.archiveteam.org/?diff=46141&oldid=46124
08:35:35hooway joins
08:44:58monoxane (monoxane) joins
08:44:58asdf123 joins
08:44:59kelsair joins
08:45:00Gaelan (Gaelan) joins
08:45:26asdf123 leaves
08:45:28Billy549 (Billy549) joins
08:45:45nickyyyyy joins
08:46:20daalekz joins
08:46:21dofe joins
08:46:34asdf123 joins
08:47:24vivid-noble joins
08:52:28TheLovinator joins
08:55:17tzt joins
08:56:12tzt quits [Client Quit]
08:56:41tzt joins
08:57:04kelsair quits [Remote host closed the connection]
09:02:06foundation joins
09:02:25Webuser572 joins
09:16:30asdf123 leaves
09:22:43rewby (rewby) joins
09:29:08monoxane quits [Client Quit]
09:44:05foundation quits [Remote host closed the connection]
09:44:53monoxane (monoxane) joins
10:05:35<purplebot>Matrix edited by Sanqui (+90) just now -- https://www.archiveteam.org/?diff=46142&oldid=45850
10:13:53<Dallas>What's the channel for https://github.com/ArchiveTeam/urls-grab ??
10:14:01<OrIdow6>#//
10:14:44<OrIdow6>Looks like you're already there
10:15:30<Dallas>ffs that name throws me of, ty
10:16:02<Dallas>woow I think that channel name breaks thelounge from logging to file
10:16:18<OrIdow6>You're welcome
10:16:43<Dallas>wait no it just saves it as '#.log' ffs
10:17:00<OrIdow6>(Name was my idea)
10:17:26<Dallas>boooo
10:17:31<Dallas>boo Orldow6
10:17:34<Dallas>;)
10:17:41<monoxane>"interesting urls" sounds like something i dont wanna run on a home connection i care about?
10:18:02<monoxane>or is it yet another playful description
10:18:08<OrIdow6>monoxane: Ask your question in #//
10:18:11<monoxane>kk
10:19:28<Billy549>just turned on the Docker warrior on my VPS and set it to 6 concurrent workers - picking ArchiveTeam's choice is useful for it to just run in background, right
10:21:39<Dallas>I think that works, its defo useful running in the background if it's picked a project up
10:22:13<Billy549>Good to know, i'll try and be around here more but it's still nice to just leave it
10:22:21Mateon2 joins
10:23:56<Billy549>it's picked up URLTeam 2 at least, which is nice
10:24:24Mateon1 quits [Ping timeout: 240 seconds]
10:24:24Mateon2 is now known as Mateon1
10:29:55chkn74 joins
10:35:05mazet joins
10:35:24TheLovinator quits [Ping timeout: 240 seconds]
10:38:19<Dallas>JAA I have channel logs for #neparlepas for the last day, should I archivebot them ?
10:38:33<Dallas>Or is someone automatically saving them ?
10:40:42czechball joins
10:40:57<czechball>Sanqui: alright..
10:41:01<Sanqui>(also join #archiveteam, #archiveteam-ot, #archivebot if you want)
10:41:19LeGoupil quits [Client Quit]
10:41:47<Sanqui>so I got a trillion things to do but I'm slowly running some projects for czech websites https://archiveteam.org/index.php?title=Template:Czech_websites
10:42:12<Sanqui>if you got some hot tips on stuff I should prioritize, or if you feel like helping with any of those, that'd be ace, because I'm the only person focusing on them lol
10:42:18<czechball>I see... I've scraped some Czech Flash game sites in the past myself
10:42:34<Sanqui>sweet!
10:42:42<czechball>(for the Flashpoint project)
10:43:03<Sanqui>major props! I haven't gotten around doing flash stuff, but I'm helping a friend with j2me stuff
10:43:29<Sanqui>most of what I do is ran through archivebot, because that way it gets absorbed into the wayback machine
10:43:58<czechball>I don't have a lot of experience with ArchiveBot, I only vaguely know what it does...
10:44:28<czechball>so, should I help simply by running the jobs listed on https://archiveteam.org/index.php?title=Webzdarma ?
10:44:51<Sanqui>so the jobs listed there are jobs that have already been run
10:44:58<Dallas>Anyone got a way to grab the media from all the posts under a hashtag ? #NoFlyList (on twitter) is filling with media atm
10:45:03<Sanqui>there's only one job remaining (20)
10:45:11<Sanqui>I've been running and monitoring those in the past year myself
10:45:14<Sanqui>20k domains
10:45:52<Sanqui>I'll be deriving more domains soon using bing search and these scrapes
10:46:19<Sanqui>honestly there's not much you can do with webzdarma this very moment, unless you want to do some coding
10:47:12<Sanqui>but if you know any low hanging fruit -- like older public forums, for example, I'd be happy to throw those in
10:48:29<czechball>well... I always thought it would be nice to archive user galleries on rajce.net
10:49:45<Sanqui>ůh, gkeljlfkuo
10:49:49<Sanqui>oh, definitely*
10:49:52<czechball>but that's not really a low hanging fruit tbh
10:49:59<Sanqui>keyboard layout oops
10:50:05<czechball>yeah that happens lol
10:50:27<justcool393>JAA: is #whoisthis4chan meant to be invite only
10:50:29<Sanqui>do you know if rajce deletes albums after some time?
10:50:44<czechball>I have no idea, I'll have to check... but I don't think so
10:51:35<purplebot>Template:Czech websites edited by Sanqui (+18, add rajce.net) just now -- https://www.archiveteam.org/?diff=46143&oldid=46120
10:51:36<Sanqui>oh, it works with subdomains... that adds a bit of an extra step before archivebot can handle it (namely subdomain discovery)
10:52:17<Sanqui>I'm guessing it's possible to have "unlisted" galleries on rajce.net, so search engine discovery would be needed too
10:52:35<czechball>I've seen galleries from 2010 so I don't think they delete inactive ones
10:54:19<Sanqui>that's good to know
10:55:34<purplebot>Rajce.net created by Sanqui (+259, stub out) just now -- https://www.archiveteam.org/?diff=46144&oldid=0
10:55:44<Sanqui>so BTW, if you go on http://dashboard.at.ninjawedding.org/3?showNicks=1, and type webzdarma on the filter, you can observe the current running job
10:56:32<czechball>oh, nice
10:56:49<czechball>so if I understand correctly, this archivebot is hosted and operated by you?
10:57:27<Sanqui>archivebot is operated by a loose collective of archiveteam members
10:57:55<Sanqui>I started this job, and I'm monitoring it for runaway scrapes (calendars or shopping carts are the devil)
10:58:19<Sanqui>we actually have enough nodes right now for the amount of jobs we're running, but I plan to spin a dedicated node for my jobs in the future
10:58:44<czechball>hmm... alright
10:59:05<Sanqui>the current job I'm running is for all of these domains https://etc.sanqui.net/archiveteam/webzdarma/webzdarma_catalogue_19
10:59:19<Sanqui>recursively, with external resources, and external links
11:01:09<Sanqui>webzdarma is not at immediate risk so I'm only running one or two jobs at a time but when we were saving internet centrum we were running several and spinned some dedicated nodes
11:01:42<OrIdow6>Dallas: AB
11:01:58<OrIdow6>Oh, missed half a sentence, never mind
11:02:06<Sanqui>archivebot architecture is better suited for website jobs, warrior is good for everything else (single pages, APIs, enumerable items etc.)
11:02:34<czechball>yeah I figured... Warrior needs a specific script while archivebot works with regular websites
11:03:08<Billy549>i'm assuming the stuff i just put into AB is fine then? (files needed for Animal Crossing: Pocket Camp and Dr Mario to launch, they're just static files downloaded on first install)
11:03:42<Sanqui>Billy549: awesome!
11:04:02<Billy549>I'll grab Mario Kart Tour files later, it kept giving me a random account error
11:05:15<Sanqui>but yeah, archiving a small to medium sized website is as easy as running !a https://example.org
11:05:52<Sanqui>so if you encounter any notable stuff (I tend to run personal websites as I see them if they haven't been archived for years, etc... a good rule of thumb is if it's http only lol)
11:06:09<czechball>so, websites archived through archivebot get actually included in wayback machine?
11:06:17<Billy549>Okay, good to know
11:06:25<Sanqui>czechball: yep
11:06:55<Sanqui>you need voice or ops to run !a scrapes, you can run !ao (single pages) if you don't have voice
11:07:10<Sanqui>if you have a website you can ping somebody who's around (or me) with a good reason and we'll probably throw it in
11:07:12<czechball>just tried !a lol
11:07:43<Dallas>OrIdow6 AB doesn't grab media right? I'll need to snscrape the hashtag and then use something (tubeup) to upload the twitter vids
11:08:08<OrIdow6>Pictures, I think it does; video, it almost certainly doesn't
11:08:52<Dallas>Yeah vids are what I want, well I have to do some work for my actual job so I'll script that up later
11:08:58<flashfire42>I don’t think archivebot has used Tubeup for a long time
11:09:14<flashfire42>it does grab videos if it comes across them
11:09:19<flashfire42>socialbot might grab them
11:09:23<flashfire42>On Twitter stuff
11:09:28<Billy549>using !ao with a URL list works at least, which is what I needed
11:09:47<Billy549>cause i just grabbed the links from the proxy and spat them onto Pastebin
11:10:42<OrIdow6>flashfire42: AB with Socialbot *does* grab videos?
11:11:31<flashfire42>I think si
11:12:24<OrIdow6>Oh
11:14:48TheLovinator joins
11:15:48<Dallas>flashfire42 I meant using tubeup locally
11:16:17<flashfire42>Oh yeah for sure you can do tubeup locally but i don’t suggest doing it on your main account for tubeup
11:16:20<Dallas>flashfire42 so if I do a `snscrape twitter-hastag` in AB it'll get all the videos on the tweets ?
11:16:43<flashfire42>In theory I don’t know exactly how socialbot and snscrape works that is all JAA with that tool
11:16:52<jodizzle>No, there's no guarantee of that. At least not as far as I know.
11:17:42<jodizzle>Videos typically need something else. Though we did recently find a realDonaldTrump tweet in the WBM where a (low-resolution) video played back normally, which was weird.
11:18:01<OrIdow6>That was because someone had SPN2d it
11:18:11<flashfire42>Yeah the video would have been spn2d
11:18:29<jodizzle>And that is able to get videos?
11:18:44<OrIdow6>Under certain conditions
11:19:15<flashfire42>It depends which way the wind is blowing and if you stick your tongue out the right way
11:19:28<jodizzle>I'm unclear on what the 'certain conditions' are. That's what struck me as weird about it.
11:19:31<jodizzle>lol flashfire42
11:20:14<flashfire42>Depends on if the video embeds right and if it’s set to capture all outlinks which often it doesnt grab all of them
11:20:16<OrIdow6>I don't know much about the SPN internals, so I can only be general here, but broadly (assuming it's HLS video), it needs to download both the m3u8 and the transport segments
11:20:34<OrIdow6>Which means that the video has to autoplay, and it has to be short enough
11:20:36<czechball>Sanqui ok so back to the first question: how can I be helpful?
11:20:41<OrIdow6>Autoplay or auto-load, I should say
11:21:25<jodizzle>Dallas: tubeup works well for getting twitter videos. Using `filter:native_video` with snscrape helps too.
11:21:55<Dallas>Should I run this locally or generate a list of media urls and dump them in AB ?
11:21:59<jodizzle>Ah, interesting OrIdow6 flashfire42
11:22:21<Sanqui>czechball: probably not immediately helpful, I don't have any specific task for you, just keep in mind you can archive websites and take part in projects that interest you 👍
11:22:26<flashfire42>Well Dallas it depend on how you want them accessible
11:22:46<Sanqui>archivebot is a super powerful tool for how easy it is to use, so I try to introduce people to it quickly. and i feel like more of the czech internet needs attention
11:22:51<Sanqui>w/r/t archival
11:23:54<flashfire42>all non English stuff needs more attention I cringe every time that anything non English is ignored in archivebot to “save room”
11:24:26<czechball>lol
11:24:41<jodizzle>Dallas: If you want to get videos into AB, I suppose you could add the URL with the m3u8 to a list, "expand" it to all the transport segments, and add those to the list as well. Then '!ao <' that
11:24:53<monoxane>HLS is only ever used for live content, theres much better codecs and transports for playback material
11:25:02<czechball>is there a recommended workflow for making lists of websites to be archived or actually collecting them in the first place?
11:25:27sec^nd quits [Ping timeout: 240 seconds]
11:25:30<Dallas>hmm that could work, my goal is getting vids in to IA since god knows stuff gets taken down a lot on twitter
11:25:35<jodizzle>But tubeup does a good job with twitter. The metadata comes out well
11:25:55<Sanqui>czechball: if it's not just a random personal website but a larger project, document stuff on the wiki
11:26:55<Sanqui>what's the website, why is it important, any relevant content notices, any peculiarities about api access etc
11:27:02<czechball>well, the first things that pop up in my mind are Czech gaming portals, like gamesites, csko.cz, craftuj, taurigaming and associated forums
11:27:36<czechball>for most of these I think that it'd be possible to use standardized phpbb scrapers
11:27:48<Sanqui>we could make a wiki page as a general list of notable czech websites
11:27:51<Sanqui>for now I've been using my template
11:28:02<Sanqui>archivebot can handle phpbb well with the "forums" filter
11:28:07<Sanqui>ignore set rather
11:28:09<jodizzle>Dallas: If you do use tubeup, you should be mindful of how much you use it, since IA has made complaints about tubeup usage in the past. On the other hand, saving twitter videos isn't the same as mirroring youtube, so it's hard to say.
11:28:27<Dallas>hmm okay I'll bare that in mind, ty
11:28:30<Sanqui>so, you can't do anything wrong by starting a page, making a list, and poking me about next steps
11:29:07<czechball>alright, I'll read through the wiki a bit... thanks for the info
11:29:19<jodizzle>I would say that an .m3u8 in AB is probably less accessible in most cases
11:29:56<Sanqui>thanks for the help! I do like a trillion things constantly so I don't have time to make lists of all the portals and forums etc, but I can monitor the archivebot jobs after they're in
11:30:44<czechball>yeah same, I'm also working on lots of stuff at once and this whole archivebot thing seems kinda overwhelming, I'll try getting into it
11:31:16<Sanqui>yeah archival is always very open-ended I'm afraid
11:31:20sec^nd (second) joins
11:31:51<Sanqui>i also archive czech video game history, gaming magazines etc as a part of retroherna.org and there's always so much to do :P
11:31:52<czechball>heh... now that https://github.com/setnicka/ulozto-downloader exists, I've been also thinking about scraping stuff from uloz.to
11:32:03<Sanqui>that would be a big project lol
11:32:49<czechball>indeed... there would be a lot of problems and ulozto would probably deploy countermeasures once they'd notice the load
11:34:36<purplebot>Uloz.to created by Sanqui (+131, Created page with "Czech file host, …) just now -- https://www.archiveteam.org/?diff=46146&oldid=0
11:35:13vivid-noble quits [Ping timeout: 252 seconds]
11:37:19<czechball>also, there's this website called dnbshare.com for sharing drum and bass mixes, tracks etc...
11:37:59<czechball>I wrote a very basic set of scripts for automated downloading https://github.com/Czechball/dnbshare-cli
11:38:15<czechball>maybe we could also find a way to archive it properly?
11:39:39<czechball>the only problem here is that all queries on that site are limited to 50 results I think
11:40:54<Dallas>Do it very slowly then lol
11:41:38<Sanqui>fair disclaimer, if the website isn't at known risk of going down, generally you'll have to spearhead projects like that yourself -- make a wiki page, scrape files, upload them to archive.org
11:49:22Dallas quits [Remote host closed the connection]
11:49:35<purplebot>Template:Czech websites edited by Sanqui (+106, + uloz.to) 16 minutes ago -- https://www.archiveteam.org/?diff=46145&oldid=46143
11:51:21Dallas (Dallas) joins
11:51:24Dallas quits [Remote host closed the connection]
11:52:32Dallas (Dallas) joins
11:53:02Dallas quits [Client Quit]
11:53:58no112 (no112) joins
11:56:59Dallas (Dallas) joins
11:59:18daalekz quits [Remote host closed the connection]
12:49:08chkn74 leaves
13:00:24blue507 (blue507) joins
13:03:23cajunjoel joins
13:05:37dissentmemo joins
13:07:40<hexa->https://www.reddit.com/r/ParlerWatch/comments/kuqvs3/all_parler_user_data_is_being_downloaded_as_we/giu04o6/ 😄
13:08:22<hexa->10 hours ago, forgive me for posting old news
13:15:48<OrIdow6>We know, it's becoming a sort of crisis
13:15:55<OrIdow6>It's on Buzfeed now
13:15:55TheJobd joins
13:16:09<cajunjoel>https://gizmodo.com/every-deleted-parler-post-many-with-users-location-dat-1846032466
13:16:22<OrIdow6>Gizmodo, Buzzfeed, same thing
13:16:23<OrIdow6>Whoops
13:17:01<cajunjoel>Buzzmodo
13:17:06<Billy549>that's "fun"
13:18:04<Billy549>Just cleared and stopped all 5 Docker volumes for Parler and yet it still hasnt seemed to have freed up the disk space, guess I need to wait
13:18:12left-shark joins
13:18:20<monoxane>at least your ram isnt all consuming
13:18:52<monoxane>my 2000 containers were fine during operation then when it went offline they suddenly decided to consume 96gb ram and 96gb swap
13:19:17<Billy549>Next time I use my HDD, didnt realise i could move it to my USB-C HDD
13:19:25<Billy549>i can just rm the docker data file right?
13:19:31<monoxane>what OS?
13:19:34<Billy549>macOS
13:19:40RJHacker11108 joins
13:19:43<left-shark>hello there
13:19:49<Billy549>I have the location for it, I've stopped Docker now (quit Docker Desktop etc)
13:19:54<RJHacker11108>where does data from AT go after archival ?
13:19:55<left-shark>can we still download parler data?
13:20:16<no112>RJHacker11108 Wayback Machine afaik
13:20:22cajunjoel quits [Remote host closed the connection]
13:20:25<no112>left-shark Currently I think its a stream of 403's
13:20:29<Billy549>it'll be accessible on archive.org in its raw data but you can also access via Wayback iirc
13:20:31<left-shark>ah bummer
13:20:38nickyyyyy quits [Ping timeout: 244 seconds]
13:20:41<OrIdow6>It will be in the Wayback Machine at web.archive.org. If you want to download the raw data, it will be here https://archive.org/details/archiveteam_neparlepas but it is still uploading at present
13:20:55<no112>Just be patient and it'll all be up in no time
13:21:06<Dallas>So have gizmoid inadvertently painted a target on our back ? Lol
13:21:15<monoxane>i think its under /Users/YOU/Library/Containers/com.docker.docker
13:21:24<Billy549>I have the location - /Users/ME/Library/Containers/com.docker.docker/Data/vms/0/data
13:21:29<Billy549>I was just making sure I could delete it :P
13:21:32<RJHacker11108>so whats the difference between wayback and archive.org data?
13:21:40<monoxane>yea or you can just obliterate it from the docker desktop menu
13:21:44<Billy549>archive.org data is just data, wayback is a nice way to present it so you can see it iirc
13:21:48<no112>Yeah, i'd be worried about some alt-right loonies coming along and attempting, at least, to damage IA or AT
13:21:52<Billy549>there's my 50GB
13:22:31<RJHacker11108>I see. anyone actively working on ways to parse comments / posts from the raw data?
13:23:19<Billy549>idk c:
13:23:28<Billy549>Try asking in the channel for Parler, people might know there
13:23:42<Billy549>#neparlepas
13:27:31rolll joins
13:32:04<rewby>Is archiveteam.org struggling at the moment? It's being very slow
13:33:44rolll quits [Client Quit]
13:33:55<ave>rewby, cnr, works fine for me
13:34:41<Billy549>seems a tad slower than earlier, but fine otherwise rewby
13:35:01<rewby>It just gave me a cached copy from the wb machine
13:35:20<monoxane>probably IOPS related to getting 56tb dumped onto them
13:35:39<rewby>"The website is currently under high load, we have loaded a cached version for you from the Internet Archive's Wayback Machine"
13:35:42<EggplantN_>monoxane we ain't sending it to at.org :P we sending it to archive.org :P
13:36:01monoxane facepalms
13:36:57LegitD joins
13:37:28<Billy549>i'm having no issues with archiveteam.org at all rn
13:38:13Jon quits [Remote host closed the connection]
13:38:36<purplebot>Parler edited by Squidboy (+189, +gizmodo coverage) just now -- https://www.archiveteam.org/?diff=46147&oldid=46141
13:43:51LegitD quits [Remote host closed the connection]
13:44:41rolll joins
13:49:17jon (jon) joins
13:54:16RJHacker11108 quits [Remote host closed the connection]
14:00:04Jonboy345 joins
14:03:33wuzzl3 joins
14:05:30dofe quits [Remote host closed the connection]
14:11:13Arcorann quits [Ping timeout: 265 seconds]
14:16:03etnguyen03 (etnguyen03) joins
14:18:05Teabread joins
14:18:19<mgrandi>Hi new friends
14:18:21r3v2d0g joins
14:18:23<yano>Showdy
14:18:25<yano>*howdy
14:18:33<Teabread>hello!
14:19:09<mgrandi>1: writing a web app to better search the data is probably out of scope for this but might be useful for the internet as large
14:19:31<mgrandi>2: what is it stored in? Most everything AT stores are in WARC files, aka web archives
14:19:45<yano>it'll be in the WBM
14:19:58<mgrandi>Tldr they are basically the raw HTTP request with some additional metadata, you can view it in a text editor usually
14:20:14<left-shark>hello there
14:20:14<mgrandi>@yano are we really attempting to do that?
14:20:16<Teabread>Makes sense, but yeah tough to parse / look at usefully
14:21:16<mgrandi>There is an ecosystem of stuff that works on warcs alrrady
14:21:38<Teabread>Is the bottleneck right now internet bandwidth between archivers and AT server?
14:21:47<mgrandi>It's a bit harder when they are huge warcs of literally everything
14:21:54<mgrandi>And yeah, pretty much
14:21:57<Teabread>or is there actual parsing / conversion happening
14:21:59<@kiska>I mean...
14:22:11<Teabread>Can we uh... mail some big ass hard drives to some AT members/
14:22:34<mgrandi>How it works is there is the distributed work, and those go to target servers , that then process them into mega warc files and those are the things that are uploaded
14:22:48<@kiska>This is ingest: https://server8.kiska.pw/uploads/218ac8808b7d33ba/image.png This is IA taking out data: https://monitor.archive.org/cacti/graph_image.php?action=edit&local_graph_id=10681&rra_id=1
14:22:57<yano>mgrandi: i'm pretty sure that's what happens to anything backed up via AT projects
14:23:06<@kiska>I am sure you can see the discrepancy in speed
14:23:26<Teabread>yeahhh
14:23:28<mgrandi>Not everything that AT gets ends up in WBM
14:23:41<mgrandi>It does end up in IA though, but as a "item"
14:23:42<Teabread>Faster than I would expect getting into WBM though, that's nice
14:24:23<Teabread>I was expecting like single digit Gbps lol
14:24:45<@kiska>What 70Gbps ingest into our servers and 1Gbps out into IA
14:24:49<Teabread>I am definitely using IA and WBM interchangeably but I probably just mean IA in all cases so I will switch to that
14:24:59<@kiska>That would be not the optimal use of resources
14:26:13<mgrandi>WBM is a different thing yeah, I don't think the data is organized enough for it to be in WBM
14:26:44<mgrandi>We did a bit bucket archive a while ago and that very nicely translates to webpage urls so we had that in wbm
14:26:44<yano>oh
14:26:48<yano>i thought everything ends up in WBM
14:26:53<@jrwr>Overall this is a somewhat common issue with WBM, You can access all the data via URL
14:26:55<mgrandi>No
14:26:59<yano>TIL
14:27:01the_emerald_x joins
14:27:02<@jrwr>but not have a working website for the most part
14:27:50<@jrwr>Most of Parler will be broken for rendering, so using the website will be hard. Some places if they want can act like a proxy to the WBM and do custom rewrites
14:27:51<mgrandi>https://archive.org/details/aoc_among_us_twitch_stream_2020_10_20
14:28:03<mgrandi>That ^ is not WBM, just normal archive.org
14:28:25<@jrwr>Yes, thats a normal item
14:28:36<@jrwr>The Parler project was doing using Wget-AT / WARCS
14:28:54<Sanqui>basically, somebody could build a frontend for all the parler data
14:29:00<Sanqui>it's all there
14:29:08<Sanqui>just takes time & dedication
14:29:33<@jrwr>Yep and basic data (meaning if you know the exact url, will be there, also the WBM is pretty OK, just not searchable
14:29:33<Dallas>Sanqui: ah the two things I never seem to have lol
14:29:44emerald joins
14:29:56<@jrwr>It will show up like any other archived website
14:30:07<yano>will there be a bittorrent of the item on IA?
14:30:08<Sanqui>the primary goal of archive team is to preserve the data
14:30:13<@jrwr>in the wayback machine at a later date (about 2 months I would suspect)
14:30:27<Sanqui>generally, we move onto new projects before we figure out the "make it easily accessible" part
14:30:33the_emerald_x leaves
14:30:42<mgrandi>The urls we were getting from parler were like api.parler.com/ebevdjsnsjwo124 anyway
14:31:03<Sanqui>that's not intentional, it's just a side effect of stuff going down all the time and our priorities
14:31:35<emerald>hello hello
14:31:43<Dallas>yano: I believe all items are assigned a torrent
14:31:50<yano>Dallas: oh nice
14:32:39emerald quits [Client Quit]
14:32:39<mgrandi>It's a HTTP torrent yeah
14:32:42<@jrwr>@yano yes and no, The project is split into lots of items, (so thousands of torrents)
14:33:00emerald joins
14:33:10<EggplantN_>yeah we did 3G megawarcs mostly
14:33:13<mgrandi>Each IA item has a torrent file generated and then it uses the concept of http seeding to basically download it
14:33:13<EggplantN_>so LOTS of torrents
14:33:13<Dallas>although no torrent link rn https://archive.org/details/archiveteam_parler_20210110180205_b63787bc
14:33:14<EggplantN_>lol
14:33:32<@jrwr>Tasks are... backlogged all to hell
14:33:34<Dallas>Oh I see how it is, kiska's name is on everyone of these ;p
14:33:35<@jrwr>give it time
14:33:43<mgrandi>@Dallas: it takes time for a item to be "derived", aka processed and have stuff like the torrent generated
14:33:43<Dallas>not surprised lol
14:33:54emerald quits [Client Quit]
14:33:55<Dallas>ah okay makes sense
14:34:01the_emerald joins
14:34:05the_emerald quits [Remote host closed the connection]
14:34:18<Dallas>Actually kiska can have his name on them since the nazi's are mad atm loool
14:34:30emerald (emerald) joins
14:34:42<mgrandi>And yeah, everyone uploads to their machine and then they upload it to IA using their account
14:35:25<yano>jrwr: oh yikes
14:35:40<@jrwr>this is normal
14:35:42<emerald>So I wonder what's next
14:35:43<@jrwr>all of it is normal
14:36:05<nnnoice>Dallas They're mad because someone has a copy of their posts? :( :(
14:36:10<nnnoice>:')
14:36:30emerald quits [Client Quit]
14:36:39<mgrandi>http://archiveteam.org/index.php?title=Deathwatch is what is next
14:36:53emerald (emerald) joins
14:37:01<mgrandi>#flashbang we are trying to rescue swf files before sites start deleting SWFs if they are smaller
14:37:03<Dallas>These people raided a gov building on live streams are are worried about their public parler posts being saved smh my head
14:37:19<emerald>do you think td.win or similar sites are worth saving?
14:37:32<Billy549>every site is probs worth saving to some degree
14:37:37<mgrandi>Td.win is hard cause they are in cloud flare attack mode
14:37:55<etnguyen03>communities.win/c/thedonald isn't though (I think?)
14:38:02<@jrwr>AT really hates cloudflare
14:38:07<@jrwr>for that atlest
14:38:17<@jrwr>Gab.com is another one
14:38:19<mgrandi>Captchas everywhere
14:38:33<emerald>gab? i'm not invested in american politics enough to know how bad it is
14:38:38<emerald>only that it's... y'know. one of those.
14:38:39<Billy549>Gab's like Parler iirc
14:39:23<nnnoice>https://thedonald.win/ Isn't using "under attack" mode, at least not for me
14:39:27emerald quits [Client Quit]
14:39:42<EggplantN_>I mean, thedonald.win probably? you could reach out to them directly
14:39:48emerald (emerald) joins
14:39:59<emerald>oops, not sure why i cut out there.
14:40:11<Billy549>thedonald.win 302d to authentication.win then back agin
14:40:13<Billy549>again*
14:40:39<mgrandi>There are different levels of it, I tried yesterday's and you have to solve the JS challenge before it lets you through
14:40:59<Billy549>yeah that's based on if they deem your browser bad(tm)
14:41:10<emerald>if gab's like parler, is there any reason why we haven't gone ahead and started archiving it too?
14:41:18<Billy549>Cloudflare's in the way
14:41:24<emerald>oh do they use cf
14:41:27<Billy549>apparently
14:41:59<nnnoice>I also use the Privacy Pass extension... which may help bypassing captchas and that under attack page (I think)
14:42:26<Billy549>i mean if we all start running scrapers
14:42:35<Billy549>Cloudflare might start getting suspicious and then automatically throw everyone more CAPTCHAs
14:42:57<mgrandi>But yeah, checkout the deathwatch page, maybe help with flashbang , we need people to help verify sites and help figure out custom rules for certain sites
14:43:14<mgrandi>That is attack mode and it's set by the site owner @ Billy549
14:43:59<Billy549>mgrandi: I'm aware Attack Mode can be enabled by site owner
14:44:00sadstan joins
14:44:09<Billy549>but they might automatically pick something up if you're all spamming it
14:44:15Webuser247 joins
14:44:23matt54 joins
14:44:42<Billy549>sadstan: if you wanna help, URLTeam2 could probably do with another Docker worker - you can pick the setting in ArchiveTeam worker to let AT decide
14:44:44<Billy549>and it'll auto-pick
14:46:03<mgrandi>Is shreddit ready yet?
14:46:13<@kiska>It's running
14:46:35<@kiska>Well it had resumed service after my target isn't on fire from parlef
14:46:39<mgrandi>I guess the auto pick might switch between urlteam and shreddit then
14:47:15<mgrandi>Anyway bedtime, hope you all stick around :)
14:47:18<nnnoice>It would be nice if the Warrior received other tasks than archiving URL shortners
14:47:26sadstan quits [Remote host closed the connection]
14:48:02Webuser247 quits [Remote host closed the connection]
14:48:55KaiserWillhelmII joins
14:49:00<mgrandi>That is a ongoing effort to fix that up, see #warrior :)
14:50:28<Dallas>https://archiveteam.org/ is dieing I assume bc it's been shared in so many articles
14:50:46<@jrwr>This is rather normal for the wiki
14:51:01<@jrwr>its why I have the wayback machine iframe when php is overloaded
14:51:18KaiserWillhelmII quits [Remote host closed the connection]
14:53:11<mgrandi>Maybe #archiveteam-dev too, the warrior docker image should work for most DPoS projects so hopefully we can fix that up to make it so all DPoS projects can go through there too
14:54:31<nnnoice>I was looking at the way archive.is archives td.win and they seem to bypass the captcha page
14:54:49<nnnoice>They get the captcha page, but then move on and save the page?
14:55:32<nnnoice>I mean, it's so cheap to autocomplete captchas, so... maybe that's what they're doing
14:55:50<Dallas>you need to think of a clever channel name for it, that's the most important part
14:56:23<@jrwr>gabber
14:56:57<nnnoice>gabberthembythehtml
14:57:05<nnnoice>it's a bad one, I know
14:57:23<@kiska>Gagboon
14:57:25czechball quits [Remote host closed the connection]
14:59:26<Dallas>Hang one
14:59:31<Dallas>Wow ty autocorrect
14:59:43<Dallas>gabgone < boring but effective
15:00:28<jut>gaboned
15:01:13<jut>gabwned
15:04:03<thuban>gabated
15:05:25greyred joins
15:05:40<thuban>gabjured. gabandon, gabort.
15:07:10fabz joins
15:07:13<Billy549>gift of the gab?
15:07:30<Billy549>https://dictionary.cambridge.org/dictionary/english/gift-of-the-gab
15:07:47<monoxane>gabageddon
15:08:59<Billy549>that is a good name
15:09:18matt87 joins
15:09:36<purplebot>Parler edited by Jrwr (+219) 8 minutes ago -- https://www.archiveteam.org/?diff=46148&oldid=46147
15:10:10<fabz>hi guys! I would like to contribute for the parler project and the docker instruction says i need an image address which i don't have, where can i find it?
15:11:19<emerald>I think the parler project is over now
15:11:23<Billy549>yeah, Parler is over now
15:11:25<phuzion>fabz: #neparlepas, but that project is done.
15:11:25<Dallas>fabz: that project is complete now
15:11:30<emerald>The site is down as AWS flipped the big switch
15:11:31tanooki (tanooki) joins
15:11:43matt54 quits [Ping timeout: 244 seconds]
15:11:53<Billy549>fabz: you can still run Warrior, which can still do URL Shortener stuff - see https://archiveteam.org/index.php?title=Warrior
15:11:53<fabz>uh, ok
15:12:05<Dallas>If you still wish to contribute to other efforts check out #gitgud #urlteam #shreddit #flashbang
15:12:08MeanieBeanie joins
15:12:18<fabz>thanks guys!
15:12:18DataHog (DataHog) joins
15:12:27<Billy549>no worries, thanks for wanting to help out :D
15:12:34<EggplantN_>Please as many as possible come and help out at #shreddit
15:12:42<EggplantN_>20 concurrent/IP is the limit for now
15:12:49<EggplantN_>But we need as much help as we can over there :)
15:13:32<blue507>is that project archiving Reddit?
15:13:36<Billy549>Yep :P
15:14:31<emerald>I'll spin up a VPS to do it later!
15:16:00dltj joins
15:16:02fabz quits [Remote host closed the connection]
15:16:32kor joins
15:19:11insomniacrando joins
15:19:16DataHog quits [Remote host closed the connection]
15:19:53user1231282 joins
15:20:32user1231282 quits [Remote host closed the connection]
15:20:43nnnoice quits [Remote host closed the connection]
15:21:44mutantmonkey quits [Ping timeout: 240 seconds]
15:23:01Teabread quits [Remote host closed the connection]
15:24:38greyred quits [Ping timeout: 244 seconds]
15:28:18AdminRoot joins
15:36:37xenocy joins
15:36:48mrquarantino joins
15:36:51NF885 joins
15:37:15PhantomTypist joins
15:37:18yolo0002 joins
15:37:45BPCZ joins
15:38:18<ave>Re: Gab channel name, gabgrab maybe?
15:40:03greyred joins
15:40:30OpSecNoob joins
15:40:34greyred quits [Remote host closed the connection]
15:41:10<insomniacrando>gabbag
15:41:15<OpSecNoob>Hi, I'm getting an error when trying to run the warrior: "failed to resize tty, using default size" is stuck in a loop. Anyone got an idea how I can fix this?
15:41:46<BPCZ>isn't gab running on pleroma? I'm pretty sure the rest of the fediverse has to blacklist Gab so they don't accidentally start hosting it
15:42:01<Billy549>It's got Cloudflare in the way for scraping
15:43:46jhkjhk joins
15:43:55Sage joins
15:44:03Sage quits [Remote host closed the connection]
15:44:12<mrquarantino>no, it's a fork of mastodon https://github.com/mordechaim/gab-social
15:45:10xenocy quits [Remote host closed the connection]
15:47:16AdminRoot quits [Remote host closed the connection]
15:49:49HotSwap joins
15:50:59wuzzl3 quits [Ping timeout: 244 seconds]
15:51:25Deluxe joins
15:52:35jlt7823 joins
15:54:16Barto quits [Ping timeout: 252 seconds]
15:55:19Deluxe quits [Remote host closed the connection]
15:57:53NF885 quits [Remote host closed the connection]
15:58:30NF885 joins
15:58:56Hackerpcs quits [Quit: Hackerpcs]
15:59:15<OpSecNoob>Hi, I'm getting an error when trying to run the warrior: "failed to resize tty, using default size" is stuck in a loop. Anyone got an idea how I can fix this?
16:00:53Hackerpcs (Hackerpcs) joins
16:01:25Interested joins
16:01:26Y48fyrCgHPR7d18W joins
16:01:47<ave>BPCZ, they're running on a fork of mastodon, and they're no longer federating
16:04:56AnotherBot joins
16:05:03dltj quits [Client Quit]
16:08:27mrquarantino quits [Remote host closed the connection]
16:08:50tanooki quits [Remote host closed the connection]
16:08:59DopefishJustin quits [Remote host closed the connection]
16:09:17yolo0002 quits [Remote host closed the connection]
16:14:14kor quits [Ping timeout: 244 seconds]
16:14:25<Dallas>OpSecNoob: are you running warrior vm or docker
16:14:55DopefishJustin joins
16:18:17abc123 joins
16:20:35abc123 quits [Remote host closed the connection]
16:21:12ddd joins
16:27:27kor joins
16:30:55Thomashs1 joins
16:31:26britmob quits [Read error: Connection reset by peer]
16:31:35<purplebot>Parler edited by Jrwr (+1657), Jrwr (-1875, -Redacted for a rewrite by sketchcow) 10 minutes ago -- https://www.archiveteam.org/?diff=46153&oldid=46148
16:31:41Barto (Barto) joins
16:32:22ddd quits [Remote host closed the connection]
16:32:29<@jrwr>Ugh, the wiki is impossible to use
16:33:31str8m4d joins
16:34:18str8m4d quits [Remote host closed the connection]
16:38:09<@jrwr>its currently buried, doing about 200rps with 180 of them being 508s (Heavy load page)
16:41:11JustSomeGuy joins
16:43:56notRelevant joins
16:44:03TestCrap joins
16:45:13<Dallas>SketchTheCow: out of interest are you putting a statement out or is the plan just lay super low ?
16:47:48<Billy549>"Here is what we need to do. We need to tie off the process, finish so there is a dataset, and then keep our heads down. "
16:49:58britmob joins
16:56:37<purplebot>Parler deleted by Jrwr (+0) 24 minutes ago -- https://www.archiveteam.org/?title=Parler
16:57:50matt87 quits [Remote host closed the connection]
16:58:04ragu quits [Ping timeout: 240 seconds]
16:58:56KronK joins
16:59:02Webuser355 joins
16:59:08groente_ joins
16:59:23operator joins
17:00:46<mazet>thedonald.win is behid cloudflare, has anyoe tried to get it to leak their origin servers ?
17:01:36<purplebot>CNET Forums edited by Nintendofan885 (+5) 2 minutes ago -- https://www.archiveteam.org/?diff=46155&oldid=45879
17:01:41<@jrwr>@mazet sign up, get a confim email and send me the email headers :#
17:01:42<mazet>i might hve to create a user there, hope i dont get cance fro that
17:02:03<mazet>might have to throw away my computer after doing that
17:02:08<EggplantN_>jrwr apparently they're limiting it to only CF IPs
17:02:15<mazet>i have some time, gonna go and take a look
17:02:17<@jrwr>As they should
17:02:21<@jrwr>its what I would do
17:02:22kageurufu (kageurufu) joins
17:02:28<@jrwr>Then GG nore
17:02:32<operator>CF?
17:02:40<Dallas>This is where the workers proxy would actually be a possible help
17:02:40<@jrwr>CloudFlare
17:02:41<mazet>parler's devops seems to have been done by an illiterate child, maybe thedonald.wi is of equal galaxy brain power
17:02:56<Dallas>mazet: what devops...
17:03:08<@jrwr>Otherwise, Archive does not condone hacking the planet
17:03:13<@jrwr>*team
17:03:21<mazet>Dallas: the ones that are paying the aws bandwidth bill from all of us scraping the shit out of them
17:03:59<mazet>jrwr: looking for misconfigurations nd trying to escalate privileges to take over a site are completely different ;)
17:04:22<operator>They are just going to shutdown the site
17:04:42<operator>and windup the company
17:05:36<operator>Imagine that team trying to run a bare metal datacenter
17:07:11dillstreetproductions joins
17:07:26<Webuser355>their ISP would boot em anyways
17:09:08<Dallas>epik
17:09:42<operator>yeah trye
17:09:48<@arkiver>what is this about?
17:09:55<operator>any idea how the misinformation on reddit got spread?
17:10:25<Webuser355>anyone can post on the internet and any idiot can screenshot it
17:10:42<Dallas>operator: people not vetting info
17:11:18<operator>Poor twillo got blamed in that
17:11:31<operator>it was kinda bizarre
17:12:41ddd joins
17:13:01<Dallas>People keep joining the parler channel and asking questions, do we have a set response or should we ignore them ?
17:13:19<@jrwr>Refer to the topic
17:13:24<@jrwr>(in the channel
17:13:29JustSomeGuy quits [Remote host closed the connection]
17:13:36<Dallas>Okay cool
17:14:05wuzzl3 joins
17:14:15sliccricc joins
17:14:31jhkjhk quits [Remote host closed the connection]
17:15:32notRelevant quits [Remote host closed the connection]
17:16:23Jonas joins
17:16:45NF885 quits [Ping timeout: 244 seconds]
17:18:11vivid-noble joins
17:18:50<Dallas>Ugh, I’m having a nap, then I’ll endeavour to start documenting Gab and what we can and can’t archive https://twitter.com/Shayan86/status/1348672777822351368
17:19:50reddog83 joins
17:21:32aphitex joins
17:25:47kelsair joins
17:26:31limonene joins
17:27:26DogsRNice (Webuser299) joins
17:27:26<sliccricc>won't being powered by mastodon make it pretty difficult to shut Gab down? I suppose there's still the case of deleted posts, though.
17:27:36jlt7823 quits [Ping timeout: 244 seconds]
17:29:05<operator>Other mastodon servers have blocked Gab
17:29:10MM46 joins
17:29:29<operator>they are basically running their own parallel infrastructure
17:29:40Thomashs1 quits [Ping timeout: 244 seconds]
17:29:49<sliccricc>sure, but with the ability to add and remove servers in varying jurisdictions with relative ease, no?
17:30:11<operator>yeah I suppose so
17:30:25<operator>easier than if you are locked into AWS
17:30:27<sliccricc>compared with AWS being single point of failure anyway
17:30:27<sliccricc>yea
17:31:28<operator>oh do yous also run http://fileformats.archiveteam.org?
17:31:46<sliccricc>and, if they were so inclined they could whip up some scripts and ask people to install them to add their PCs, phones, rPis, whatever to their federation to support theh cause
17:31:51kelsair quits [Remote host closed the connection]
17:32:19<operator>would mastodon be able to handle that?
17:32:22reddog83 leaves
17:32:37<sliccricc>no idea, just thinking out loud. i'm not that familiar with it
17:33:00<operator>I don't think it's that decentralized
17:33:16<operator>it's not bittorrent
17:37:56wuzzl3 quits [Ping timeout: 244 seconds]
17:38:38seyonce joins
17:38:41<sliccricc>just curious about the overall endgame here. if they manage to silence Gab, where next? and if mastodon et. al are currently "not bittorrent" -- is that where they're headed? particularly with threats from both sides against sec 230 CDA. food for thought. anyway, good day all!
17:38:58Jonas quits [Ping timeout: 244 seconds]
17:40:16<jut>sliccricc: IRC?
17:40:29<@JAA>Dallas: No, project channel logs are generally not meant to be public. Also, no channel notice informing people of that.
17:40:35<@JAA>justcool393: Yes, that's intentional.
17:40:45<Dallas>It’s not like that it’s more like if I have an account dallas@myhost.com and it’s part of the federated network I’m able to follow you@yourhost.com and jason@textfiles.com and vice versa
17:40:55Dallas deletes logs 👀
17:40:59<justcool393>oh
17:41:16jeff-kao-propublica-org joins
17:41:57<justcool393>JAA: wait what are you referencing
17:42:45<Dallas>My first message was a reply to sliccricc my /me was a joke to you
17:43:14aphitex quits [Remote host closed the connection]
17:43:52limonene quits [Remote host closed the connection]
17:44:14<atphoenix>Recent projects are mostly done with docker images and containers (and mostly used on Linux x86 hosts or Linux VMs) (I don't know enough about Docker and these containers to speak to cross-architecture and cross-OS compatibility). If the device can run docker, presumbly it can be used as a project worker. There are still bandwidth and disk space considerations for some devices, as project requirements may vary
17:44:14<atphoenix>significantly. Some projects tend to be under 1 MiB per item. Others can exceed 10 GiB per item.
17:45:47jeff-kao-propublica-org quits [Remote host closed the connection]
17:46:05<justcool393>jut: more likely something like Matrix
17:46:06jeff-kao-propublica-org joins
17:46:09<justcool393>but regardless
17:47:56xavi (xavi) joins
17:48:02Nessie joins
17:49:20<justcool393>mazet: TD.win apparently only accepts connections from Cloudflare
17:49:28Nessie quits [Remote host closed the connection]
17:50:57<justcool393>long live espernet
17:51:15holbrooke leaves [bye!]
17:51:29kiiwiiwastaken (kiiwiiwastaken) joins
17:51:34<justcool393>*eris
18:01:36WH1sp3r joins
18:03:19bluecmd joins
18:04:01<bluecmd>Hello! I want to help you by running a Warrior VM. I see it is GPG signed but I can't seem to find a reference to what GPG key to expect it to be signed by.
18:04:28<bluecmd>Browsing the Wiki is quite slow as I'm sure you are very aware off, so I might be missing something :-)
18:12:57willow joins
18:13:39<Dallas>#warrior may be the best place to ask that :)
18:14:35<bluecmd>Thanks! I will try :) Didn't know it existed!
18:14:54megotperc joins
18:15:35archzz joins
18:16:12Webuser355 quits [Remote host closed the connection]
18:17:35wumbo joins
18:23:43Jonboy3451 joins
18:25:04ddd quits [Remote host closed the connection]
18:27:04Jonboy345 quits [Ping timeout: 240 seconds]
18:27:18mikeairw joins
18:27:21buffalo_man joins
18:27:31<buffalo_man>yo guys
18:27:36AK (AK) joins
18:28:15mikeairw leaves
18:29:44<EggplantN_>Hi buffalo_man
18:32:10yawkat (yawkat) joins
18:35:39ccafe joins
18:36:04<insomniacrando>I firmly insist that any attempt to stuff from gab should be called gabbag
18:36:18SockUps joins
18:36:35Jonas joins
18:36:36mgd joins
18:36:45megotperc quits [Client Quit]
18:39:04<jut>I choose you gabort
18:39:26<ccafe>insomniacrando: consider gabbage :P
18:40:04<insomniacrando>damn that's almost worth it not being a palindrome anymore
18:40:22<Dallas>Gab bag like grab bag ? Like the bag you grab when you run out of a burning building ? I like the metaphor
18:40:28<@JAA>I like gabbage.
18:40:35Dallas tries nodding smartly
18:40:49<insomniacrando>indeed dallas
18:41:16<jut>gabbage as in baggage or garbage?
18:41:36<ccafe>In all seriousness though gab should be fairly straightforward, it's activitypub based iirc. Mastodons evil twin
18:41:47<insomniacrando>like cabbage
18:41:59PhantomTypist quits [Remote host closed the connection]
18:42:10<yawkat>is cloudflare not going to be a problem?
18:42:11<@arkiver>if JAA likes gabbage lets to gabbage
18:42:19<groente_>i had scripts for scraping gab before they moved to mastodon, which was pretty trivial back then
18:42:19grumpton joins
18:42:19<@arkiver>do
18:42:21<@arkiver>#gabbage
18:42:42<groente_>i'd say chances are good it will still be pretty easy
18:42:49<ccafe>jut: Sounded like cabbage in my head and made me giggle is all :P
18:43:03<insomniacrando>gabbage patch
18:43:40Barto quits [Ping timeout: 252 seconds]
18:43:48mholland joins
18:43:51yloz joins
18:43:59Obelar_ (Obelar_) joins
18:44:06sickit joins
18:44:09xaviair joins
18:44:23<SockUps>try looking into clouthub
18:44:52<SockUps>i think the exodus is probably bigger to clout than gab
18:45:15<SockUps>https://g.clouthub.com/
18:45:23lincoln (lincoln) joins
18:45:51baviair joins
18:46:00Barto (Barto) joins
18:48:01yloz quits [Remote host closed the connection]
18:48:04xavi quits [Ping timeout: 252 seconds]
18:48:23lincoln quits [Client Quit]
18:49:37xaviair quits [Ping timeout: 265 seconds]
18:49:57<insomniacrando>clouthub is much harder to come up with channel names for though
18:50:01SockUps quits [Remote host closed the connection]
18:51:12megotperc joins
18:51:22lincoln joins
18:51:40<AK>Hmm, cloutchasers?
18:51:54grumpton quits [Remote host closed the connection]
18:51:58<Aoede>clouddud
18:54:43<@arkiver>jrwr: how is your wiki doing?
18:54:54<@jrwr>its down to about 100rps
18:55:01<yano>not bad
18:55:15rio joins
18:55:21<@jrwr>I keep seeing hits from The Knowledge AI
18:55:35rio quits [Remote host closed the connection]
18:56:40<SketchTheCow>AND HOW IS EVERYONE'S DAY
18:56:56<yano>surviving
18:56:57<AK>Not bad, how about yours?
18:57:00Webuser572 quits [Ping timeout: 244 seconds]
18:57:52newguy22 joins
18:57:55<SketchTheCow>Zesty
18:58:10<insomniacrando>cuz of the .zst files?
18:58:22<@jrwr>Booo, bad jokes are bad
18:58:23<@jrwr>:p
18:58:33<jut>I wake up to a screenshot calling us left wing extrimist and it when downhill from there
18:58:40<yano>last time i made a pun i got kicked
18:59:03seyonce quits [Ping timeout: 244 seconds]
18:59:28<Dallas>The finger guns may have had something to do with it too...
18:59:35<yano>(✌ﾟ∀ﾟ)☞
18:59:39yano is kicked by @jrwr [yano]
18:59:44yano (yano) joins
18:59:44<@jrwr>yep, I think that was it
18:59:45<yano>lol
18:59:47<yano>haha
18:59:48<Dallas>ahahahaha
18:59:52<Dallas>Perfection
18:59:55newguy22 quits [Remote host closed the connection]
18:59:57<yano>i have 1 more alias for finger guns
18:59:59Dallas claps
18:59:59<yano>lol
19:01:03jonnyd joins
19:01:56yano *takes notes; jrwr doesn't like fingerguns*
19:02:10<AK>So no finger guns, but ban puns are good? (Just working out the rules as a newbie)
19:03:01<phuzion>AK: sounds like you've got your finger on it. And that's a bad finger pun.
19:03:18kiiwiiwastaken quits [Remote host closed the connection]
19:03:51<yano>lol
19:04:22Webuser572 joins
19:04:23AK quits [Remote host closed the connection]
19:04:40AK joins
19:04:52<phuzion>Ah, looks like archiveteam.org got linked by gizmodo which is why the wiki is straight up shitting itself haha
19:05:22<phuzion>And the gizmodo article is on the front page of /r/all on reddit
19:06:04jtt123 (jtt123) joins
19:09:09<yano>yikes
19:09:53<Dallas>oh dear god
19:10:50lmx joins
19:12:15justsomeonewhoarchives joins
19:13:16<SketchTheCow>This is my request:
19:13:26<SketchTheCow>Just avoid interacting with folks until... let's say Friday
19:13:50<SketchTheCow>Let impeachement and the inevitable explosion at some currently unknown building take the stage before then
19:14:02Jonas quits [Ping timeout: 244 seconds]
19:15:12<Dallas>sounds good... maybe lock the parler channel? Or will that just drive people here :/
19:15:37<@jrwr>We don't lock channels
19:15:45<SketchTheCow>Yeah, we REALLY don't lock channels
19:15:56<SketchTheCow>Also I am comfortable with that being a canary for general zeitgeist
19:16:04<phuzion>Hell, even the old channels on efnet aren't +m right now lol
19:16:32<@jrwr>Everything is /fine/ we did our jobs, now its time for others to do theirs
19:16:33<SketchTheCow>I'm also comfortable with dying over this, because then I will ascend to the top of archiveteam again
19:16:50<SketchTheCow>My star's getting dim with all these new young jerks taking my fire
19:17:14<SketchTheCow>But I really don't need people who didn't sign up for violence and mis-aligned rage to get caught in the crossfire
19:17:22<SketchTheCow>Whole lives ahead of them, etc.
19:17:25<@jrwr>The day you don't have a raging fire going on is the day the world has already come to a end and everything has been saved onto plates of glass floating in space
19:17:59<justsomeonewhoarchives>that doesn't sound like a long term solution at all
19:18:31<Sanqui>everybody: parler
19:18:37<Sanqui>me: obscure czech 2000 php freehost
19:18:49<Dallas>Doing gods work Sanqui
19:18:53Dallas salutes
19:18:58<justsomeonewhoarchives>fun fact, I hadn't even heard about parler before two days ago or so... then again I'm not from the us
19:19:04Eighty (Eighty) joins
19:19:21operator quits [Remote host closed the connection]
19:19:41<SketchTheCow>Imagine someone looked at Twitter and went "Too Erudite"
19:20:00<Dallas>SketchTheCow: fair point, I'm sure whatever ridiculous shit happens next will be far more interesting to people haha
19:20:06<SketchTheCow>Yes
19:20:12<SketchTheCow>I just went the next thing to go in
19:20:32<SketchTheCow>Look, I've run TEXTFILES.COM for 23 years - I've seen some shit
19:21:00<SketchTheCow>I was interviewed by the FBI in my underwear on my front stoop one time
19:21:01<Dallas>Twitter can be toxic af idk why you'd look at and go 'hmm needs nazis and more death threats"
19:21:03<justsomeonewhoarchives>And we love you for it, especially when you give a talk about some of the weirder shit
19:21:27<Dallas>ahaha, nooo, that's pretty funny
19:21:42<phuzion>If you don't believe that SketchTheCow has been through some shit, you need to watch his DefCon 17 talk: That Awesome Time I Was Sued For Two Billion Dollars https://www.youtube.com/watch?v=KSWqx8goqSY
19:22:13<@jrwr>@SketchTheCow Pics or it did happen (FBI interview)
19:22:18jonnyd quits [Remote host closed the connection]
19:22:25<Dallas>phuzion: ty
19:22:41<phuzion>Dallas: very welcome
19:22:57<@jrwr>SketchTheCow: do you still get letters from the guy?
19:23:15<SketchTheCow>No, he hasn't figured out how to find me
19:23:26<SketchTheCow>He's pretty angry about it though
19:23:44<@jrwr>hahaha
19:23:45mholland quits [Remote host closed the connection]
19:24:00alamazy joins
19:24:07<SketchTheCow>He tried to find me
19:24:21<SketchTheCow>And since PAULANDREWMITCHEL goes to a Paul Mitchell Hair Salon, that made him angriet
19:24:24<SketchTheCow>Good times
19:24:33<SketchTheCow>"HELLO, PAUL MITCHELL"
19:24:39<SketchTheCow>Imagine getting that when you call
19:24:39alamazy quits [Remote host closed the connection]
19:25:01<Dallas>Ah the day's when 1tb of data was loads, this is so nostalgic *pretends I wasn't 12 in 2009*
19:25:16<@jrwr>the 2PB we moved when we did Google Plus
19:25:21<@jrwr>poor IA
19:25:27<@jrwr>was over a year to get it all uploaded
19:25:51<Dallas>That was how I stumbled in here, good times, that project was less death threaty
19:25:59<Sanqui>and i feel bad for my 1gb's worth of Accidental Recursive Calendar
19:26:07<justsomeonewhoarchives>So I've had a probably stupid idea. I was thinking to myself: There are a lot of pages out there that no one knows about that nevertheless might be interesting to archive, maybe even before they go full public. So I thought to myself why not just go ahead and use certificate transparency logs to get a list of at least all https enabled websites out there and start archiving that!
19:26:13<justsomeonewhoarchives>Turns out ... not that easy, especially if the waybackmachine ratelimits you about 10 pages in. So the question is: Is this an absolutely stupid idea or is this actually something that would be interesting to do and might be worth putting some time into?
19:26:49<SketchTheCow>It's a stupid idea but that never stopped anyone
19:26:50<Sanqui>one of my bigger confidence scores as to whether to archive a website is if it's over plain http
19:26:51<phuzion>justsomeonewhoarchives: any time you try to do anything to "all of the internet" it generally ends in catastrophic failure.
19:27:37<Sanqui>...now i'm seeing browsers deprecating plain http in 10 years and us starting a project to archive everything that's http only
19:27:38<Sanqui>jesus
19:27:41<Sanqui>this vision is too real
19:27:50<SketchTheCow>Stop smoking
19:27:53<SketchTheCow>you did it
19:27:59<SketchTheCow>you hit the end level boss of blazing up
19:28:11<@JAA>In 10 years? Didn't Google already announce something in that direction?
19:28:13<Sanqui>i truly wish i could blaze it tonight.
19:28:14<justsomeonewhoarchives>SketchTheCow: True. Being unable to do anything with the resulting archive is something that kind of stops you though, as that is pretty useless.
19:28:25<AK>10 years! Probably gonna need to start next week then to get it all done
19:28:34<justsomeonewhoarchives>phuzion: I get that point entirely. We are talking about 2 billion urls approximately
19:28:37OpSecNoob quits [Remote host closed the connection]
19:28:50<Dallas>I already did Sanqui :p
19:28:57<justsomeonewhoarchives>I'm also not talking about deep crawling every single page, but basically taking a snapshot of it, maybe a couple well known sub-urls
19:29:22<Dallas>Yeah I thought at least full page warnings about http sites where coming in a few years ?
19:29:36<justsomeonewhoarchives>Dallas: Well we are at default https at least soonish
19:30:04<phuzion>justsomeonewhoarchives: so, you have to crawl the page one way or the other. If you're gonna do something like that, at least make a database of the URLs accessible and searchable so we can load shortened URLs into urlteam?
19:30:22<justsomeonewhoarchives>if full page warnings don't break any precious enterprise workflow we'll get it sooner rather than later
19:30:25<@jrwr>There is
19:30:31<@jrwr>The warc format has the CDX
19:30:35<@JAA>Ah no, it was Mozilla discussing that in 2015. https://blog.mozilla.org/security/2015/04/30/deprecating-non-secure-http/
19:30:40<@jrwr>its a data index of the contents of the WARC
19:30:45<Dallas>Gab is such an eloquent place of discourse.... https://up.dallasthe.dev/firefox_24gLFHvi07.png
19:31:08<AK>justsomeonewhoarchives: They 100% will break stuff. There'll have to be an easy way to disable the warning for it to get anywhere I suspect
19:31:26<phuzion>Dallas: Ah yes, such eloquence and intelligence coming from..... "Cat turd"
19:32:33<justsomeonewhoarchives>phuzion: so my basic plan of action was this so far: parse certificate transparency logs for all urls contained therein that are not wildcards (already got that working with the help of some tools other people wrote (axeman, certstream, etc.), 2. check if url is reachable and returns a 200 response, the question is what to do after that right now
19:33:14<justsomeonewhoarchives>I tried checking waybackmachine whether the page is available there already (which a lot already are) and automatically triggering a pagesave if that wasn't the case, but that broke about 1 minute into the process due to ratelimits
19:33:31<@jrwr>yep
19:33:33<@jrwr>and you will
19:33:49<@jrwr>the internet archive has other data brokers they get web data from and their do their own crawling
19:34:02<justsomeonewhoarchives>I also don't intend to break archive.org by just polluting it with nonsense requests
19:34:03<@jrwr>we are not even close to the main source of data for the wayback machine
19:34:22<justsomeonewhoarchives>though as a single person thats probably not going to happen anytime soon
19:34:26<phuzion>justsomeonewhoarchives: I mean, if you're gonna be saving that many pages, you're gonna have to WARC them yourself, convince IA that you're trustworthy not to tamper with the data, and then MAYBE you can get them ingested into IA. You're not gonna be able to use the "Archive this page" button for 2B pages.
19:35:29<justsomeonewhoarchives>phuzion: exactly why I came to the people who are most familiar with archiving a ton of things (even if they might sound stupid!) and getting them into IA!
19:35:47<Sanqui>justsomeonewhoarchives: sometimes less is more
19:35:48<@jrwr>Overall we have two projects in place to archive random websites
19:35:56<Sanqui>focus on single things you know that matter
19:36:03<Sanqui>rather than casting a too wide of a net.
19:36:04jut quits [Ping timeout: 240 seconds]
19:36:19<@jrwr>we tend to keep them out of the limelight to keep away trolls and people not really good at archiving things
19:36:44<@jrwr>most people are not good at archiving things
19:37:10<yano>this is why i just run workers and chew bubble gum
19:37:14<yano>and i'm all out of bubble gum
19:37:17jut (jut) joins
19:37:18godane (godane) joins
19:37:26<justsomeonewhoarchives>Sanqui: I would if I knew what mattered in the first place
19:37:34<Sanqui>justsomeonewhoarchives: do you have any hobbies
19:37:59<justsomeonewhoarchives>Sanqui: As a matter of fact... I do! Random weird projects is one of them
19:38:15<phuzion>justsomeonewhoarchives: I'll reiterate my earlier point. Almost any time you try to do anything to "all of the internet" it generally ends in catastrophic failure. This isn't because the internet is out to get you, or that you're bad at this or anything. The internet is absurdly, ridiculously, hilariously huge. Orders of magnitude more huge than you realize.
19:38:42<justsomeonewhoarchives>phuzion: Oh I'm pretty certain I get the scale of stupid this idea ranges in
19:38:55<Sanqui>by focusing on preserving topics familiar to you and working with the communities you know, you can make the maximum impact
19:39:32<justsomeonewhoarchives>I'll admit when I found out that 0.1% of a single CT log amounted to 2mil urls I was a tad bit surprised once again by the scale
19:40:00<Sanqui>here's what i'll tell you: preserve one website and preserve it well, then preserve ten, then preserve a 100, before you think about 2 millions
19:40:16<phuzion>^^^
19:40:38<AK>What's the "best" way for me/us/people to help right now? I'm guessing it's running either a Warrior or the scripts off github? (Just want to make sure I'm doing the right thing)
19:40:43<justsomeonewhoarchives>Sanqui: I'm totally aware that from a pure preservation standpoint its probably complete garbage to try and save the index page of every url out there, it still sounded like a fun if stupid idea.
19:41:07<Sanqui>it's not complete garbage
19:41:10<Sanqui>it's just really, really hard to do.
19:41:43<phuzion>justsomeonewhoarchives: If you want to help ArchiveTeam, we have PLENTY of work to go around without starting new projects. URLTeam has an enormous backlog of URL shorteners that need analyzed and entered into the tracker. We can ALWAYS use more URLteam runners. The Wiki always needs cleanup and documentation. Damn near every single github project has issues that need fixed.
19:41:44<Sanqui>AND wayback machine happens to already be pretty good at it.
19:42:36sdf joins
19:44:44jut quits [Ping timeout: 240 seconds]
19:45:04<phuzion>AK: See my message to justsomeonewhoarchives above for some ways to help.
19:45:46jut (jut) joins
19:48:09<justsomeonewhoarchives>phuzion: damn, the url shortener landscape has completely gotten out of hand it seems
19:49:02<justsomeonewhoarchives>why even use them ...
19:49:33<justsomeonewhoarchives>its not as if the urls you get out of them are any easier to remember if they don't use custom names
19:56:36<purplebot>Current Projects edited by Wickedplayer494 (+0, It's finished) 1 minute ago -- https://www.archiveteam.org/?diff=46156&oldid=46131
20:03:38Webuser572 quits [Ping timeout: 244 seconds]
20:04:37Webuser572 joins
20:05:51<justsomeonewhoarchives>phuzion: hmm, URLTeam's scraper implementation seems suboptimal as it spends most time waiting for new items from the tracker which apparently hands them out quite slowly
20:06:05<phuzion>justsomeonewhoarchives: that's 100% by design.
20:06:14<phuzion>We're trying to not get the scrapers banned.
20:06:19<justsomeonewhoarchives>oh? due to not wanting to stress them too much?
20:06:21<justsomeonewhoarchives>I see
20:06:42<phuzion>And given that we've scraped 13.7 billion URLs, I'd say the strategy is working quite well.
20:06:57<AK>I'm presuming Dallas must have a tonne of ips/servers to be scrolling the list that quickly compared to everyone else?
20:07:17<AK>Thought I was doing well earlier, but I barely see my name anymore at all
20:07:36<justsomeonewhoarchives>makes sense I guess, slows the whole thing down for any single person though
20:08:21<Dallas>AK: 300 atm
20:08:47<justsomeonewhoarchives>I guess the basic idea is run many projects in parallel rather than any single one then
20:08:57<AK>Well, that's impressive
20:09:13<Dallas>justsomeonewhoarchives: it won't slow anyone else down if the queue has more todo
20:10:01<AK>I'm wondering about spinning up a container that runs 5 concurrent on 4/5 different projects, then I can just spin that up around the place instead of the slightly less easy github download method
20:10:19<justsomeonewhoarchives>Dallas: eh, just seeing basically no real load on this system or my connection, so I'd at least like to run something that can use the resources I have available
20:10:31<Dallas>Yeah do all the rojects
20:10:36<AK>justsomeonewhoarchives: reddit seems to be working pretty well for me at the moment
20:11:00<Dallas>URLTeam github reddit urls all need workers
20:11:10<Dallas>urls you can throw loads of resource at
20:11:29puperscuper joins
20:13:07<AK>Am I understanding right that urltime is url shorteners, and urls is different? (currently using grab-urls off github and it's also spending a lot of time "No item received."ing)
20:13:15<AK>*urlteam
20:15:07puperscuper quits [Remote host closed the connection]
20:15:20<Dallas>urlteam is shortners and urls is various interesting urls
20:15:50Holla joins
20:16:36Holla quits [Remote host closed the connection]
20:16:57<justsomeonewhoarchives>anyhow, got to run. be back later
20:17:02<yawkat>urlteam is spending plenty of time 404ing too :)
20:17:15justsomeonewhoarchives quits [Client Quit]
20:18:39Webuser206 joins
20:19:12lennier2 joins
20:19:22godane quits [Ping timeout: 252 seconds]
20:20:29godane (godane) joins
20:22:25lennier1 quits [Ping timeout: 265 seconds]
20:22:32lennier2 is now known as lennier1
20:27:37KronK quits [Remote host closed the connection]
20:27:42holbrooke (holbrooke) joins
20:27:50mgd quits [Remote host closed the connection]
20:33:25buffalo_man quits [Client Quit]
20:35:37sickit quits [Remote host closed the connection]
20:41:32godane1 joins
20:44:10godane quits [Ping timeout: 265 seconds]
20:46:31<Barto>are you guys topping the news? :p
20:46:45LeGoupil joins
20:49:22LeGoupil quits [Client Quit]
20:50:37<SketchTheCow>We are and always have been the news
20:50:45<SketchTheCow>What the hell else is going on
20:51:19<SketchTheCow>I'm only half-paying attention
20:51:22<SketchTheCow>I'm mostly cleaning up
20:51:31<SketchTheCow>Also getting the hate mail to archiveteam so you all don't
20:52:21Core7036 joins
20:53:48<thuban>Sanqui: speaking of http-only sites, you (or someone else voiced) mind throwing http://keithlynch.net/ into AB?
20:54:19<thuban>small but good site, http only, last updated about two decades ago, author still alive last i checked but getting on a bit.
20:54:24<Sanqui>thumbs up
20:54:30<thuban>thanks!
20:55:24dissentmemo quits [Ping timeout: 240 seconds]
20:57:10<TestCrap>That was some incredibly bad reporting on it, the fact I woke up to the reddit post, and tons of articles already with tons of misinformation just sucks. A lie travels around the world multiple times before the truth gets it's shoes on
21:01:09<atomicthumbs>i tweeted someone's actual explanation of what went on and it seems to be gaining some traction
21:01:40aphitex (aphitex) joins
21:02:16<maxfan8>Yay
21:06:36MM46 quits [Remote host closed the connection]
21:07:20Kenshin quits [Quit: ZNC - http://znc.in]
21:08:51PhantomTypist (phantomtypist) joins
21:10:03nc_eiu joins
21:10:10Webuser206 quits [Remote host closed the connection]
21:11:11<PhantomTypist>so, parler scrape aside, any one up for doing a HOPE conf session on Archive Team?
21:12:06crazysane (crazysane) joins
21:12:52Y48fyrCgHPR7d18W quits [Ping timeout: 244 seconds]
21:16:30<nc_eiu>Question: so the Archive Team plans on making Parler scrape available on Archive.org? I was able to get about 16k URLs on Archive.org before Parler went off/Archive.org automatically stopped me.
21:17:12<atphoenix>The speed of lies vs. truth has been studied. Lies spread faster. They're generally more 'interesting' and elicit stronger psychological responses. One reference: https://science.sciencemag.org/content/359/6380/1114.8.full
21:17:20osteo joins
21:18:07osteo quits [Remote host closed the connection]
21:18:09<PhantomTypist>nc_eiu yes, the warc's are going there. just the sheer amount of data will take time
21:20:20dissentmemo joins
21:22:18crazysane quits [Client Quit]
21:22:29ivan leaves
21:24:24Core7036 quits [Ping timeout: 240 seconds]
21:26:03<nc_eiu>Got it. I'm sure IA is going to be busy with this considering the data. I do hope there's a coordinated effort between the main team who assisted with the scrape and IA. IA understandably has a data cap (error 429) on their website which is frustrating for researchers who want that data or are using automated scraping programs.
21:30:24<mgrandi>i have work now, but i think we should make some readme about this data and how to access it, etc
21:31:43<mgrandi>nc_eiu: it will end up here: https://discord.com/channels/724855685904465932/724855686458245153/798296627667927070
21:33:55<mgrandi>can put it in the description of the collection ^ possibly
21:34:02<nc_eiu>I do not have access to that channel. Is it under construction or does it need an invite?
21:34:25<mgrandi>.... i copied it from discord and it copied the message link instead of the link i pasted lol
21:34:37<mgrandi>https://archive.org/details/archiveteam_neparlepas?tab=about
21:34:39<mgrandi>that ^
21:35:03Jerb joins
21:35:19Y48fyrCgHPR7d18W joins
21:36:35<nc_eiu>Good deal. Bookmarked. I'll keep an eye on that. This was a fantastic effort to watch unfold, even if I don't have the technical skills to really help with anything (unfortunately)
21:36:38<Webuser572>is there an archiveteam discord?
21:36:55<mgrandi>i think there is one for voice chat only
21:37:24<@arkiver>there will likely be a project for gab
21:37:46<Webuser572>mgrandi oh just the one you linked made me wonder what discord that was
21:37:52<nc_eiu>so much data, so little time
21:39:57dissentmemo quits [Read error: Connection reset by peer]
21:39:58dissentmemo (dissentmemo) joins
21:40:41<atphoenix>No Discord. Reasons: https://archiveteam.org/index.php?title=User:JustAnotherArchivist/Discord
21:42:04<atomicthumbs>woe upon the archivist who uses a system that cannot be archived
21:42:07<SketchTheCow>16:21 <tkellen> You seem like a total asshole, but I'm listening if you're talking.
21:42:16<SketchTheCow>New Archiveteam Slogan
21:42:25aphitex quits [Client Quit]
21:43:20<mgrandi>you can archive discord but its way harder than it should be
21:43:41<Webuser572>hmm might make sense to setup matterbridge and keep stuff on irc but mirror it to telegram/discord/etc
21:43:42aphitex joins
21:43:45<Sanqui>#discard
21:43:49<Sanqui>i'm working on it
21:43:56<Sanqui>and if we ever make an official bridge for anything it's going to be matrix
21:43:56dissentmemo quits [Read error: Connection reset by peer]
21:44:04aphitex quits [Client Quit]
21:44:07dissentmemo joins
21:44:32<Ajay>Hackint doesn't allow public matrix bridges
21:45:35<Sanqui>i don't think that's true
21:45:50<SketchTheCow>-!- #archiveteam-bs You're not a channel operator
21:45:51<SketchTheCow>ahahahaha
21:45:51<Sanqui>or rather, they probably don't allow the "puppeting" bridge
21:45:59<Sanqui>which is not really necessary imo
21:46:08Kenshin joins
21:46:10<Sanqui>nothing stops us from making a good old bot bridge
21:46:33<Webuser572>yea bot bridge is what i mean
21:46:44<Webuser572>https://github.com/42wim/matterbridge
21:46:54<Ajay>https://hackint.org/archive#20181028_Matrix_Bridging_Sunset
21:47:07<Webuser572>bridge between mattermost, IRC, gitter, xmpp, slack, discord, telegram, rocketchat, twitch, ssh-chat, zulip, whatsapp, keybase, matrix, microsoft teams, nextcloud, mumble and more with REST API (mattermost not required!)
21:47:11<Ajay>> Additionally we want to express clearly, that we disallow the bridging of Matrix in general, since some of these issues are universal to its bridging stack and at the same time intolerable.
21:47:36<Sanqui>Ajay: yes, they're talking about the puppeting stack
21:47:46<Sanqui>we don't need to use that one.
21:47:55<Webuser572>yeah bridging using the like server software, not like third parties just running bots as clients
21:48:14<Ajay>But #3 would rule out single bits right?
21:48:24<Ajay>*bots
21:48:32<Webuser572>matterbridge is nearly the best discord irc bridge, and the best discord irc bridge is merging into matterbridge
21:48:37<Sanqui>no, I'm sure there would be no problem with just a single bot
21:49:24<Sanqui>this is purely theoretical, we could always discuss the matter with the hackint admins, but 1) it's not what they're talking about at all and I'm sure they'd be fine with it, and 2) it's far off in the future anyway.
21:51:39<Webuser572>i'm looking to setup a discord for a new open source project to bridge their irc. could combine efforts on that
21:55:52megotperc quits [Client Quit]
21:56:10dissentmemo quits [Client Quit]
21:59:10<Sanqui>we don't need a discord
22:01:54<Webuser572>you'll get one and you'll LIKE IT
22:02:36<Billy549>And it wont be used
22:02:36<Billy549>:p
22:02:54<Billy549>I use Discord most of the time, it's my primary platform for most stuff (incl. VGPC) - IRC is great bc it's simple
22:03:18<Billy549>I have a cell phone from 2005 that can connect to IRC with no data cap [bc it's 2G, carrier doesn't do anything]
22:05:02<Webuser572>yeah i'm rly not a fan of discord's policies but it's so damn slick. loads all these chats up quite fast, with media support and all. i fully expect it to implode in the next 5-10 years. but as long as stuff is bridged it makes communities more accessible. more volunteer potential
22:06:01<thuban>"accessible" is a subjective term. we've had this conversation approximately one billion times
22:06:34<Billy549>I don't think that's a fair argument, I could load up a browser and speak here.
22:11:13<Webuser572>i'm doing that now but over the past two days my connections has dropped about five times. any backlog is gone
22:11:27<Webuser572>but that's true, talk is cheap. experiments are best
22:12:43<@hook54321>-> #-ot
22:12:58<Webuser572>anyway just thinking out loud, who would get angry if i setup a bridge and tried to promote it. not a huge fan of being a moderator but someone would have to
22:13:00<Webuser572>oh?
22:13:25<Webuser572>more like -meta
22:14:29Obelar_ leaves
22:15:07<maxfan8>I think the folks here perfer to communicate in IRC
22:17:07aphitex (aphitex) joins
22:17:40<thuban>i don't think anyone would be _angry_--there's a reason at describes itself as "a loose collective". but i don't think you'd get many takers--there's a reason at describes itself as "a loose collective" :p
22:20:00daalekz joins
22:20:28MeanieBeanie quits [Remote host closed the connection]
22:20:46<thuban>that said there's a place for discord and similar popular-albeit-questionable platforms (as JAA's page says) "possibly as a way for people to report shutdowns to us". there is e.g. a subreddit, which some of us keep an eye on and occasionally provide support in
22:21:04Y48fyrCgHPR7d18W quits [Ping timeout: 244 seconds]
22:23:08WH1sp3r quits [Ping timeout: 244 seconds]
22:25:21daalekz1 joins
22:25:28daalekz quits [Remote host closed the connection]
22:25:58<SketchTheCow>ops please
22:27:31@kiska sets mode: +o SketchTheCow
22:27:36<@SketchTheCow>Thank youuuuuuuuuuu
22:28:01<@SketchTheCow>Anything else needs my attention? I have been in a LOT of meetings today
22:28:08<@SketchTheCow>FOR SOME REASON
22:28:16<@kiska>Anywhere else I need to give you +o?
22:30:00<@SketchTheCow>Main #archiveteam
22:30:03<@SketchTheCow>That should be enough
22:34:58cereal_killer joins
22:35:44cereal_killer quits [Remote host closed the connection]
22:36:47billotronic (billotronic) joins
22:39:40nc_eiu quits [Ping timeout: 244 seconds]
22:50:33bluecmd quits [Remote host closed the connection]
22:52:49<@JAA>thuban: Believe me, I would be angry.
22:55:22<thuban>nvm Webuser572, JAA'd be angry. hth
22:56:11Y48fyrCgHPR7d18W joins
22:56:21Billy549 quits [Client Quit]
23:01:56<mgrandi>so I was uploading a thing to IA through the cli and it failed halfway probably cause the parler stuff, did it actually create the item?
23:02:33<mgrandi>i don't see it in the search but i dunno if its in a weird state or not
23:02:36Billy549 (Billy549) joins
23:04:38Twigz joins
23:04:52Twigz quits [Remote host closed the connection]
23:05:44TheJobd quits [Remote host closed the connection]
23:06:04vivid-noble quits [Remote host closed the connection]
23:06:11vivid-noble joins
23:06:57Craigle quits [Quit: Ping timeout (120 seconds)]
23:07:21Craigle (Craigle) joins
23:11:57kelsair joins
23:14:55<@JAA>mgrandi: If no file was completely uploaded, the item was never created.
23:15:23<@JAA>You don't upload directly to the item; it goes through an intermediate server and is then queued for copying to the data servers.
23:19:17<Webuser572>haha anger noted
23:20:31happypotato joins
23:22:21bee joins
23:22:54bee quits [Remote host closed the connection]
23:24:41<Ajay>the site is getting recursive now https://user-images.githubusercontent.com/12688112/104249859-40fb9d00-543a-11eb-89a0-8ece940b26c6.png
23:40:50<billotronic>A question for the ops: I still have a few links to dl's for parler packs. Would you prefer those were not given out til the collection is set back to public? I'm new here and don't want to go pissing people off.
23:41:38kelsair quits [Remote host closed the connection]
23:42:12<mgrandi>like packs of files that were not part of the DPoS project?
23:43:28<billotronic>no like the 'official' uploads to IA? archiveteam_parler_20210111163753_9b5a11be for example
23:43:40<billotronic>pardon if my lingo is not right
23:43:42<mgrandi>i mean if they are uploaded then anyone can find them if they search
23:43:57<mgrandi>the only thing IA is going to do is add them to the collection after they are done
23:46:10Y48fyrCgHPR7d18W94 joins
23:47:11kelsair joins
23:48:54Y48fyrCgHPR7d18W quits [Ping timeout: 244 seconds]
23:52:12godane (godane) joins
23:52:38<@JAA>mgrandi: They were noindex'd for the time being.
23:53:39sliccricc quits [Client Quit]
23:53:58<mgrandi>Ah ok, then you should probably wait then
23:54:14sliccricc joins
23:54:36godane1 quits [Ping timeout: 265 seconds]
23:58:16godane quits [Ping timeout: 252 seconds]
23:59:23godane (godane) joins

HomeSearchPrevious dayNext day