| 00:01:39 | | Ruthalas (Ruthalas) joins |
| 00:02:35 | <purplebot> | Elections edited by JustAnotherArchivist (-1, Fix Dutch general election entry) just now -- https://www.archiveteam.org/?diff=46107&oldid=45832 |
| 00:19:55 | | hooway quits [Client Quit] |
| 00:22:34 | <purplebot> | Elections/2021 Tweede Kamerver election moved by JustAnotherArchivist (+0) 20 minutes ago -- https://www.archiveteam.org/?diff=46105&oldid=0 |
| 00:28:43 | | Ruthalas quits [Client Quit] |
| 00:30:03 | | Arcorann_ joins |
| 00:34:43 | | Ruthalas (Ruthalas) joins |
| 01:00:56 | <OrIdow6> | Assuming we can run a headless browser, whose traffic is saved to WARC, at sufficient scale, how would playback look with the CF checker thing? |
| 01:01:54 | <OrIdow6> | Suspect you'd have to filter out records containing the check |
| 01:02:32 | <OrIdow6> | As I wouldn't think it lets you get a cookie once and then access 100000 pages |
| 01:03:06 | <@JAA> | It does, actually. Well, it depends how many you can get, but at least last time I did something with it, thousands of requests after passing the challenge were not problem on some sites. |
| 01:03:24 | <@JAA> | On other sites, the check was really weird and sometimes immediately broke again. |
| 01:04:02 | <@JAA> | As in, pass the challenge, get it again without any valid response between. |
| 01:06:13 | <OrIdow6> | Perhaps they're doing some unpredictable machine learning thing, or similar |
| 01:07:01 | <OrIdow6> | That's good news for anything that does use a headless browser, though, and if the useless page rate is one in thousands I think that's acceptable |
| 01:08:22 | <OrIdow6> | And I suppose that's good for the RECAPATCHA pages, too |
| 01:09:04 | <@JAA> | Well, somewhat, but you can't solve those automatically currently I think. |
| 01:09:53 | <OrIdow6> | No, but I think you can cheaply outsource them to a CAPATCHA-solving farm |
| 01:10:36 | <OrIdow6> | Though maybe they've figured out how to stop those, not something I follow closely |
| 01:11:29 | <@JAA> | I don't think it's impossible to stop those completely. |
| 01:11:36 | <@JAA> | it's possible* |
| 01:24:42 | | Arcorann_ quits [Remote host closed the connection] |
| 01:25:08 | | Arcorann_ joins |
| 01:41:44 | | HackMii quits [Ping timeout: 240 seconds] |
| 01:42:31 | | HackMii (hacktheplanet) joins |
| 02:00:01 | | Dallas quits [Client Quit] |
| 02:03:47 | | Dallas (Dallas) joins |
| 02:37:01 | <tech234a> | “Amazon Is Booting Parler Off Of Its Web Hosting Service” https://www.buzzfeednews.com/article/johnpaczkowski/amazon-parler-aws |
| 03:08:41 | <atphoenix> | new idea for those who want help AT but don't want to run a warrior or run docker: solve captchas for the warrior! |
| 03:15:16 | <mgrandi> | We can do that? |
| 03:15:37 | <mgrandi> | Is that for uhhh parler / thedonald.win? |
| 03:27:19 | <atphoenix> | not currently. It's just an idea. |
| 03:27:42 | <atphoenix> | the closest thing is to have a project use one of the existing anticaptcha services |
| 03:43:34 | <purplebot> | Parler edited by JustAnotherArchivist (+1032) just now -- https://www.archiveteam.org/?diff=46108&oldid=46095 |
| 03:46:34 | <purplebot> | Deathwatch edited by JustAnotherArchivist (+25, /* 2021 */ Add Parler) 1 minute ago -- https://www.archiveteam.org/?diff=46110&oldid=46072 |
| 03:47:34 | <purplebot> | Current Projects edited by JustAnotherArchivist (+169, Add Parler) just now -- https://www.archiveteam.org/?diff=46111&oldid=46096 |
| 03:53:35 | <purplebot> | File:Parler-logo.png uploaded by JustAnotherArchivist (+0, Converted from https://parler.com/512ae92f/images/branding/logo.svg …) just now -- https://www.archiveteam.org/?diff=46112&oldid=0 |
| 03:54:35 | <purplebot> | File:Parler - 1-9-21.png uploaded by Wickedplayer494 (+0) just now -- https://www.archiveteam.org/?diff=46114&oldid=0 |
| 04:01:10 | <mgrandi> | I mean I'd be willing to do it if it wasn't literally every request, I dunno how many requests you get before another captcha |
| 04:07:19 | <@JAA> | Traditional captchas are pretty easy to do that way, just host the captcha image somewhere temporarily, display it to the human that is to solve it, send response back to the worker. reCAPTCHA is trickier with those shitty 'select all bicycles that are ridden by a woman under 30 wearing, have a red-green-striped backpack on the rack, and have exactly 76 spokes'. |
| 04:08:33 | <flashfire42> | JAA you mind if I post that to reddit and a facebook group so the 6 brain cells I have left make the happy chemical? |
| 04:10:23 | <@JAA> | flashfire42: Go ahead. (Remove the 'wearing' typo though.) :-) |
| 04:11:27 | <flashfire42> | Oh its done as a screenshot. Oh well its a facebook group not like they will notice |
| 04:11:38 | <mgrandi> | so are we attempting to scrape thedonald.win at the moment? or are we still struck on CF |
| 04:16:35 | <purplebot> | Parler edited by JustAnotherArchivist (+33), JustAnotherArchivist (+25, Add logo), Wickedplayer494 (+30, Image of its HTTP presence) 21 minutes ago -- https://www.archiveteam.org/?diff=46115&oldid=46108 |
| 04:17:06 | <wickedplayer494> | and reCAPTCHA is utter dog shit in that even if you follow Google's prompts to a tee, it might still say "fuck you, do it again" as punishment just because you dared to have javascript disabled |
| 04:19:39 | <@JAA> | Also, if you click too quickly, you're rewarded with more images. |
| 04:19:57 | <atphoenix> | we learned that well on the Yahoo groups project |
| 04:23:24 | <OrIdow6> | As discussed, there are RECapatha-solving services (largely for use of spammers) that are fairly effective |
| 04:23:30 | <OrIdow6> | That farm them out to humans |
| 04:24:18 | <@JAA> | Yes, and we were discussing building our own so random people could help us and we wouldn't have to pay for those services. |
| 04:24:41 | <atphoenix> | they're not free services, and people do ask 'how can we help', thus the idea here |
| 04:24:59 | <@JAA> | Does any of those services have an open-source code base? |
| 04:25:28 | <OrIdow6> | Doubt their whole codebases are open source, but their client libs might be |
| 04:25:35 | | qw3rty_ joins |
| 04:25:45 | <atphoenix> | at least one as an API and browser plugin that we used for YG. lennier1 did you write some code for that? |
| 04:27:59 | <atphoenix> | maybe these are relevant? https://github.com/dessant/buster https://sourceforge.net/projects/opensourcecaptchasolver/ |
| 04:29:26 | | qw3rty__ quits [Ping timeout: 265 seconds] |
| 04:30:19 | <atphoenix> | this was one of the tools for YG: https://anti-captcha.com/clients/help/apilinks |
| 04:31:09 | <atphoenix> | if that doesn't open, perhaps an account is required first. Otherwise this should be viewable by all: https://anti-captcha.com/faq |
| 04:33:00 | <@JAA> | I think buster works very well except it runs into rate limits all the time because too many people are using it. It uses some speech-to-text services with API keys to solve the audio challenge. |
| 04:39:42 | <atphoenix> | Buster may be rate limited by IP. It seemed to work okay with sparse usage. I don't know if there is another rate limit beyond per-IP. |
| 04:47:41 | <maxfan8> | JAA: I'll be trying to archive as much of the interesting/political parler stuff as I can |
| 04:52:35 | <purplebot> | Parler edited by Phuzion (+113) 21 minutes ago -- https://www.archiveteam.org/?diff=46116&oldid=46115 |
| 05:00:01 | | treora quits [Quit: blub blub.] |
| 05:01:19 | | treora joins |
| 05:42:15 | <brad> | So, would I be considered unwelcome, if I were to vote against archiving parler, because I’m tired of hearing their dogshit lies that they constantly spew? Is it possible that their bilious sewage can avoid being archived for all eternity? |
| 05:50:19 | <maxfan8> | The goal of archiving (as I see it) is to be able to provide a historical record that future people can cite/look back on and analyze |
| 05:50:22 | | kiiwiiwastaken joins |
| 05:50:26 | <kiiwiiwastaken> | hello |
| 05:50:55 | | kiiwiiwastaken is now authenticated as kiiwiiwastaken |
| 05:51:23 | <maxfan8> | brad: I personally disagree and believe that Parler is worthy of archival -- and is probably more worthy for archival than many other things (since it will be taken down and lost to history very soon) |
| 06:04:19 | | kiiwiiwastaken quits [Remote host closed the connection] |
| 06:34:00 | <atphoenix> | IMO, archiving web stuff is something akin to taxidermy. An interactive site is usually not preserved as a 'live working site'...as full interaction would require all the databases and scripts we don't have direct access to, nor do we have a way to put those into WBM. Instead, what we get is something that shows how things looked, and maybe shows how certain parts worked, at the time of the archival. Site features like |
| 06:34:01 | <atphoenix> | submitting new posts and submitting custom search queries won't work on an archived site. |
| 06:36:36 | <atphoenix> | I personally find more value in archiving technical and gaming forums, but I am sure historians and political scientists will find social media archives of great interest. |
| 07:02:12 | | godane (godane) joins |
| 07:53:31 | | mutantmonkey quits [Remote host closed the connection] |
| 07:53:58 | | mutantmonkey (mutantmonkey) joins |
| 08:56:55 | | Doran is now known as Doranwen |
| 08:56:57 | | Doranwen is now authenticated as Doranwen |
| 08:57:20 | | hooway joins |
| 10:20:59 | | Arcorann (Arcorann) joins |
| 10:23:59 | | Arcorann_ quits [Ping timeout: 244 seconds] |
| 11:16:47 | | jut (jut) joins |
| 11:39:23 | | jut quits [Client Quit] |
| 11:39:44 | | jut (jut) joins |
| 12:16:44 | | Barto quits [Remote host closed the connection] |
| 12:17:08 | | Barto (Barto) joins |
| 12:21:13 | <EggplantN_> | Do we have a parler channel |
| 12:21:23 | <@arkiver> | #neparlepas |
| 14:40:38 | | ragu is now authenticated as ragu |
| 15:34:59 | | Arcorann quits [Ping timeout: 265 seconds] |
| 16:16:35 | <purplebot> | Template:Czech websites edited by Sanqui (+151) just now -- https://www.archiveteam.org/?diff=46117&oldid=43764 |
| 16:20:34 | <purplebot> | Pikachu.cz created by Sanqui (+803, Created page with "{{Infobox project …) just now -- https://www.archiveteam.org/?diff=46118&oldid=0 |
| 16:26:16 | | etnguyen03 (etnguyen03) joins |
| 16:45:34 | <purplebot> | Template:Czech websites edited by Sanqui (+21), Sanqui (+17) 22 minutes ago -- https://www.archiveteam.org/?diff=46120&oldid=46117 |
| 16:51:06 | | HP_Archivist (HP_Archivist) joins |
| 16:55:34 | <purplebot> | Pikachu.cz edited by Sanqui (+296) 22 minutes ago -- https://www.archiveteam.org/?diff=46121&oldid=46118 |
| 17:13:42 | <@JAA> | atphoenix: Last time I tried it, there were issues of it being rate-limited at the API key level on those speech-to-text APIs. There are a few issues about that on GitHub as well, e.g. https://github.com/dessant/buster/issues/217 . You can fix it by supplying your own API keys though. |
| 17:17:10 | | kiiwiiwastaken (kiiwiiwastaken) joins |
| 17:22:29 | | kiiwiiwastaken quits [Ping timeout: 244 seconds] |
| 17:23:51 | | kiiwiiwastaken (kiiwiiwastaken) joins |
| 17:28:55 | | VerifiedJ quits [Quit: The Lounge - https://thelounge.chat] |
| 17:29:24 | <kiiwiiwastaken> | Hi |
| 17:30:12 | <kiiwiiwastaken> | I'm archiving some websites as warcs as described on the archiveteam site and I was curious how many times should I back up again? every day/week/month? |
| 17:30:40 | | VerifiedJ (VerifiedJ) joins |
| 17:50:34 | <purplebot> | Parler edited by Wessel1512 (+56) just now -- https://www.archiveteam.org/?diff=46122&oldid=46116 |
| 17:53:14 | <wessel1512> | kiiwiiwastaken backup what |
| 17:53:37 | <kiiwiiwastaken> | just some websites of linux distros |
| 17:53:47 | <kiiwiiwastaken> | I'm using this to download them https://archiveteam.org/index.php?title=Wget |
| 17:54:47 | <wessel1512> | which sites are we talking about |
| 17:55:47 | <kiiwiiwastaken> | linuxmint.com ubuntu.com and getsol.us |
| 18:01:25 | <wessel1512> | for backing up best is to do it every day if you can |
| 18:01:35 | <kiiwiiwastaken> | alrighty |
| 18:02:16 | <@JAA> | Basically, it depends on how often the site changes, how many resources you have/are willing to dedicate to this, how much you're willing to lose in the worst case, etc. |
| 18:04:53 | <kiiwiiwastaken> | Well I did some math and if the Linux Mint site stays at the same size it'd take up ~100gb if I back up everyday for a whole year |
| 18:05:29 | <kiiwiiwastaken> | But that site is 441mb whereas some like manjaro and mxlinux are around 30mb |
| 18:06:20 | <@JAA> | I guess you're not retrieving the installation media then. |
| 18:07:28 | <@JAA> | But yeah, run a crawl that retrieves everything you want, then do the maths based on that (including some buffer), then decide how often is reasonable. :-) |
| 18:08:33 | <@JAA> | A more elaborate setup could deduplicate data between the crawls, but I'm not aware of an easy solution for that. |
| 18:09:04 | | DogsRNice (Webuser299) joins |
| 18:12:53 | <kiiwiiwastaken> | Like a delta backup? lol |
| 18:13:49 | <@JAA> | Sort of, storing references (revisit records) for duplicates instead of another full copy. |
| 18:14:02 | <kiiwiiwastaken> | Ah |
| 18:14:20 | <kiiwiiwastaken> | Meh it's fine tbh since the command is very hands off |
| 18:14:39 | <@JAA> | Yeah, if it's this small, I wouldn't bother either. |
| 18:14:51 | <kiiwiiwastaken> | Put the commands in a shell script so all i have to do is run that whenever, it automatically adds the date to the end :> |
| 18:15:34 | <purplebot> | Parler edited by Wessel1512 (+8) 24 minutes ago -- https://www.archiveteam.org/?diff=46123&oldid=46122 |
| 18:15:36 | <@JAA> | I suppose you could play around with --warc-cdx and --warc-dedup though if you want. No idea whether it works, never tried it. |
| 18:16:09 | <kiiwiiwastaken> | Oh the wiki had the argument --warc-cdx and I'm using it |
| 18:16:14 | <kiiwiiwastaken> | Don't know what it does though |
| 18:17:07 | <@JAA> | Also not sure if it writes revisit records. The man page says about --warc-dedup: 'Do not store records listed in this CDX file'. Implies that they're dropped entirely. But maybe that's just inaccurate documentation. |
| 18:17:24 | <kiiwiiwastaken> | ah |
| 18:17:40 | | Jonboy3451 joins |
| 18:17:41 | <@JAA> | --warc-cdx produces an index of the WARC file. |
| 18:18:22 | <@JAA> | Useful for accessing the content later and enables things like --warc-dedup. |
| 18:18:30 | <kiiwiiwastaken> | Oh like all the links in it? |
| 18:19:00 | <@JAA> | Each URL retrieved along with various metadata like datetime of access, status code, content hash, and whatnot. |
| 18:19:17 | <kiiwiiwastaken> | ah cool |
| 18:19:27 | <kiiwiiwastaken> | I'll just keep the arguments as-is |
| 18:21:04 | | jonboy3452 quits [Ping timeout: 240 seconds] |
| 18:54:45 | | sec^nd quits [Remote host closed the connection] |
| 18:55:11 | | sec^nd (second) joins |
| 19:06:16 | | lukash7 quits [Client Quit] |
| 19:13:31 | | lukash7 joins |
| 19:16:17 | | spirit quits [Client Quit] |
| 19:27:22 | | Xypter joins |
| 19:27:23 | | kiiwiiwastaken quits [Remote host closed the connection] |
| 19:28:33 | | Xypter quits [Remote host closed the connection] |
| 19:43:03 | | Ryz quits [Remote host closed the connection] |
| 19:43:46 | | Ryz (Ryz) joins |
| 20:05:34 | <purplebot> | Parler edited by JustAnotherArchivist (+195, Add new statement on migration) 2 minutes ago -- https://www.archiveteam.org/?diff=46124&oldid=46123 |
| 20:17:46 | | HP_Archivist quits [Client Quit] |
| 20:20:51 | | LeGoupil joins |
| 20:21:16 | | LeGoupil quits [Client Quit] |
| 20:24:14 | | godane quits [Client Quit] |
| 21:41:12 | | godane (godane) joins |
| 21:46:25 | | Flutters joins |
| 21:48:31 | | Flutters quits [Remote host closed the connection] |
| 21:49:35 | <purplebot> | Current Projects edited by Wickedplayer494 (+0, Wheeeelp, the Parler panicgrab …) just now -- https://www.archiveteam.org/?diff=46125&oldid=46111 |
| 21:54:26 | | ave (ave) joins |
| 22:00:09 | | kelsair joins |
| 22:01:40 | | nepeat joins |
| 22:02:04 | | nepeat is now authenticated as nepeat |
| 22:14:27 | | PokeyHokie joins |
| 22:15:04 | | PokeyHokie quits [Remote host closed the connection] |
| 22:19:52 | | PokeyHokie joins |
| 22:20:02 | | kelsair quits [Remote host closed the connection] |
| 22:28:50 | | PokeyHokie quits [Remote host closed the connection] |
| 22:30:35 | <purplebot> | List of websites excluded from the Wayback Machine edited by Rexma (+19, added nyaa.si, a torrenting site) 1 minute ago -- https://www.archiveteam.org/?diff=46126&oldid=45881 |
| 22:42:34 | <purplebot> | Template:Navigation box edited by Wickedplayer494 (+395, Assorted additions) just now -- https://www.archiveteam.org/?diff=46127&oldid=44378 |
| 22:44:35 | <purplebot> | File:Tumblr - 1-10-21.png uploaded by Wickedplayer494 (+0) just now -- https://www.archiveteam.org/?diff=46128&oldid=0 |
| 22:44:35 | <purplebot> | Tumblr edited by Wickedplayer494 (+31, + an image) just now -- https://www.archiveteam.org/?diff=46129&oldid=44143 |
| 22:46:26 | | Jonboy3451 quits [Remote host closed the connection] |
| 22:55:47 | <lennier1> | atphoenix: Yes, my YG project used a Chrome extension that worked automatically if you also set up a separate captcha solving extension. I used anti-captch.com, which cost $2 per 1000 captchas solved and worked well. https://github.com/lennier1/yahoogroups-auto-list-joiner |
| 22:56:51 | <lennier1> | I believe there's also an API if you don't want to use the extension, but for my purposes the extension was easier to implement. |
| 23:08:34 | <purplebot> | File:Periscope - 1-4-21.png uploaded by Wickedplayer494 (+0) just now -- https://www.archiveteam.org/?diff=46132&oldid=0 |
| 23:08:34 | <purplebot> | File:Periscope logo.png uploaded by Wickedplayer494 (+0) just now -- https://www.archiveteam.org/?diff=46133&oldid=0 |
| 23:12:35 | <purplebot> | Current Projects edited by Wickedplayer494 (+56, Add Periscope to upcoming, take …) 9 minutes ago -- https://www.archiveteam.org/?diff=46131&oldid=46125 |
| 23:16:22 | | igloo22225 (igloo22225) joins |
| 23:20:34 | <purplebot> | Periscope created by Wickedplayer494 (+4501, Start of this page) just now -- https://www.archiveteam.org/?diff=46135&oldid=0 |
| 23:23:35 | <purplebot> | Vine edited by Wickedplayer494 (+4, Surprisingly nobody ever set this …) just now -- https://www.archiveteam.org/?diff=46136&oldid=26792 |
| 23:32:35 | <purplebot> | Template:Navigation box edited by Wickedplayer494 (+20, Can't forget Periscope since I'm …) 18 minutes ago -- https://www.archiveteam.org/?diff=46134&oldid=46127 |
| 23:55:55 | | Arcorann (Arcorann) joins |
| 23:57:10 | | pew quits [Ping timeout: 265 seconds] |