| 00:14:56 | | lunik1 quits [Client Quit] |
| 00:19:36 | | lunik1 joins |
| 00:21:36 | | lunik1 quits [Client Quit] |
| 00:22:01 | | lunik1 joins |
| 00:25:13 | | lunik1 quits [Client Quit] |
| 00:25:30 | | lunik1 joins |
| 01:06:56 | <lennier1> | Visa and Mastercard have suspended payment processing for Pornhub. https://www.theverge.com/2020/12/10/22168240/mastercard-ending-pornhub-payment-processing-unlawful-videos |
| 01:08:02 | <lennier1> | PH already announced policy changes like ending anonymous uploading. https://www.foxnews.com/us/pornhub-rules-sex-trafficking-backlash |
| 01:09:38 | <lennier1> | It's not clear to me that users uploading illegal content is any worse than on more mainstream sites like Twitter, but easier to demonize an adult site if I may editorialize. |
| 01:10:58 | <lennier1> | I'm curious what they'll do about all the content already uploaded by users they can't identify. Most of it is presumably fine, but how can you prove it? |
| 01:12:26 | <@JAA> | I'm surprised this didn't happen years ago. It's not like these problems haven't been publicly known for years. |
| 01:29:14 | <OrIdow6> | Do you have to pay to use PornHub, or does it get its money from ads? |
| 01:31:08 | <lennier1> | It's mostly free and ad-supported. You can get a premium subscription with higher resolution videos and some premium-only videos. Users can also have paid only videos or accept tips. |
| 01:33:29 | <flashfire42> | The Problem is that there are far to many stories of pornhub refusing to take down content unless you get a lawyer involved plus for a platform that claims to check every video for illegal content (Which would be impossible in itself) there is a lot of illegal content there |
| 01:33:50 | <Ryz> | I'm not sure of the viability of archiving Pornhub since a ton of it is videos; like if WBM/IA can't handle hosting the content overall, then it may be unlikely it will handle the flood of YouTube content if what Jason said will happen in the future |
| 01:34:25 | <flashfire42> | I am against archiving pornhub for the simple reason that there are too many non consensual videos up there and I dont wanna be the one that helps accidentally push CSEM into the wayback machine |
| 01:34:47 | <flashfire42> | if we do a few captures of the front page maybe the categories page to show how many videos were up |
| 01:35:14 | <flashfire42> | It gains money from ads but it puts ads in front of videos of very questionable origin |
| 01:36:06 | <Ryz> | I'm talking from the assumption there was no vast amounts of illegal material (compared to I guess say YouTube or Dailymotion), which I did not realize Pornhub harbors that kind and amount of content oo; |
| 01:36:30 | <Ryz> | ...There's still countless porn websites though |
| 01:36:37 | <flashfire42> | Yeah Youtube and dailymotion do have a problem with illegal content but not nearly on the scale of Pornhub. |
| 01:37:13 | <@JAA> | Also, on other platforms it's usually copyright infringement. Here, it's rape videos and CP. |
| 01:37:15 | <flashfire42> | Mindgeek and WGCZ Holding are the 2 biggest porn companies in the world and both of them dont give a shit about CSEM on their platform if they can make money off of it |
| 01:37:18 | <lennier1> | I mean, literally any website that allows uploading of adult material without strict verification is going to have some illegal material. I'm sure that was true of Tumblr, is true of Twitter, etc. |
| 01:37:42 | <@JAA> | Yep |
| 01:37:51 | <Ryz> | Huh, what do they mention by 'anonymous uploading'? Like you don't have to make an account and just upload it? |
| 01:38:05 | <@JAA> | The difference is that, from what I've heard, Twitter et al. remove that content if reported. PornHub doesn't. |
| 01:38:16 | <flashfire42> | Twitter needs it pointed out to them to have it taken down which is a fucking annoyance but they usually take it down when reported. Tumblr was a nightmare from the start let us be honest here. But pornhub doesnt report it unless a lawyer gets involved |
| 01:38:32 | <lennier1> | You need an account, but they don't verify your real name (i.e. by requiring a credit card or driver's license). |
| 01:38:34 | <flashfire42> | Take the case of Rose Kalemba who had to impersonate a lawyer to get her assault videos taken down |
| 01:39:15 | <lennier1> | I'm not sure if they've even finalized the new policies, but I think the idea if if someone uploads illegal material, they'll know who did it. |
| 01:39:36 | <flashfire42> | also AFAIK twitter doesnt claim to screen every tweet before its posted Pornhub has claimed on multiple occassions they check every video and use fingerprinting despite the fact that illegal content that is the same video has been uploaded multiple times there |
| 01:40:14 | <@JAA> | Let's split off the 'why PornHub sucks' discussion into -ot and deal with any potential archival only here. |
| 01:40:17 | <flashfire42> | sorry |
| 01:40:48 | <@JAA> | No worries, some context is fine here. |
| 01:41:40 | <flashfire42> | Taking aside that Pornhub is a shitty company that hosts non consensual content the size of it is not viable for an archive when we consider that a lot of porn content doesnt often have a lot of variety |
| 01:42:17 | <flashfire42> | If we were to grab maybe the top 10 videos in each category that might be viabkle |
| 01:42:37 | <flashfire42> | But I doubt IA has or wants to provide that level of bandwidth to pornography |
| 01:43:33 | <lennier1> | Is PH (potentially) removing content that different from Tumblr removing adult content? |
| 01:43:51 | <flashfire42> | In a perfect world I would love to archive the internet but we need to pick our battles and I dont think Pornhub is in the scope of archiveteam. Hell there was a debate to eroshare |
| 01:44:09 | <@JAA> | Tumblr was mostly hosting shitty GIFs, PornHub is hosting high-resolution videos. So there's a massive size difference. |
| 01:44:57 | <flashfire42> | Not to mention the fact that a fair amount of tumblr was also text posts. and small images. Pornhub doesnt have a functioning forum or large text based area |
| 01:45:58 | <lennier1> | Tumblr didn't remove text posts, only images they auto-IDd as nudity. I forget if Tumblr had videos. |
| 01:46:25 | <Ryz> | Tumblr does have videos uploaded, and also audio too |
| 01:46:25 | <flashfire42> | Not like xnxx which has sexstories(dot)com which could be a viable thing to save if it comes to it. We are talking lots of videos some of which are of assaults and I dont know about you but I dont wanna be the one to contribute to someone suing IA because their rape is part of history. Of course that would be an edge case but I am Just putting out |
| 01:46:25 | <flashfire42> | I am against this archival for the first time in my life I dont wanna see something gobbled up by IA |
| 01:46:36 | <@JAA> | Well, if they did, it wasn't a very large part I think. It was mostly known for its poor-quality GIFs. |
| 01:46:49 | <lennier1> | But I'll grant there could be a big data size difference, no real ideas of actual numbers. |
| 01:47:43 | <flashfire42> | I would wager there may even be up to a petabyte of porn from pornhub. Not to mention this will most likely affect MindGeeks other sites when people catch on that 2 companies hold a monopoly on the porn tube sites |
| 01:47:57 | <lennier1> | PH does have a lot of images as well. Plus blog-type posts and comments, though obviously that's not the site's primary draw. |
| 01:48:28 | <flashfire42> | Anyway I have said my bit I may say more but I think I have made my stance clear 😓 |
| 01:51:53 | <Ryz> | How many websites does Mindgeek have? I'd imagine there'd be a pornado sprouting out from that hell of a mess |
| 01:54:50 | <@JAA> | RedTube, YouPorn, SpankWire, etc. Dozens of sites like this. They also own various porn production companies. |
| 02:00:02 | | Dallas quits [Client Quit] |
| 02:00:15 | <lennier1> | They also own Xtube, which doesn't seem to be as popular anymore, but has lots of older user-generated content. |
| 02:03:45 | | Dallas (Dallas) joins |
| 02:06:13 | <VerifiedJ> | On a unrelated note, I've finished discovery for archiviolastampa.it and have started downloading. 63GB so far. Should be done in a day or two. |
| 02:07:50 | <jodizzle> | VerifiedJ: Excellent! I was worried about that one. |
| 02:08:00 | <jodizzle> | Are you planning on uploading to IA when done? |
| 02:08:28 | <VerifiedJ> | yes that's the plan. |
| 02:09:01 | <@JAA> | Nice |
| 02:09:49 | <jodizzle> | Would there be any value in processing a list of asset URLs in archivebot as well? |
| 02:14:07 | <OrIdow6> | Looks like it's a privately-held company, which makes it harder to tell how this will affect them |
| 02:14:57 | <@JAA> | VerifiedJ: WARCs or plain files? |
| 02:17:13 | <VerifiedJ> | plain files. I'm extracting the images and creating _images.zip. |
| 02:19:38 | <VerifiedJ> | All the important content is loaded via flash with custom session ids and cookies. |
| 02:21:42 | <@JAA> | Right, so having it in the WBM (via AB) is probably of little value. |
| 02:21:54 | <@JAA> | Unless IA adds some site-specific session ID removal. |
| 02:44:22 | <lennier1> | To the extent that video archiving is a question of storage size, it could be cool to use fingerprinting to not save duplicate videos. There's definitely a mix of people uploading self-produced videos and people uploading videos they saved elsewhere. |
| 02:57:18 | <OrIdow6> | Yes, could work |
| 02:57:58 | <@arkiver> | so |
| 02:58:02 | <@arkiver> | pornhub going away? |
| 02:58:03 | <@arkiver> | or? |
| 02:58:04 | <OrIdow6> | Though of course video is subject to generation loss just as much as video |
| 02:58:11 | <@arkiver> | what is this all about |
| 02:58:27 | <OrIdow6> | And that's more of ahypothetical; don't think IA has enough money to host all video ever put online :) |
| 02:58:43 | <@arkiver> | no we dont |
| 02:58:46 | <@arkiver> | selected video yes |
| 02:58:48 | <@arkiver> | low res maybe |
| 02:59:04 | <OrIdow6> | arkiver: No, they just may be losing payments from sources |
| 02:59:25 | <OrIdow6> | Which means long-term threat, if consumers paying them directly, as opposed to advertisments, are a major source of their money |
| 02:59:49 | <@arkiver> | right |
| 03:00:00 | <OrIdow6> | *from some sources |
| 03:00:33 | <@JAA> | And potential loss of very problematic content if they finally take their role seriously and take down rape videos etc. |
| 03:00:54 | <@JAA> | Nothing we'd want to archive except possibly metadata. |
| 03:03:44 | <OrIdow6> | Unless IA wants it for whatever reason |
| 03:07:17 | <lennier1> | They've announced changes including requiring verification to who can upload videos moving forward. Then the payment processing decision came soon afterwards, so apparently Visa and Mastercard felt their changes so far were inadequate. Anything beyond that is speculation, but I do think there's some risk of mass removal of videos with no obvious illegal content, but of unknown provenance. |
| 03:09:25 | <lennier1> | Not sure what percentage of their revenue comes from credit card payments vs. ads, but obviously there's potential risk to their whole business as well. |
| 03:10:42 | <OrIdow6> | On fingerprinting for deduplication of large sets of video: one thing I can see working, given natural and artificial "generation loss", is doing it on the level of segments of the video, vs. the whole thing |
| 04:54:20 | | qw3rty_ joins |
| 04:57:44 | | qw3rty quits [Ping timeout: 240 seconds] |
| 05:00:01 | | treora quits [Quit: blub blub.] |
| 05:01:16 | | treora joins |
| 05:13:19 | | Stilett0 quits [Remote host closed the connection] |
| 05:50:37 | | rsn_ joins |
| 05:50:37 | | rsn_ quits [Remote host closed the connection] |
| 06:09:45 | <OrIdow6> | Running a very crude script that OCRs the country code from the .eu registry, if one exists |
| 06:10:01 | <OrIdow6> | Fairly strict rate limit |
| 06:16:16 | | Craigle quits [Ping timeout: 244 seconds] |
| 06:17:34 | | Stiletto joins |
| 06:20:14 | <OrIdow6> | Now checking the language of the registrant if country is not present |
| 06:32:32 | <thuban> | what's the rate limit? |
| 06:33:38 | <OrIdow6> | Don't know |
| 06:33:48 | <thuban> | (& is this just for testing, or can we not take an ocr dependency in the warrior?) |
| 06:33:53 | <OrIdow6> | Strict rate-limiting, I should say |
| 06:34:15 | <thuban> | ? |
| 06:34:29 | <OrIdow6> | Somewhere in 1-10 seconds, but once it limits you it takes a lot of exponential backoff before it lets you back |
| 06:34:42 | <thuban> | ah |
| 06:34:43 | <OrIdow6> | Safe, no-429 rate limit is somewhere in that range |
| 06:35:20 | <OrIdow6> | I'd rather not have Tesseract on the warrior - presumably there are a finite number of country code PNGs, so we can look for those, maybe |
| 06:35:32 | <thuban> | mm, good thought |
| 06:36:27 | <OrIdow6> | Though curiously, I'm not seeing "UK" in any country code, and hardly any no-address domains except for those with a language listed as English |
| 06:37:30 | <thuban> | "GB" is the iso 3166-1 for the uk. don't ask me |
| 06:37:53 | <Ryz> | !ig 4efdbcgenejtrjbusqs03fug9 ^https?://paydayloansbits\.com/ |
| 06:37:55 | <Ryz> | Oops |
| 06:38:15 | <OrIdow6> | Oh, I see one for GB of ~30 |
| 06:38:34 | <OrIdow6> | Now it's 429ing me, 15 seconds was too fast, apparently |
| 06:38:36 | <thuban> | i checked two i turned up at random and they're bit-identical |
| 06:38:42 | <OrIdow6> | Good |
| 06:38:52 | <thuban> | (https://whois.eurid.eu/en/search/?domain=eldridgehouse.eu, https://whois.eurid.eu/en/search/?domain=coleridge.eu) |
| 06:39:30 | | Craigle (Craigle) joins |
| 06:40:38 | <thuban> | what about fallbacks, then? english-language isn't very specific; emails are better (.co.uk, .uk.com, etc) but bring back the ocr question |
| 06:41:36 | <OrIdow6> | Seems like most of the emails I've looked at are Gmail, Hotmail, etc. |
| 06:42:37 | <thuban> | do we want to have one warrior round for discovery with web-whois and another for actual archiving? |
| 06:45:03 | <OrIdow6> | Just looking at the results I have so far, fewer than 1/5 domains are from the UK, and presumably even fewer of those are going to be something other than a parking page etc. |
| 06:45:47 | <OrIdow6> | So that could get the warriors blocked, considering those tend to have a smaller number of IPs, than would be possible for a dedicated discovery project |
| 06:46:11 | <OrIdow6> | But I suppose they could just have a delay |
| 06:46:55 | <thuban> | what's the dedicated-discovery setup look like |
| 06:51:34 | <OrIdow6> | So I don't think there's been a separate-discovery warrior project in the time I've been here, but looking at what's happened in the past, it's like a normal one, except wget-lua (or for some, a specialized Python script) writes to a text file instead of warcs |
| 06:51:50 | <OrIdow6> | However, I presume backfeed might be used now |
| 06:52:52 | <thuban> | oh--i thought you meant that there was some alternative arrangement with a larger number of ips that we've used/could use |
| 06:55:32 | <OrIdow6> | I get the general impression that it is easier to run a small number of high-bandwidth workers, or a bunch of many ips, than do both |
| 06:55:42 | <OrIdow6> | But maybe that's not the case |
| 06:56:27 | <OrIdow6> | So I suppose this is really a question for the subset of AT that does the running |
| 06:56:38 | <thuban> | aye |
| 07:46:41 | | Craigle quits [Ping timeout: 244 seconds] |
| 07:59:27 | | Craigle (Craigle) joins |
| 08:12:44 | <tech234a> | Orldow6: I don’t think you need to do OCR for each image... it seems that each country code is it’s own base64 encoded image separate from the rest of the address and consistent for each country code and you could probably just collect a list of domains with each base64 country code and then just look at the final list |
| 08:13:03 | <tech234a> | Are you scraping the similar domains list also? |
| 08:13:15 | <OrIdow6> | <OrIdow6> I'd rather not have Tesseract on the warrior - presumably there are a finite number of country code PNGs, so we can look for those, maybe |
| 08:13:51 | <OrIdow6> | <thuban> i checked two i turned up at random and they're bit-identical |
| 08:14:08 | <tech234a> | Whoops missed that but yeah that’s what I also found |
| 08:15:35 | <Ryz> | !concurrency aod8v7tn4csv9ys3xam351jg4 1 |
| 08:15:48 | <Ryz> | Oops |
| 08:16:20 | <OrIdow6> | No, I am not; the point of this scrape is just to gather statistics on how many domains are in the UK |
| 08:17:54 | <tech234a> | Yeah I thought that might help with finding more domains but it probably adds complexity |
| 08:18:34 | <OrIdow6> | Speaking of that, looks like .ευ and .ею might be having their registrations revoked, too |
| 08:18:46 | <OrIdow6> | "The same registration requirements as for .EU and .ею apply, that is to say that a presence in the European Union, Iceland, Liechtenstein or Norway is required, or EU citizenship." - https://www.lexology.com/library/detail.aspx?g=bc0b9ac7-1284-4884-91d9-ffabf39d9724 |
| 08:24:02 | <tech234a> | Their WHOIS policy actually seems kind of reasonable, perhaps we could request a disclosure instead of scraping?: https://eurid.eu/d/205797/whois_policy_en.pdf |
| 08:24:32 | <OrIdow6> | Appears to be one domain each in the WBM |
| 08:26:34 | <thuban> | ^ hmm, maybe |
| 08:26:56 | <OrIdow6> | Looks to me like that process pertains to "thedisclosureofpersonaldatanotpublishedintheweb-basedWHOISbutprocessedbytheRegistry" (add the spaces yourself) |
| 08:28:00 | <OrIdow6> | Doesn't explain why some of them don't have addresses listed, considering the registry supposedly "publishes" that |
| 08:28:14 | <thuban> | ...which would include whether any given uk-owned domain is noncomplying and will be removed; we know they have that information |
| 08:28:20 | <OrIdow6> | Well, city and country |
| 08:28:37 | <thuban> | that's only for legal (as opposed to natural) persons |
| 08:29:05 | <OrIdow6> | Oh, you're right |
| 08:29:32 | <thuban> | there are some i've seen that look like they're noatural persons, but do include addresses; dunno whose end that happens on |
| 08:31:07 | <thuban> | i guess the question is, if we ask nicely, are we likely to get btfod in a way that would compromise any subsequent scraping operation. i don't have a lot of experience with stuff like that. is this something we should rope jason in on? |
| 08:32:46 | <OrIdow6> | From the way it's written, it sounds to me like it's intended for information on individual domains, rather than a list of everything |
| 08:34:07 | <OrIdow6> | It's not like the registry has control over the contents of the websites, and it's unlikely they'll suspend it early if someone asks |
| 08:35:11 | <OrIdow6> | But I can see there being a possibility - how likely, I don't know - for adverse legal effects |
| 08:38:49 | <OrIdow6> | Here's the form - https://eurid.eu/d/5294809/request_form_disclosure_personal_data_en.docx - clear that it's for a single domain |
| 08:39:08 | <thuban> | ah, disappointing |
| 08:39:26 | <thuban> | i suppose we _could_ ask anyway |
| 08:42:42 | <OrIdow6> | Very unlikely, I think, that they'd give it up |
| 08:43:07 | <OrIdow6> | Well, maybe |
| 08:43:41 | <OrIdow6> | More likely, but still unlikely, if you just asked for a list of domains being revoked |
| 08:45:03 | <OrIdow6> | But the questions are - how willing would they be willing to do that (given that it would be used for web scraping)? And from how high does the decision have to be made? |
| 08:46:35 | <OrIdow6> | (Normally we make a distinction between scraping and what happens here, but to many website owners, and I'd guess in legal contexts, they're the same thing) |
| 08:57:13 | | mutantmonkey quits [Remote host closed the connection] |
| 08:57:31 | | mutantmonkey (mutantmonkey) joins |
| 09:26:47 | | godane1 quits [Read error: Connection reset by peer] |
| 09:26:56 | | qw3rty_ quits [Read error: Connection reset by peer] |
| 09:27:29 | | godane1 joins |
| 09:28:45 | | qw3rty joins |
| 09:31:00 | | godane2 joins |
| 09:31:29 | | godane1 quits [Read error: Connection reset by peer] |
| 09:55:08 | | Atom joins |
| 09:56:11 | | Atom-- quits [Ping timeout: 252 seconds] |
| 10:15:58 | | godane1 joins |
| 10:18:32 | | godane2 quits [Ping timeout: 252 seconds] |
| 10:37:04 | | Barto quits [Ping timeout: 240 seconds] |
| 10:50:06 | | hooway joins |
| 10:53:00 | | HP_Archivist quits [Read error: Connection reset by peer] |
| 10:55:15 | | Barto (Barto) joins |
| 13:06:05 | | godane1 quits [Client Quit] |
| 13:20:39 | <@JAA> | The .eu CDX API scan should be done in a few hours I think. |
| 14:42:04 | | Arcorann quits [Ping timeout: 240 seconds] |
| 14:52:52 | | Ryz quits [Remote host closed the connection] |
| 14:53:36 | | Ryz (Ryz) joins |
| 16:42:07 | | AeonG joins |
| 16:42:18 | | AeonG_ joins |
| 19:47:30 | | lukash quits [Client Quit] |
| 19:52:18 | | mupf joins |
| 19:59:24 | | lukash (lukash) joins |
| 20:40:04 | | Barto quits [Ping timeout: 240 seconds] |
| 20:44:34 | | Barto (Barto) joins |
| 21:02:15 | | AeonG_ quits [Read error: Connection reset by peer] |
| 21:02:16 | | AeonG quits [Read error: Connection reset by peer] |
| 21:04:38 | | spirit quits [Remote host closed the connection] |
| 21:09:00 | | Barto quits [Ping timeout: 252 seconds] |
| 21:52:27 | | Barto (Barto) joins |
| 23:10:45 | | lennier2 joins |
| 23:14:14 | | lennier1 quits [Ping timeout: 252 seconds] |
| 23:14:15 | | lennier2 is now known as lennier1 |
| 23:20:53 | | Arcorann (Arcorann) joins |
| 23:36:16 | | hooway quits [Client Quit] |