HomeSearchPrevious dayNext day

00:53:03<@JAA>So that finished earlier, almost 4M results.
01:02:14<@JAA>https://transfer.notkiska.pw/wLmO9/wbm-cdx-api-eu.gz
01:02:18<@JAA>3971842 domains
01:02:35<@JAA>I stripped one leading www since we should try all of the domains with and without anyway.
01:34:35<Larsenv>JAA: what the hell is that
01:34:41<Larsenv>almost 4M domains of what?
01:37:21<@JAA>See discussion from the last few days. .eu domains because UK-owned ones are terminated in a few weeks.
01:41:24<Larsenv>JAA: a whole TLD is being terminated?!
01:42:06<@JAA>From the wiki: 'January 1 (0:00 CET): EURid will withdraw all .eu domain names owned by UK residents who are no longer eligible to own such domains through an EU citizenship. The domains will be frozen and non-functional for one year and will then be revoked (i.e. become available for registration again) in early 2022.'
01:42:13<Larsenv>what the fuck
01:42:34<Larsenv> I'm surprised that file is only 70ish MB
02:00:01Dallas quits [Client Quit]
02:03:46Dallas (Dallas) joins
02:12:38Zerote quits [Read error: Connection reset by peer]
02:13:57<mgrandi>https://www.texasgop.org/chairman-allen-wests-response-to-scotus-decision/ is that url working for anyone?
02:14:12<mgrandi>I think their server is struggling
02:14:46Zerote joins
02:15:14<@JAA>Works fine here.
02:16:36<mgrandi>Strange
02:21:37<Ryz>500 for me at this instance
02:21:48<Ryz>Attempt 2 went through
02:53:03hexa- quits [Quit: WeeChat 2.9]
02:53:58britmob joins
02:58:10hexa- (hexa-) joins
04:03:32<OrIdow6>Results from my sample of .eu domains: https://transfer.notkiska.pw/MlRYa/Run%202%20summary
04:04:47<OrIdow6>Long story short, 1% - 12% are UK-owned
04:06:37<OrIdow6>Figuring out where it is in that range depends on what proportion of individual-registered domains with language set as "English" (I'm making the assumption that if somebody has one of the non-English languages in that list, they're not in the UK) are in the UK
04:06:49<OrIdow6>*have registrants in the UK
04:09:14<OrIdow6>(Obviously that doesn't include the statistical margin of error, but that's very small)
04:09:53<OrIdow6>Next time: use the CDX list, set a timeout so it doesn't freeze forever on an SSL handshake
04:11:42<OrIdow6>https://transfer.notkiska.pw/Xv4s5/Run%202 raw data
04:12:52<OrIdow6>tech234a thuban: So we need a way to cut down those "English" ones
04:13:47<OrIdow6>I didn't count the error in the 2090, by the way
04:14:10<OrIdow6>Never mind, yes I did
04:21:30<tech234a>FYI Squad was acquired by Twitter, shutting down "December 12 at 5pm" https://twitter.com/squad/status/1337398991210725379
04:21:57<tech234a>Seems like most of their website is down already
04:23:19<tech234a>https://www.google.com/search?q=site:squadapp.io&filter=0
04:25:26<tech234a>Orldow6: so you'd rather go by language rather than base64 country code images?
04:25:56<tech234a>Ah not all of them have country codes?
04:27:08<OrIdow6>No - thuban figured out that the subset in question (which only have language ane email) are those registered by individuals, and this information gets withold per their privacy policy
04:27:29<tech234a>Ah
04:29:38<tech234a>Also: do Scotland, Wales and Northern Ireland have different country codes/languages? (they're also part of the UK) source: https://www.government.nl/topics/brexit/question-and-answer/which-countries-make-up-the-united-kingdom
04:30:45<OrIdow6>I am almost certain not
04:31:17<OrIdow6>You don't need a source for that :)
04:32:01<OrIdow6>It's called GB, which is the ISO 3166 code
04:33:57<tech234a>Did you figure out how to work with the rate limit?
04:34:33<OrIdow6>Not really
04:34:42<OrIdow6>I put it at 15 and that was mostly stable
04:34:47<OrIdow6>But I'm sure it could go lower
04:36:01<OrIdow6>It's CloudFlare - do you know how that would work? Do they set a rate limit in some config menu, or is that something that CloudFlare does for them?
04:50:17<tech234a>https://support.cloudflare.com/hc/en-us/articles/115001635128
04:57:03qw3rty_ joins
05:00:44qw3rty quits [Ping timeout: 252 seconds]
05:01:48<OrIdow6>Thanks
05:06:12<OrIdow6>I can make 5 requests a few seconds before it tells my to "Retry-After" in 62 or 63 seconds
05:06:15<OrIdow6>*me
05:13:01<OrIdow6>*in a few seconds
05:27:30endrift quits [Quit: +++CARRIER LOST+++]
05:47:54<thuban>hmmm... so i'm comparing those numbers to the official eurid statistics and getting pretty different results.
05:48:03<thuban>https://eurid.eu/en/welcome-to-eurid/statistics/
05:49:28<thuban>3665052 total registrations; numbers for individual country codes sum to 3647842 leaving 17210 presumably-individual registrations; number for GB is given as 118656
05:50:37<thuban>which is tilted almost 10:1 GB-registered to individual, as opposed to OrIdow6's data, which is 1:10 GB-registered to (English-language-only!) individual.
05:58:20<thuban>i'm agnostic about why, at least for now. my main questions are:
05:58:30<thuban>(1) are those numbers, if we believe them, within our archiving capacity, or do we need to filter harder (either on the ambiguous domains, by e.g. checking registrant emails for uk domains, or on the GB domains, by e.g. skipping them if they already redirect to another tld)?, and
05:58:35<thuban>(2) how good is our discovery coverage? JAA sez 3971842, but that's > 3665052. how much of that is defunct domains and/or subdomains, and what proportion of current domains are we missing?
06:02:23<OrIdow6>- Where does it say that the per-country data is only for non-individuals? 0.4% individuals sounds low
06:03:22<OrIdow6>- Not all non-address domains are English-language
06:05:53<thuban>in their WHOIS policy (previously linked: https://eurid.eu/d/205797/whois_policy_en.pdf)
06:06:04<thuban>yes, that's why i specified the distinction
06:07:43<thuban>meanwhile, i munged JAA's cdx discovery to remove subdomains and got 2831856 distinct domains
06:10:01<thuban>(not that i'm suggesting we ignore subdomains; at least some of them are clearly webhosts, etc--but for coverage estimation)
06:12:16<OrIdow6>- Does the WHOIS privacy policy apply to aggregate statistics as well?
06:12:23<OrIdow6>- Where does "10:1 GB-registered to individual" come from?
06:14:27<thuban>i don't know
06:14:28<thuban>118656 (listed number of domains registered with a GB country code) : 3665052 (listed number of domains registered) - 3647842 (listed number of domains registered with any country code)
06:15:30<OrIdow6>Archiving capability: difficult to say, because it's a long tail
06:20:11<OrIdow6>I suppose we might want to start work on the grab script and sample that
06:20:30<OrIdow6>The sizes
06:22:52endrift joins
06:25:40<thuban>combining JAA's cdx scrape with kiska's dns records ("name" field only) yields 3239053 distinct domains
06:26:10<OrIdow6>Going to check my script's results vs. the actual pages
06:26:10<thuban>that sounds good but i've not got any experience with it :X
06:31:21<OrIdow6>At least, it's an opportunity to be creative
06:31:32<OrIdow6>So I suppose here's the plan:
06:31:45<OrIdow6>- Write a grab script
06:32:48<OrIdow6>- Set up a grab project
06:33:40<OrIdow6>- Run a small fraction of the total domains in that project (discovery at the scale of a few thousand doesn't need a dedicated discovery project) and find statistics on their sizes
06:34:14<OrIdow6>- If it turns out it's too big, or big enough that the IA might have problems with it, revise the discovery process
06:34:25<OrIdow6>- Otherwise run the whole thing
06:35:43<thuban>incidentally, https://whois.eurid.eu/en/json/distance/distance-list/?domain=NAME.eu&distance_email_request=False (which is html, despite the url) gives a short list of registered .eu domains similar to NAME, regardless of whether NAME is itself a registered domain
06:37:55<thuban>(have we considered just asking icann https://czds.icann.org/ for the zone file? can anyone convincingly describe themselves as a "researcher" https://czds.icann.org/help ?)
06:39:36<OrIdow6>Even if someone was, it sounds to me like that would require dishonesty about what it's going to be used for
06:41:03<thuban>dunno, might depend how much they like the ia over there
06:45:10<OrIdow6>By doing that, the ICANN, and indirectly the registrar, would effectively be giving approval for, and aid in, archival
06:46:18<OrIdow6>There is a subset of website owners (primarily, as I get the impression, owners of junk sites that make money through ads) that doesn't like this (because it lets spammers, shady SEO people, etc. copy their site)
06:49:10<OrIdow6>Could cause backlash against them
06:49:55<OrIdow6>(Though I don't mean to pretend that the only reason people don't want their sites archived is for money)
06:51:41<OrIdow6>So if someone wants to ask, go ahead, but I think there's almost no chance it will work
08:36:53nerdguy1138 quits [Ping timeout: 252 seconds]
08:52:45nerdguy1138 (nerdguy1138) joins
09:09:11hooway joins
10:12:14spirit joins
10:28:09Zerote quits [Read error: Connection reset by peer]
10:40:55Zerote joins
11:34:17<purplebot>Discord edited by Sanqui (+278, /* Software */) just now -- https://www.archiveteam.org/?diff=45894&oldid=45834
11:43:27lunik1 quits [Client Quit]
11:53:25lunik1 joins
12:02:18<purplebot>Discord edited by Sanqui (+6, /* Software */) 23 minutes ago -- https://www.archiveteam.org/?diff=45895&oldid=45894
12:09:20DLoader_ joins
12:12:50DLoader quits [Ping timeout: 252 seconds]
12:12:50DLoader_ is now known as DLoader
13:52:02qw3rty_ quits [Ping timeout: 252 seconds]
14:00:04spirit quits [Client Quit]
14:42:19Dj-Wawa quits [Quit: Dj-Wawa]
14:42:51Dj-Wawa joins
15:30:44Arcorann quits [Ping timeout: 240 seconds]
15:51:46qw3rty joins
17:21:24britmob quits [Ping timeout: 240 seconds]
18:42:50unfoxo joins
18:42:58<unfoxo>Hello!
18:43:45<unfoxo>Sorry if asking here, perhaps it's not the correct place, but i'm having some issues recovering some website snapshots. I have a couple of warc archives i'd like to extract data from, but without completely downloading them (since they are quite huge).
18:44:35<unfoxo>I've downloaded the .cdx files, and i'm trying to look at the "V" and "S" column (aka compressed arc file offset and compressed record size), but it looks like the file offset is not correct
18:45:07<unfoxo>The file i'm interested in is at height "4191779331222" which (in bytes) corresponds to 4.19TB
18:45:30<unfoxo>The archives however are only ~50gb each, so i don't understand how i would get such value
18:53:21spirit joins
18:53:51<spirit>is anyone using https://www.scaleway.com/en/c14-cold-storage/ ? i am thinking of dropping some archivs into it
18:54:16<spirit>~30€/month to store 30TB is really neat
19:09:23<@JAA>unfoxo: Sounds like a broken CDX. Link?
19:14:17<unfoxo>JAA https://archive.org/download/archiveteam_furaffinity_20150413182755/archiveteam_furaffinity_20150413182755.cdx.gz
19:14:56<unfoxo>https://imgur.com/oLZuSsM.png sizes seem correct but offsets are all over the place
19:16:49<@JAA>spirit: 1) It's €2 per TB and month, not 1. 2) Retrieving the data again will be €10+ per TB. 3) Someone here recently almost lost all their C14 data due to Scaleway's incompetence. (Not going to reveal more information about that without their permission.) 4) #archiveteam-ot for further discussion about that please.
19:18:27<@JAA>unfoxo: The offsets in that screenshot are fine at least.
19:18:51<@JAA>4191779331222 doesn't appear in that file.
19:21:28<@JAA>Largest offset in it is 41967009734, which fits the file size of the WARC.
19:21:40<unfoxo>Is that offset in bytes or what?
19:21:50<@JAA>Yes
19:22:53<@JAA>E.g. curl -sL --range "${offset}-$((${offset}+${length}))" "$warcfile" where offset and length are those two columns.
19:24:28<unfoxo>Uh, wait. I'll see where i found that "222"
19:31:15<unfoxo>... Nevermind, i'm stupid. There was a bug in my script
19:32:23<@JAA>Glad to hear stuff isn't broken. :-)
19:39:48<unfoxo>Perhaps is there an already made software that can download all urls that follow a specific pattern from multiple warcs, given all the indexes?
19:42:34<@JAA>Not that I know of.
19:46:17<@JAA>But that's basically a shell script of a few lines.
19:46:58<@JAA>Or well, even a one-liner I suppose.
19:51:51<unfoxo>I don't think HTTP supports seeking, or does it?
19:52:22<unfoxo>I'm afraid to request every single file (or well, non contiguous ones since it looks like the files i need are scattered all over the place) with a different request.
19:52:54<@JAA>Nope, you'll have to do one request per record unless they're directly adjacent.
19:53:38<unfoxo>At this point i wonder if downloading directly from wayback machine could be more efficient
19:54:09britmob joins
19:54:26<@JAA>Well, depends on how many URLs you're trying to retrieve, I guess.
19:55:09<@JAA>If it's not too many, yeah, WBM is easier. Otherwise, range requests are better.
19:55:47<unfoxo>11 million
19:55:48<unfoxo>Lol
19:56:18<@JAA>Uhh
19:56:49<unfoxo>If the files were in the same order as the indexes, that would be amazingly convenient
19:56:52<unfoxo>But they aren't apparently
19:56:57<@JAA>Check with IA first then.
19:57:43<@JAA>No, indexes are sorted lexicographically, WARCs are whatever we produced there (concatenated items in some order).
19:58:31<@JAA>If it's a significant part of the files you're trying to retrieve, I'd suggest just downloading the WARCs completely and doing the extraction on your side.
20:03:33<unfoxo>Yeah, that'd be the simpler way. I'll see what i can come up with. It looks like there are an incredible amount of repeated files
20:07:10<unfoxo>And i'd like to update the archive from 2015, but my stuff isn't in warc format but rather just the response body. Do you still accept that?
20:07:24<unfoxo>That's around 40 mil files
20:28:02<@JAA>Well, you can certainly pack that up and upload it, but it won't go into the Wayback Machine.
20:33:15Dj-Wawa quits [Client Quit]
20:33:51Dj-Wawa joins
20:42:41<unfoxo>I see :)
20:43:06<unfoxo>They have to be proper warc files to be uploaded there? Or they just can't be uploaded and have to be crawled by archive.org themselves?
20:47:44<@JAA>They have to be WARCs uploaded by an account that's whitelisted for inclusion in the WBM.
20:48:53<unfoxo>I see
21:06:39<unfoxo>Apparently i'm a bit lucky :)
21:07:57<unfoxo>Letting the software search files in a 128k area (so two files 128kb apart are still downloaded as unique range and the useless data discarded), i can download 30~40 files/range
22:03:07<spirit>JAA: re scaleway: uh yuck. i thought transfer was free
22:04:14<unfoxo>The cheapest i've found so far was 1fichier, but you have to work with their shitty APIs
22:13:57spirit quits [Client Quit]
22:16:18<OrIdow6>By the way petabox servers support multipart range requests
22:16:21<OrIdow6>*,
22:17:31<OrIdow6>curl -L "https://ia801405.us.archive.org/31/items/ArchiveNFT-discord-dump-2020-10-08/Archive%20-%20Text%20Channels%20-%20general%20%5B761039098760134730%5D.html" -H "Range: bytes=0-50, 60000-61000"
22:20:33<OrIdow6>In any case, especially if you're downloading media, I've used the approach of including pages you don't want in blocks, if they're not too big
22:20:59<OrIdow6>So if it's 10000 bytes something I want, 1000 something I don't, 10000 something I do, just download the whole thing
22:21:51<unfoxo>I've tried multipart range request but they seem to be ignored
22:22:00<OrIdow6>Strange
22:22:11<unfoxo>Perhaps i'm doing it wrong
22:22:56<unfoxo>Perhaps too manyboundaries?
22:23:02<unfoxo>*too many ranges
22:25:31<unfoxo>I don't even seem to be able to decompress the resulting files
22:28:13<OrIdow6>Maybe it needs to be cached
22:29:51Hintswen2 joins
22:31:30<OrIdow6>Though seems like it works for me on rarely-viewed items
22:31:38<OrIdow6>Maybe you're not parsing the boundry things
22:31:44Hintswen quits [Ping timeout: 240 seconds]
22:32:19<unfoxo>Parsing the boundary things?
22:32:19<unfoxo>Oh
22:43:10Lord_Nightmare2 (Lord_Nightmare) joins
22:44:26Lord_Nightmare quits [Ping timeout: 252 seconds]
22:44:26Lord_Nightmare2 is now known as Lord_Nightmare
23:02:29britm0b joins
23:06:04britmob quits [Ping timeout: 240 seconds]
23:07:17britmob joins
23:09:10britm0b quits [Ping timeout: 252 seconds]
23:12:14britm0b joins
23:15:46britmob quits [Ping timeout: 252 seconds]
23:28:18godane (godane) joins
23:28:27Arcorann (Arcorann) joins
23:28:57unfoxo quits [Remote host closed the connection]
23:35:35<Ryz>...Huh, I found something interesting, certain Blogspot websites have stuff like https://chatterbusy.blogspot.com/ads.txt and https://anakpertamakudeveloper.blogspot.com/ads.txt - not all of it do have it, but unsure if it applies only to the new ones or if retroactive
23:35:59<Ryz>...I first found it from https://play.google.com/store/apps/details?id=com.parkjam.parkink3d by going to "Visit Website" to get that said link
23:37:18<Ryz>Oh, it doesn't seem to limit to Blogspot websites, but maybe any website? There's this: https://handwritinguniversity.com/ads.txt
23:38:28<Ryz>I wonder if there's a way to make AB auto-grab those files, since robots.txt and sitemap.xml also get auto-grabbed
23:39:09<klg>TIL https://www.adstxt.com/
23:39:31<Ryz>Ooo wow~
23:39:47<Ryz>We should also proactively grab humans.txt as per https://en.wikipedia.org/wiki/Robots_exclusion_standard#Alternatives
23:40:09<Ryz>And I suppose credits.txt as per https://creditstxt.com/
23:58:08wickedplayer494 quits [Ping timeout: 252 seconds]
23:59:06wickedplayer494 joins

HomeSearchPrevious dayNext day