HomeSearchPrevious dayNext day

00:47:05systwi (systwi) joins
00:50:23HP_Archivist (HP_Archivist) joins
01:08:22Arcorann_ quits [Ping timeout: 252 seconds]
01:14:26Muad_Dib joins
01:14:56Muad-Dib quits [Read error: Connection reset by peer]
01:22:41klg quits [Ping timeout: 252 seconds]
01:24:16klg (klg) joins
01:25:33Arcorann (Arcorann) joins
02:00:01Dallas quits [Client Quit]
02:03:53Dallas (Dallas) joins
02:09:54<Ryz>Hmm, was checking https://www.machbird.com/ and there's this link being https://business.facebook.com/Machbird2018/?business_id=2136321180018785 - unsure if there's anything significant worth prying out other than one or two interface changes
02:10:38<Ryz>By the way, the business ID number doesn't seem to do anything, if say I have it as https://business.facebook.com/Machbird2018/?business_id=21 (and not any alphabetical letters), it goes to that same kind of link,
02:10:52<Ryz>So we can't do a numerical mining of all the business on Facebook...
02:13:58rsn_ joins
02:17:04rsn quits [Ping timeout: 240 seconds]
02:19:43britmob joins
02:22:24britm0b quits [Ping timeout: 240 seconds]
02:28:43lunik1 quits [Client Quit]
02:28:58lunik1 joins
03:27:42Doranwen quits [Ping timeout: 252 seconds]
03:29:42Doranwen joins
03:32:54godane (godane) joins
04:04:40<HP_Archivist>Maybe someone can help, it's a long shot. Around 2007'ish this restaurants website had an intro after this page (which was captured) with music playing. I was hoping WBM captured the intro/tour music
04:04:42<HP_Archivist>https://web.archive.org/web/20060210121613/http://www.tangerinerestaurant.com/
04:04:57<HP_Archivist>Seems like the pop up that's launched was captured but it doesn't do anything
04:05:01<HP_Archivist>at least not in Chrome
04:06:00<HP_Archivist>Clicking the 'launch site' brings up this https://web.archive.org/web/20060117031556/http://www.tangerinerestaurant.com/tangerine_checker.html
04:06:05<HP_Archivist>But nothing happens sadly
04:10:43<@JAA>That references an SWF file that was captured in 2005: https://web.archive.org/web/2005*/http://www.tangerinerestaurant.com/checker.swf
04:11:32<HP_Archivist>Awesome, I'm wondering if that's the track
04:12:51<HP_Archivist>I'm trying this in Edge and installed Flash for the hell of it, but it just downloads the swf
04:13:47<@JAA>Judging by the filesize, that doesn't contain any audio.
04:15:10<@JAA>Try https://web.archive.org/web/*/http://www.tangerinerestaurant.com/tangerine.swf ?
04:16:23<@JAA>Looks like that references a bunch of other SWFs that weren't captured.
04:16:41<HP_Archivist>I got as far as this in Edge, but it didn't recognize my install of Flash https://imgur.com/jLLJyqh
04:18:15<HP_Archivist>Thanks JAA but that only auto downloads the swf like the other link
04:18:19<@JAA>http://www.tangerinerestaurant.com/tangerine_music.swf is probably the one you're looking for, but it wasn't captured.
04:19:03<HP_Archivist>Damn
04:19:32<HP_Archivist>Oh well, heh. I mean maybe there we other tracks Idk. But this intro music was very suave/lounge type
04:20:00<HP_Archivist>Also, this restaurant no longer exists. Was in a nice location in Philadelphia
04:20:33<HP_Archivist>How did you find these other links, just in the timeline?
04:22:11<@JAA>checker.swf contains a reference to tangerine.html, which lists tangerine.swf, which has references to the various other SWFs including _music.
04:22:26<@JAA>I just looked at the SWFs with less.
04:23:04<@JAA>Also useful: https://web.archive.org/web/*/http://www.tangerinerestaurant.com/*
04:23:24<@JAA>No other SWFs were captured.
04:23:48<HP_Archivist>That is useful ^
04:23:52<@JAA>^ This is also for the person who claimed the other day that SWFs referencing other SWFs are rare.
04:24:53qw3rty__ joins
04:26:07<HP_Archivist>So adding * to the end of a website in the timeline displays all the outlinks/references?
04:27:44<HP_Archivist>Or was that only specific to this url
04:27:45<@JAA>Adding a * to any URL returns a list of all URLs that were captured by the WBM and share the prefix.
04:28:14<HP_Archivist>Ah okay cool. Didn't know that. That's a nifty feature
04:28:28<@JAA>In this case, it lists everything under www.tangerinerestaurant.com, but you can also do e.g. https://web.archive.org/web/*/http://www.tangerinerestaurant.com/tang*
04:28:35qw3rty_ quits [Ping timeout: 252 seconds]
04:30:04<HP_Archivist>what does tang represent?
04:30:09<HP_Archivist>I don't follow that part
04:32:08<@JAA>It lists all URLs that start with http://www.tangerinerestaurant.com/tang regardless of what follows.
04:37:17<HP_Archivist>Hm alright cool, thanks for the tip JAA
05:11:38britm0b joins
05:15:20britmob quits [Ping timeout: 252 seconds]
05:27:40@HCross quits [Read error: Connection reset by peer]
05:28:03HCross joins
05:57:40OrIdow6 quits [Ping timeout: 252 seconds]
06:00:38OrIdow6 (OrIdow6) joins
07:49:36<OrIdow6>VerifiedJ: So do you have this down? It seems you have a lot more capacity than I do
08:09:48Arcorann quits [Read error: Connection reset by peer]
08:12:55<jodizzle>Also, should we throw the archiviolastampa.it in AB? It won't get the really relevant content, but I guess it will pick up the surrounding site.
08:21:27<OrIdow6>Wouldn't hurt
08:21:50<OrIdow6>I imagine that you might have to ignore the search, though
08:22:11<OrIdow6>Only the viewer itself is Flash
08:29:52<jodizzle>Okay, I threw it in
08:30:35Arcorann (Arcorann) joins
08:30:40<jodizzle>I know it's not the immediate goal, but do we know how "replayable" any of this site is going to be after archiving?
08:37:44OrIdow6 quits [Ping timeout: 240 seconds]
08:40:36OrIdow6 (OrIdow6) joins
08:44:02<OrIdow6>Not at all without extensive site-specific playback modifications
08:44:08<OrIdow6>It's got state associated with a session ID cookie, random identifiers in URLs, and more
08:44:14<OrIdow6>Both VerifiedJ and I seem to have approached it from the angle of downloading the images in order to (in my intention, at least) upload them to IA and have them play back in the BookReader
08:55:18hooway joins
08:55:46<jodizzle>Yeah, that's probably for the best. It would be nice to have it work with the Flash application, but that sounds hard.
08:59:53<jodizzle>OrIdow6: What were the search pages you mentioned I might have to ignore in the AB job? Are they the URLs that contain the 'task,search' path component?
09:00:21britmob2 quits [Quit: britmob2]
09:00:27<OrIdow6>Yeah
09:01:10<OrIdow6>Though it wouldn't completely surprise me if AB finds search URLs that don't have that in it, as well
09:01:22<OrIdow6>But I've been using path,search
09:04:17<jodizzle>Okay, I'll pause it and look more in detail later. This site is confusing
09:07:58Hackerpcs quits [Client Quit]
09:09:49Hackerpcs (Hackerpcs) joins
10:02:05katocala quits [Read error: Connection reset by peer]
10:02:27katocala (katocala) joins
10:04:07Arcorann_ joins
10:07:04Arcorann quits [Ping timeout: 240 seconds]
10:29:48<OrIdow6>So here is what I am talking about with my idea of identifying .eu domains that will go away https://transfer.notkiska.pw/inline/eFl3a/test.html
10:31:26<OrIdow6>This obviously needs to be improved, but I think you can see the general idea
10:32:53<OrIdow6>It can be seen that the presence of "gmbh" in the lowered, tokenized text of the page might be a good criterion on which to exclude sites from archiving, for example
10:51:24<OrIdow6>(Assuming it only appears as part of the company type, and not as a common word or spuriously - I don't know German)
10:51:48<thuban>("gmbh" is not otherwise a word; "gen
10:51:57<thuban>" is but is antiquated and rare)
10:52:26<thuban>reasonable approach, if there's actually enough time to process a meaningful portion of ~10e8 domains in this way
10:53:45<thuban>if you're looking for more tokens you might also consider £, imperial units, major and uk place names (as rule-in); most rule-out tokens would be redundant with a language classifier but other legal entity abbreviations (provided they don't collide with english words or abbreviations) would not
10:55:20<thuban>s/major and/major/
10:56:06<OrIdow6>Pounds are in there, there were just less than 3 when this table was generated
10:56:24<thuban>nor would (the english versions of) non-uk eu place names, although they would probably be less reliable
10:56:26<thuban>aha
10:59:09<OrIdow6>Both of those others are good ideas
10:59:56<OrIdow6>I'm already checking for some company types (that's where "spe" comes from) but it needs to be more comprehensive
11:00:11<thuban>https://en.wikipedia.org/wiki/List_of_legal_entity_types_by_country potentially useful
11:00:12<OrIdow6>*"gen"
11:00:21<OrIdow6>Yeah, that's where I got the ones I'm testing witgh
11:00:25<OrIdow6>*with
11:09:09<OrIdow6>Here's how it is now https://transfer.notkiska.pw/inline/hW7nX/test.html
11:09:51<OrIdow6>Interested in suggestions
11:10:06<OrIdow6>More suggestions
11:10:21<thuban>did you say you were doing this on the (front) page text?
11:11:55<OrIdow6>So it turns out where it says it's checking for dollars, it's really checking for pounds, copy and paste error
11:12:14<OrIdow6>Yes, whatever BeautifulSoup's get_text property does
11:12:57<OrIdow6>With a bit more complexity related to redirects and the presence of the www. subdomain
11:13:20<thuban>fwiw, whois.eurid.eu displays registrant information which is not disclosed in normal `whois`--including address.
11:14:09<thuban>i don't think it's accessible through an api but it doesn't require javascript so we could just scrape the page
11:15:33<OrIdow6>Hm, looks like it does
11:15:38<OrIdow6>Could obviate the need for this
11:18:07<thuban>the address formatting isn't always clear or consistent (you know what would be really handy? some sample non-qualifying uk .eu domains for testing) but i bet there's enough information there to make reasonable judgments
11:20:42<thuban>although... all the ones i've checked do have a country code
11:22:25<thuban>(probably worth saving registrar/organization/language in addition to address, in case we need to do some double-checking)
11:25:22<thuban>ah, but some domains don't have an address listed at all. eg https://whois.eurid.eu/en/search/?domain=colbridge.eu
11:27:20<thuban>(/email)
11:29:54<thuban>(oh lol the emails are images. well grab it, or run it through tesseract if you must--mostly legible but seems to lose '.' sometimes)
11:31:50<thuban>might also be worth prioritizing by checking for (a lack of) redirects; at least one of the GB .eus i turned up already redirects to a .co.uk
11:35:36<thuban>i'm going to bed. if nobody's written a scraper for this by next week i'll do it
12:14:34<@kiska>thuban OrIdow6 https://kiska.b-cdn.net/XEm0b/eudomains.zst Maybe this is helpful?
12:21:46<thuban>someone who knows more about dns than me should convert, or explain how to convert, that data into a list of candidates for web-whoising (i'm guessing it's not as simple as stripping the "name" field to the second-level domain since some of those records have *.eu only in "value")
13:15:05britmob joins
13:18:04britm0b quits [Ping timeout: 240 seconds]
13:50:10britmob2 joins
14:34:45Jonboy345 joins
14:47:53Arcorann_ quits [Ping timeout: 252 seconds]
15:39:04Jonboy345 quits [Ping timeout: 240 seconds]
17:08:36Lord_Nightmare quits [Quit: ZNC - http://znc.in]
17:12:03Lord_Nightmare (Lord_Nightmare) joins
17:12:40britm0b joins
17:16:04britmob quits [Ping timeout: 240 seconds]
17:26:48<@JAA>.eu CDX API scan is at page 40k, so a bit under half-way done. 1.24M domains so far.
18:06:38katocala quits [Remote host closed the connection]
18:12:08katocala (katocala) joins
18:14:18Jonboy345 joins
18:30:42Jonboy345 quits [Read error: Connection reset by peer]
18:46:02qw3rty__ quits [Ping timeout: 252 seconds]
19:10:41qw3rty joins
19:11:36JensRex quits [Quit: JensRex]
19:12:05JensRex (JensRex) joins
20:23:25britmob joins
20:26:18britm0b quits [Ping timeout: 252 seconds]
20:50:56<OrIdow6>thuban: I don't think saving emails is necessary
20:52:19lukash quits [Quit: The Lounge - https://thelounge.chat]
20:52:49<OrIdow6>I did a thing to extract from Kiska's list, let me do it again
20:55:45<OrIdow6>Anyhow, I would like to see what proportion of these things, roughly, have either a country listed, or easily inferred
20:57:37<OrIdow6>https://transfer.notkiska.pw/vz7Yf/notkiskas_list_processed.zst
21:02:30<OrIdow6>And as usual... I would rather not anyone try to scrape all of EURID right now - no need for that and it will just make trouble for them
21:03:23<OrIdow6>Right now I think that can be done by the warrior, per-item (one item presumably being a domain), at grab time
21:16:12lukash (lukash) joins
21:43:55lukash quits [Client Quit]
22:44:04Arcorann_ joins
23:04:13lukash (lukash) joins
23:34:18<purplebot>Endomondo edited by Tech234a (+603, Organize page list, add information …) just now -- https://www.archiveteam.org/?diff=45891&oldid=45888
23:38:21<tech234a>Any recommendations for listing files on a given domain? files.endomondo.com has some interesting stuff on it and I've found links from Common Crawl and IA CDX, as well as manual scraping of Google, Bing, Yandex, Ask, Qwant, and DuckDuckGo.
23:43:34hooway quits [Client Quit]

HomeSearchPrevious dayNext day