HomeSearchPrevious dayNext day

00:01:27<OrIdow6>Might want to throw in something from that FDNS list or something similar too
00:03:35<@JAA>Yeah. kiska: Can you get all .eu domains from Sonar?
01:23:01Mikesky quits [Client Quit]
01:27:50<benjins>Parler's APIs used by their mobile app reverse engineered into a Python lib: https://github.com/d0nk/parler-tricks
01:28:01<benjins>Don't know if there's anything in there not in the web api
01:28:36lennier1 (lennier1) joins
01:48:06britm0b quits [Read error: Connection reset by peer]
01:48:29britm0b joins
01:57:53Hintswen2 quits [Read error: Connection reset by peer]
01:58:13Hintswen joins
02:00:01Dallas quits [Client Quit]
02:03:46Dallas (Dallas) joins
02:04:14HP_Archivist (HP_Archivist) joins
02:29:08@Fusl quits [Ping timeout: 252 seconds]
02:32:39Hintswen quits [Read error: Connection reset by peer]
02:33:10Hintswen joins
02:34:42britmob joins
02:37:04lunik1 quits [Ping timeout: 240 seconds]
02:37:34britm0b quits [Ping timeout: 252 seconds]
02:39:44Fusl (Fusl) joins
02:39:44@ChanServ sets mode: +o Fusl
02:43:19Arcorann (Arcorann) joins
02:44:10Arcorann quits [Remote host closed the connection]
02:44:34Arcorann (Arcorann) joins
02:57:04<@JAA>Welp, my CDX API scan got rate-limited and I didn't notice...
02:57:19<@JAA>Didn't even know that was a thing.
02:57:50<@Fusl>need ips?
03:03:20<OrIdow6>Thought there were only 94k?
03:05:12<@JAA>Fusl: Maybe. I don't really want to crash IA's systems, and we already know what a sorry state their APIs are in.
03:05:27<@JAA>OrIdow6: 94k pages of results, so that's 94k requests.
03:07:05<OrIdow6>Oh
03:13:47@Fusl quits [Excess Flood]
03:14:05<OrIdow6>I forgot that it rate-limits, but looking at some old scripts evidently I knew at one point
03:14:12Fusl (Fusl) joins
03:14:12@ChanServ sets mode: +o Fusl
03:14:42<@Fusl>JAA: what do we need that cdx api scan for?
03:16:55<@JAA>Fusl: To get a good list of .eu domains since a good chunk will disappear at the end of the year.
03:27:56<Ryz>Lost loot :c
03:28:00<Ryz>Hunt dor the loot
03:28:03<Ryz>*for
03:28:36<Ryz>Wondering if it would be possible to run the smallest ones first and ramp up gradually
03:28:55<Ryz>Willing to make room if possible for that to happen
03:29:29<@JAA>Ryz: We're talking about more sites here than we've had AB jobs since the inception of AB. There are millions of .eu domains. We're not going to grab it all.
03:29:45<@JAA>(Especially not with AB)
03:31:35<Ryz>That's true, but in this case, we're just doing a surface grab for these websites sadly? :c
03:35:58<tech234a>JAA: if you need a list of domains I mentioned the a Common Crawl one yesterday 5:04 PM <tech234a> Common Crawl released a domains list https://commoncrawl.org/2020/10/host-and-domain-level-web-graphs-julaugsep-2020/
03:36:20<tech234a>5:05 PM <tech234a> they have 89 million domains, or 539 million if you include subdomains
03:36:36<tech234a>5:05 PM <tech234a> though I think they drop www
03:44:22<@JAA>Yup, that's a nice list, 1.52 million .eu domains there.
03:44:48<@JAA>Thanks for the reminder.
03:45:00<OrIdow6>Common Crawl goes into the WBM, doesn't it?
03:45:29<OrIdow6>Well, whatever, if it doesn't, not hard to merge
03:49:10<@Fusl>> they have 89 million domains, or 539 million if you include subdomains
03:49:20<@Fusl>damn. i should have kept my list of 143 billion domains
03:49:41<tech234a>How did you collect those?
03:49:56<@Fusl>dns queries
03:49:57<@JAA>I thought there were only a couple hundred million?
03:50:02<@JAA>Or does that include subdomains?
03:50:04<@hook54321>OrIdow6: https://archive.org/details/commoncrawl?sort=-publicdate
03:50:14<@Fusl>it includes subdomains
03:51:10<tech234a>350 million+ domains according to Google
03:51:31<@JAA>https://transfer.notkiska.pw/ELyhN/cc-main-2020-jul-aug-sep_host_vertices_eu.zst
03:51:51<@JAA>curl -s https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-jul-aug-sep/host/vertices/part-00009-06151a0d-8182-4fdf-976d-b1fc9e211616-c000.txt.gz | zcat | grep -F $'\teu.' | sed 's,^.*\t,,' | sed -E 'G;:t;s/(.*)(\.)(.*)(\n)(.*)/\1\4\5\2\3/;tt;s/(.*)\n(\.)(.*)/\3\2\1/' | zstd
03:54:36<@JAA>Looks like these graphs aren't on IA though.
03:57:25<OrIdow6>hook54321: Good, that's what I though
03:57:29<OrIdow6>*thought
04:03:24<mgrandi>Should probably throw this account into archive bot if someone hasn't done it recently: https://twitter.com/GenChuckYeager/status/1336150145369444352
04:04:35<@JAA>Done
04:11:13<@arkiver>Fusl: damn thats some big list
04:19:05<@kiska>JAA: I can download the thing onto kiska-instagram
04:19:13<@kiska>You'll just have to process it
04:19:22<@Fusl>arkiver: i dumped it when anyqast became a thing
04:20:11<@JAA>kiska: I was hoping I wouldn't have to process it. :-P
04:20:38<@kiska>Um oof?
04:20:38<@kiska>:D
04:22:24<@kiska>Otherwise I am going to do a very generic .eu grep :D
04:24:39HP_Archivist quits [Client Quit]
04:25:52qw3rty_ joins
04:29:24qw3rty__ quits [Ping timeout: 240 seconds]
04:58:19<@kiska>OrIdow6: Wanna process the fdns data?
05:06:07<OrIdow6>kiska: Sorry, busy right now
05:25:16<@kiska>I'll process once I wake up in 8 hours
06:31:28Arcorann_ joins
06:32:26Arcorann quits [Ping timeout: 252 seconds]
07:53:50Dallas quits [Client Quit]
08:14:18<purplebot>Endomondo edited by Tech234a (+3013, Add additional pages, embed URLs, …) just now -- https://www.archiveteam.org/?diff=45888&oldid=45887
08:20:44@Fusl_ quits [Ping timeout: 600 seconds]
08:21:02Fusl_ (Fusl) joins
08:21:02@ChanServ sets mode: +o Fusl_
08:34:16hooway joins
08:39:33mutantmonkey quits [Remote host closed the connection]
08:40:01mutantmonkey (mutantmonkey) joins
08:40:20mutantmonkey quits [Remote host closed the connection]
08:40:39mutantmonkey (mutantmonkey) joins
09:00:18lunik1 joins
09:32:18<purplebot>Coronavirus edited by Jodizzle (+34, /* United States */ Add "Florida …) just now -- https://www.archiveteam.org/?diff=45889&oldid=45745
09:41:44Ryz quits [Remote host closed the connection]
09:54:29Ryz (Ryz) joins
11:29:44mutantmonkey quits [Ping timeout: 240 seconds]
11:34:55mutantmonkey (mutantmonkey) joins
11:54:42FrankZappa joins
11:56:56<FrankZappa>Hi Archive Team. Do you have a project to backup podcasts? If not, do you know any group that does?
12:00:55FrankZappa quits [Remote host closed the connection]
12:23:05FrankZappa joins
12:27:52FrankZappa quits [Remote host closed the connection]
13:56:15Craigle quits [Ping timeout: 244 seconds]
13:59:30Craigle (Craigle) joins
14:02:28britm0b joins
14:02:39<@kiska>So... My command for doing sonar data stuff
14:02:39<@kiska>zgrep -F '.eu"' 2020-12-04-1607040401-fdns_any.json.gz | zstd -o output.zst
14:02:39<@kiska>Its great :D
14:05:48systwi quits [Ping timeout: 252 seconds]
14:06:11britmob quits [Ping timeout: 252 seconds]
14:34:47Arcorann_ quits [Ping timeout: 252 seconds]
15:03:30<@kiska>Whoever wants to do further processing https://kiska.b-cdn.net/XEm0b/eudomains.zst
16:38:34<@kiska>JAA: ^
16:42:55<@JAA>Thanks
16:44:14<@JAA>My slower CDX scan is at page 14k by the way. Should take a couple days like this.
16:44:17<@kiska>So wanna do additional processing? :D
16:44:28<@JAA>Not really no. :-P
16:44:40<@kiska> /*stdin*\ : 11.25% (2640807680 => 297209973 bytes, output.zst)
16:44:41<@kiska>:D
16:44:58<@JAA>:-)
16:45:56<@kiska>Its a lot easier now though :D
16:53:00Dallas (Dallas) joins
17:16:51britmob joins
17:19:45HP_Archivist (HP_Archivist) joins
17:19:47britm0b quits [Ping timeout: 252 seconds]
17:28:02balrog quits [Ping timeout: 252 seconds]
17:40:50yano quits [Client Quit]
17:40:57yanome quits [Client Quit]
17:42:06balrog joins
17:43:33yanome (yano) joins
17:46:37yano (yano) joins
17:54:36balrog quits [Ping timeout: 252 seconds]
18:09:35balrog joins
18:46:24yanome quits [Client Quit]
18:46:32yanome (yano) joins
19:28:55spirit joins
19:29:12HP_Archivist quits [Client Quit]
19:30:03HP_Archivist (HP_Archivist) joins
19:57:58mutantmonkey quits [Remote host closed the connection]
19:58:19mutantmonkey (mutantmonkey) joins
20:05:19mutantmonkey quits [Remote host closed the connection]
20:06:56mutantmonkey (mutantmonkey) joins
20:16:31paul2520 joins
20:17:13<paul2520>This guy is deleting Twitter tomorrow, and I was wondering if his tweets could be preserved. https://twitter.com/FrancoThePerson/status/1336358218709295104
20:22:54<@JAA>Yep, done.
20:24:13<paul2520>thank you!
20:28:43<anarcat>all of twitter?
20:28:50<anarcat>that wouldn't be that bad
20:31:54<paul2520>that works, too
20:44:57<paul2520>oh gosh, I just got your joke. well done.
20:45:58<@kiska>:D
20:50:53<@JAA>Everybody liked that.
20:57:17<mgrandi>https://twitter.com/jrsinhbca his account should be thrown in archivebot, see https://www.thedailybeast.com/former-raytheon-missile-engineer-james-robert-schweitzer-accused-of-leaking-classified-info
21:03:56<@JAA>Done
21:11:54paul2520 quits [Remote host closed the connection]
21:16:29mutantmonkey quits [Remote host closed the connection]
21:16:46mutantmonkey (mutantmonkey) joins
21:38:24Zerote quits [Ping timeout: 240 seconds]
21:41:50Zerote joins
21:49:10<VerifiedJ>Scraping archiviolastampa.it found 47K issues of the paper from 1867 to 2005. http://verifiedjoseph.com/archiveteam/la-stampa-scrape/master.csv I've also written a basic download script that saves issues as _images.zip https://verifiedjoseph.com/archiveteam/la-stampa-scrape/download.php.txt
21:55:26<OrIdow6>VerifiedJ: As I said, I've been working on that too
21:55:41<OrIdow6>Do you have other editions, e.g. La Stampa Sera?
21:57:36<OrIdow6>Additionally, it might be useful if we coordinated
21:57:43<OrIdow6>Instead of each doing it separately
22:15:02<OrIdow6>Additionally do you know what the letter after the issue number in the ID is? I think it relates to scan resolution
22:19:49<VerifiedJ>"21:55:41 <OrIdow6> Do you have other editions, e.g. La Stampa Sera?" No, not yet. So far I've all got ids and date for the main paper.
22:21:03<VerifiedJ>Do you mean IDs like this 1168_01_1924_0066A_0001
22:24:20<OrIdow6>Yes
22:28:20<VerifiedJ>I'm not sure what it denotes, but removing it return a different issue of the paper, in this case the day before.
22:36:02<OrIdow6>Oh, an instance I saw was just the same thing at a different resolution (or compression level)
22:49:37<VerifiedJ>Great, I love it when thing are inconsistent.
22:49:50<OrIdow6>Let me try to dind it
22:51:02britm0b joins
22:54:32britmob quits [Ping timeout: 252 seconds]
22:57:04HP_Archivist quits [Ping timeout: 240 seconds]
22:59:23<VerifiedJ>From playing with http://www.archiviolastampa.it/index2.php?option=com_lastampa&task=issue&no_html=1&type=neighbors&headboard=1 there are 8 different "headboards" La Stampa is 1, La Stampa Sera is 2, [La Stampa] Europa is 3, Tutto libri is 4
23:01:17<VerifiedJ>Tutto scienze is 5, Tutto dove is 6, Tutto come is 7, and Torino Sette is 8.
23:04:19<OrIdow6>Can't find it
23:05:36<OrIdow6>That corresponds to id_testata and nome_testata in the "index2.php?option=com_lastampa&task=issue&no_html=1&type=info&issueid" request, and I think to the second digit group in the ID
23:11:50<OrIdow6>I thinkj
23:13:13<VerifiedJ>yep, looks that way.
23:14:44Arcorann_ joins
23:26:44klg quits [Ping timeout: 240 seconds]
23:28:46klg (klg) joins
23:33:15<OrIdow6>Yeah, can't find that one with the letter, so maybe it was an error on my part
23:52:18<purplebot>FTP/List edited by Flacs (+34, add Korean FTP with lots of old …) just now -- https://www.archiveteam.org/?diff=45890&oldid=45291
23:58:37hooway quits [Client Quit]

HomeSearchPrevious dayNext day