HomeSearchPrevious dayNext day

00:03:46<billotronic>aye aye. Thanks
00:05:09kelsair quits [Remote host closed the connection]
00:14:08JSharp (JSharp) joins
00:25:46no112 quits [Ping timeout: 252 seconds]
00:39:23EleventyFour joins
00:47:52tachikoma joins
00:50:32wuzzl3 joins
00:52:34hooway quits [Client Quit]
00:53:16WH1sp3r joins
00:57:35Arcorann (Arcorann) joins
01:00:59<PhantomTypist>who deals with the https://github.com/ArchiveTeam/universal-tracker repo?
01:03:49EleventyFour quits [Remote host closed the connection]
01:10:51<PhantomTypist>just asking because the last time anyone closed a PR in that repo was forever
01:11:24rsn joins
01:11:29rsn quits [Remote host closed the connection]
01:20:49tachikoma quits [Remote host closed the connection]
01:25:24<mgrandi>@JAA might know
01:26:06daalekz1 quits [Client Quit]
01:26:47Y48fyrCgHPR7d18W94 quits [Remote host closed the connection]
01:30:15trinsic joins
01:30:42jeff-kao-propublica-org quits [Client Quit]
01:35:45zyphlar1 joins
01:36:27Jerb quits [Remote host closed the connection]
01:37:02zyphlar1 quits [Client Quit]
01:37:18zyphlar1 joins
01:40:13<Kaz__>plenty of people. If you have a specific question, go ahead and ask
01:40:29<Kaz__>https://dontasktoask.com/
01:41:44sliccricc quits [Ping timeout: 240 seconds]
01:43:30<@JAA>PhantomTypist made a PR and was wondering who to bug to take a look at it.
01:45:12<@JAA>I think the pointer cursor is a leftover from when the tracker still had graphs.
01:45:19<@JAA>https://github.com/ArchiveTeam/universal-tracker/pull/42
01:47:10jeff-kao-propublica-org joins
01:47:50<Kaz__>does it need to be specified if the thing you're specifying is default?
01:48:24<@JAA>Nope
01:48:50<@JAA>(Unless some parent element sets a non-default value.)
01:49:34<PhantomTypist>i left that there because i really didn't spend the time looking over everything
01:49:48<Kaz__>sign of a great commit right there
01:49:54<PhantomTypist>if the graph is gone (i never saw it before), then yea, it can be obliterated
01:50:28<PhantomTypist>want me to nuke that line?
01:50:55<PhantomTypist>any reason why the graph for each person was deprecated?
01:51:05<Kaz__>it hurts to render it
01:51:09<PhantomTypist>gotcha
01:51:16<PhantomTypist>makes sense
01:56:45jtt123 quits [Remote host closed the connection]
01:57:55jtt123 (jtt123) joins
02:00:46sliccricc joins
02:01:53prebala-publici joins
02:05:06jtt123 quits [Client Quit]
02:06:54sdf quits [Remote host closed the connection]
02:07:13insomniacrando quits [Remote host closed the connection]
02:07:15sliccricc quits [Client Quit]
02:10:09sliccricc joins
02:14:53crispyalice2 joins
02:19:23sliccricc quits [Remote host closed the connection]
02:22:00sliccricc joins
02:24:01trinsic quits [Ping timeout: 252 seconds]
02:24:09trinsic joins
02:27:51kelsair joins
02:40:51nertzy (nertzy) joins
02:40:54crispyalice2 quits [Remote host closed the connection]
02:53:19WH1sp3r quits [Remote host closed the connection]
02:53:55jtt123 (jtt123) joins
02:54:34Hubbard joins
02:59:05crispyalice2 (crispyalice2) joins
03:00:14nertzy quits [Client Quit]
03:05:04ccafe quits [Ping timeout: 240 seconds]
03:10:18ccafe joins
03:21:51<PhantomTypist>fudge. AT should get started on this one: https://www.archiveteam.org/index.php?title=Docker_Hub
03:24:37<@JAA>There's a channel for that, and yes, it will happen.
03:34:06wumbo quits [Remote host closed the connection]
03:41:31TestCrap quits [Remote host closed the connection]
03:51:55Arcorann quits [Ping timeout: 265 seconds]
03:53:45Arcorann (Arcorann) joins
03:58:19demoneyes (demoneyes) joins
04:01:42seyonce joins
04:02:13seyonce74 joins
04:03:00DLoader quits [Quit: DLoader]
04:03:14esobrev joins
04:05:35smm684555 joins
04:05:55seyonce quits [Remote host closed the connection]
04:07:46demoneyes leaves
04:13:30MuffinCumberbun joins
04:18:49TheJobd joins
04:19:57Arcorann quits [Ping timeout: 265 seconds]
04:23:52qw3rty_ joins
04:26:10esobrev quits [Client Quit]
04:27:04qw3rty__ quits [Ping timeout: 240 seconds]
04:32:02Hubbard quits [Client Quit]
04:39:41DogsRNice quits [Read error: Connection reset by peer]
04:40:49seyonce74 quits [Ping timeout: 244 seconds]
04:48:26mjh (mjh) joins
05:01:44etnguyen03 quits [Client Quit]
05:07:55zyphlar1 is now known as zyphlar
05:13:53willow quits [Ping timeout: 244 seconds]
05:22:10willow joins
05:34:01smm684555 quits [Client Quit]
06:06:53daalekz joins
06:06:59daalekz1 joins
06:07:03daalekz1 quits [Remote host closed the connection]
06:12:26PhantomTypist quits [Remote host closed the connection]
06:26:13Webuser572 quits [Ping timeout: 244 seconds]
06:49:56blue5072 joins
06:53:10blue507 quits [Ping timeout: 265 seconds]
07:10:52neatneatneat (neatneatneat) joins
07:16:37LeGoupil joins
07:16:51neatneatneat quits [Remote host closed the connection]
07:26:48jeff-kao-propublica-org quits [Client Quit]
07:28:13nickyyyyy joins
07:28:13wuzzl3 quits [Ping timeout: 244 seconds]
07:33:12no112 joins
07:33:50<no112>where are these left wing extremists
07:35:54kelsair quits [Remote host closed the connection]
07:36:08<flashfire42>👀
07:42:24<no112>im not actually one of *those* people dw i just had to make the joke
08:18:53DLoader joins
08:19:33Arcorann (Arcorann) joins
08:23:43daalekz quits [Ping timeout: 252 seconds]
08:27:38willow quits [Ping timeout: 244 seconds]
08:28:00daalekz joins
08:47:59hooway joins
08:54:02baviair quits [Read error: Connection reset by peer]
08:57:45ana joins
08:58:15ana quits [Remote host closed the connection]
09:00:26ragu joins
09:21:31lincoln1 (lincoln) joins
09:22:34lincoln quits [Ping timeout: 252 seconds]
09:33:51benjinsmith joins
09:36:03benjins quits [Ping timeout: 265 seconds]
09:41:49vivid-noble quits [Ping timeout: 252 seconds]
09:43:02<AK>Which is the best channel for the /urls-grab repo?
09:43:47yawkat quits [Ping timeout: 265 seconds]
09:46:32<jodizzle>AK: The channel is #//
09:47:02<AK>Ahh, my mistake, I thought that was a typo when I saw it in the repo
09:47:39<jodizzle>One of archiveteam's best names
09:47:48yawkat (yawkat) joins
09:59:53trinsic quits [Read error: Connection reset by peer]
10:00:21trinsic joins
10:02:27emerald quits [Read error: Connection reset by peer]
10:08:51atomicthumbs quits [Quit: http://quassel-irc.org - Chat comfortably. Anywhere.]
10:13:27hexa- quits [Ping timeout: 600 seconds]
10:17:34Datechnom quits [Ping timeout: 252 seconds]
10:22:05hexa- (hexa-) joins
10:23:14Datechnom (Datechnom) joins
10:28:13<flashfire42>https://www.godlikeproductions.com/ would love to see this grabbed at some point
10:28:40<Sanqui>cloudflare :/
10:28:48<flashfire42>I know that’s the problem
10:28:59<Sanqui>i think there's a list of sites behind cf on the wiki
10:29:07<Sanqui>you can at least add it there I suppose
10:30:55<flashfire42>I’ll do it later if I remember and if there isn’t a page like that on the wiki I’ll do that tomorrow as well. I made the excluded from wayback list ages ago
10:32:05<Sanqui>awesome, thank you or your work
10:32:09<Sanqui>s/or/for/
10:43:04atomicthumbs joins
10:52:11ccafe_ joins
10:52:36<jut>https://archiveteam.org/index.php?title=Parler Meaby change the status from in progress to something else?
10:53:55<Billy549>It might also be a good idea to also mention that the site might make a return via Epik and link the Tweet
10:54:25ccafe quits [Ping timeout: 252 seconds]
11:06:16revi quits [Read error: Connection reset by peer]
11:06:16brad quits [Write error: Connection reset by peer]
11:06:44brad joins
11:07:46revi (revi) joins
11:15:42Doranwen quits [Remote host closed the connection]
11:15:58Doranwen joins
11:23:44Jake quits [Quit: Leaving for a bit!]
11:26:36<lmx>jut isn't it kinda still in progress? Upload isn't done
11:28:07<Billy549>I agree it's still in progress
11:33:18Jake (Jake) joins
11:33:59<mgrandi>Epik? and yes i'm confident they will attempt to return, although migrating away from amazon probably sucks
11:34:20<Billy549>The people who host Gab
11:34:27<Billy549>have said they will host Parler
11:34:40<Billy549>https://twitter.com/parlertakes/status/1348766514816512000
11:58:50<AK>Dallas: Looks like I'm gonna overtake you on reddit soon :P probably gonna be the only thing I get close on
12:02:44<Dallas>:( I don't have the ip's for that project so I'm sticking to urls lol
12:21:52trinsic quits [Ping timeout: 252 seconds]
12:23:24Mateon1 quits [Ping timeout: 240 seconds]
12:26:40Mateon1 joins
12:45:24Lord_Nightmare quits [Ping timeout: 240 seconds]
12:58:07ccafe_ is now known as ccafe
12:59:35jeff-kao-propublica-org joins
13:26:52ccafe quits [Client Quit]
13:28:24ccafe (ccafe) joins
13:28:59ccafe quits [Client Quit]
13:33:40Lord_Nightmare (Lord_Nightmare) joins
13:36:20benjinsmith is now known as benjins
13:40:31Peetz0r quits [Ping timeout: 252 seconds]
13:40:50Peetz0r (Peetz0r) joins
13:45:33etnguyen03 (etnguyen03) joins
13:49:44Yg84WJchYH87d3A5tF joins
14:09:04Datechnom quits [Ping timeout: 240 seconds]
14:18:25<blue5072>Does the ArchiveBot take exact copies of the website?
14:18:28trinsic joins
14:18:30<blue5072>the whole website ?
14:19:24<@arkiver>yes
14:19:33<@arkiver>only public data
14:19:44<@arkiver>and sometimes horrible JS can cause problems
14:21:11<blue5072>:)
14:21:13<blue5072>nice one
14:21:40<blue5072>i guess it has to play catch up for sites which are always growing?
14:21:43some_donkus joins
14:21:56<blue5072>sorry for the questions I am new here
14:23:16<some_donkus>I am having permission denied when running the watchtower script. Is there something else I need to do after installing docker?
14:24:03Datechnom (Datechnom) joins
14:30:02<etnguyen03>some_donkus: prefix with "sudo"? Unless you added your user to the docker group you have to run commands as root IIRC
14:31:08<Craigle>blue5072: AB basically runs through the site and grabs a copy. For sites that change, it would need to be re-run periodically. That's why it is generally run for sites that are considered "in danger" as the priority
14:31:22Arcorann quits [Ping timeout: 265 seconds]
14:32:44<Craigle>some_donkus: To add to what etnguyen03 said, here are the instructions for creating the group and granting rights: https://docs.docker.com/engine/install/linux-postinstall/
14:32:59<some_donkus>etnguyen03 Thank you that appears to be working
14:33:01<Craigle>If you'd rather not have to sudo everything
14:33:24<some_donkus>Danke, Craigle
14:35:42<some_donkus>Is this the right chat to get the deets for a project container?
14:36:42<Craigle>What project are you looking for?
14:40:59<some_donkus>I suppose github or reddit, anything really that needs the assistance.
14:41:16<some_donkus>I see the specific project channels now
14:42:03<Craigle>Github is in #gitgud and the docker is at ancestor=atdr.meo.ws/archiveteam/github-grab
14:42:17<AK>Docker images are at: atdr.meo.ws/archiveteam/[project_name]:latest (e.g. urls-grab or reddit-grab)
14:42:19<Craigle>Reddit is in #shreddit and the docker is at ancestor=atdr.meo.ws/archiveteam/reddit-grab
14:42:21<AK>Doh, got beaten
14:42:41<Craigle>Haha, no problem
14:42:55<Craigle>On that note, the URLs projects is in #//
14:43:24Datechnom quits [Ping timeout: 240 seconds]
14:45:01Datechnom (Datechnom) joins
14:46:18trins_ joins
14:49:44trinsic quits [Ping timeout: 265 seconds]
14:56:39<@JAA>flashfire42: If you do create such a list, let me know, and I'll instruct my bot to do the sorting etc. like on the WBM exclusion list.
14:59:35<EggplantN_>hey arkiver can we start looking to implement airsync-meta in projects by default?
14:59:47<EggplantN_>i.e have it try, if it fails just continue as normal
15:00:49Yg84WJchYH87d3A5tF quits [Ping timeout: 244 seconds]
15:01:26Kenshin quits [Client Quit]
15:02:10Yg84WJchYH87d3A5tF joins
15:02:22nickyyyyy quits [Ping timeout: 244 seconds]
15:03:49blue5072 is now known as blue507
15:04:57Kenshin joins
15:10:08HugsNotDrugs joins
15:13:45HugsNotDrugs waves
15:17:46daalekz quits [Ping timeout: 265 seconds]
15:35:18MuffinCumberbun quits [Remote host closed the connection]
16:00:39PhantomTypist (phantomtypist) joins
16:12:54Ruthalas quits [Read error: Connection reset by peer]
16:14:04Doranwen quits [Ping timeout: 240 seconds]
16:18:50some_donkus quits [Ping timeout: 244 seconds]
16:22:28Doranwen joins
16:22:50DogsRNice (Webuser299) joins
16:35:24jut quits [Ping timeout: 240 seconds]
16:35:58trins_ quits [Ping timeout: 252 seconds]
16:38:16TheFluffyCat joins
16:42:04@kiska quits [Ping timeout: 240 seconds]
16:42:04s-crypt quits [Ping timeout: 240 seconds]
16:42:04flashfire42 quits [Ping timeout: 240 seconds]
16:44:35<TheFluffyCat>Does anyone know of a flickr project? I don't see it in the list on the Mediawiki. A lot of amazing content on there, not sure if there's any backups.
16:45:11<@JAA>https://archiveteam.org/index.php?title=Flickr
16:45:20Doranwen quits [Remote host closed the connection]
16:45:36Doranwen joins
16:45:59<TheFluffyCat>I know it's THERE, but it seems fairly untouched project-wise
16:46:16<@JAA>We had a project two years ago before the free account purge.
16:46:57<Billy549>I wonder if it'd make sense to add it to warrior at some point?
16:47:26<TheFluffyCat>Warrior's a little old- might be best to make a new docker container for it.
16:47:52<TheFluffyCat>Unless they update it. I heard buzz of that happening in #warrior.
16:47:53<Billy549>well isn't there the new docker container warrior
16:48:26<Billy549>Yeah - some people would probably want to use one Docker container that can be changed to do other projects
16:48:43<TheFluffyCat>The docker one only works for URLTeam at the moment :/
16:48:47<Billy549>cause iirc it only works with URLTeam2 rn, so i have the Reddit Docker and the general Docker
16:48:48<Billy549>yeah
16:49:28<Billy549>I hope that'll get fixed - i got one of the docker containers that can apparently update your other ones in the background, it was mentioned here before
16:49:56jut (jut) joins
16:49:58<TheFluffyCat>Yeah, that's the one they used on the Parler archive to push updates
16:55:13jut quits [Ping timeout: 252 seconds]
16:59:33jut (jut) joins
17:21:10vela (vela) joins
17:21:37billotronic quits [Ping timeout: 252 seconds]
17:22:44billotronic (billotronic) joins
17:29:47aleph joins
17:32:44vivid-noble joins
17:40:17spirit joins
17:43:16whattheheckisthis joins
17:44:32whattheheckisthis quits [Remote host closed the connection]
17:45:05sdfsdfsdfsdfs joins
17:45:16sdfsdfsdfsdfs quits [Remote host closed the connection]
18:05:35some_donkus joins
18:08:49Doranwen quits [Remote host closed the connection]
18:09:06Doranwen joins
18:14:37<atphoenix>new post on https://twitter.com/archiveteam/status/1349054073589420036
18:40:37kageurufu quits [Client Quit]
18:50:13Doranwen quits [Remote host closed the connection]
18:50:29Doranwen joins
19:02:34<rewby>Hey, I noticed that the grabs don't talk to tracker.archiveteam.org but to trackerproxy.archiveteam.org. Is the source code for trackerproxy online somewhere?
19:11:07<billotronic>ffs thanks to whomever modded neparlepas. crap like thats probably gonna get worse before it gets better right?
19:13:35<@JAA>If people keep taking it to other channels, yes.
19:14:28<@JAA>rewby: https://git.kiska.pw/ArchiveTeam/trackerproxy
19:17:20<@SketchTheCow>And how is the shitshow today
19:17:43<@SketchTheCow>How many more "members" did we get
19:18:56<jut>People want to talk to the "manager"
19:19:09<Dallas>#neparlepas is at 192 members, it may have been higher at some point with the amount of people disconnecting and reconnecting on web clients :/
19:21:32divi (divi) joins
19:21:44<Dallas>loving that this lead to the @archiveteam account tweeting for the first time in 5 years haha
19:21:52bruhmomento (bruhmomento) joins
19:22:03brabo joins
19:22:12<rewby>JAA: Interesting. Looks less like a proxy and more like a complete tracker backend.
19:23:54<@SketchTheCow>Please address all fuckstickery to me
19:24:07Mineroboter joins
19:24:15phuzion receives a bullshit email at work, forwards to SketchTheCow
19:24:29<@JAA>rewby: Yeah, it was originally a proxy but evolved into this. The original tracker wouldn't be able to handle the load nowadays.
19:24:34<@SketchTheCow>I have a guy who doesn't like my "tone" lighting up my DMs on here explaining what a shame Archive Team isn't allowing him to collaborate with him, where he says he can do "All the work you're doing, alone, in half the time."
19:24:48<@SketchTheCow>And I'm watching him work himself up.
19:24:56isthisthedarkweb joins
19:25:06<Craigle>Great, he should do that. The alone thing
19:25:12<phuzion>^^
19:25:14<@JAA>I feel like I know who that is.
19:25:17<rewby>JAA: I can imagine. Although I noticed even this struggling during parler.
19:25:21<Dallas>I can guess who that is from the convo before
19:25:30<@SketchTheCow>I like that you think you know who it is and there's, like, 10 suspects
19:26:00<@JAA>Heh, true.
19:26:34<rewby>And now I'm theorycrafting the complexity of scaling up the tracker"proxy"
19:26:43<rewby>Why do I get nerdsniped so easily
19:26:57<@JAA>It's already being worked on.
19:27:09<rewby>Is there anywhere to watch that work?
19:27:12<rewby>I'm curious
19:27:21<@JAA>Don't think so.
19:27:23<@SketchTheCow>================= FIRST RULE OF HELPING ARCHIVE TEAM IS DO NOT WANDER INTO THE MAJOR PROJECTS THINKING YOU ARE GOING TO TESLA THE PROJECT ===============
19:27:31<rewby>I wasn't planning to
19:27:39<rewby>I just like seeing how people do things
19:27:41<@SketchTheCow>We've probably been working on the problem you've hackernews-sketched for 10 minutes, for months
19:27:56<rewby>I know, and I'd love to know what kind of solution you came up with
19:27:59<@SketchTheCow>I'm live-streaming adding new flash loops to the archive
19:28:17<bruhmomento>whats your twitch/yt
19:28:21<phuzion>rewby: The tl;dr is basically: If we want contributions on something, it's on github under the archiveteam group. If it's not meant for the public to see, it's not publicly accessible.
19:28:37<@JAA>phuzion: Except for the stuff that's on git.kiska.pw. :-P
19:28:38<rewby>That makes sense, I'll shut up then.
19:28:48LordThanatos joins
19:28:54<@SketchTheCow>Well, shut up and watch
19:28:56weezel joins
19:28:56FriskyandSickly (FriskyandSickly) joins
19:29:13<rewby>SketchTheCow: Yes, I wanted to watch. And now I know I can't watch.
19:29:17<@SketchTheCow>If arkiver is any indication, the key to domination is to start watching when you're 9 and then jump in at full at just the right time
19:29:37<@SketchTheCow>Then boom, we all wonder how long he's been in charge
19:30:00<brabo>:)
19:30:53<rewby>I'm sorry if it came off as such, I have no intention of meddling with people doing actual work. I just enjoy being a fly-on-the-wall and seeing what decisions get made and why.
19:31:13<Kaz__>rewby: I'm pretty sure you do know the right place to ask those questions ;) but also, throwing more things (as well as some smart people) https://usercontent.irccloud-cdn.com/file/qbMaHdsY/image.png
19:31:32hex joins
19:31:45<Kaz__>oh that cuts my message off
19:31:46<rewby>Kaz__: I don't know the right places.
19:31:54<Kaz__>'throwing more things and smart people at it usually helps'
19:32:03<rewby>But SketchTheCow was clear that I should just stay away.
19:32:16<Kaz__>then you're probably not the person I thought you were
19:32:31truthful (truthful) joins
19:33:11doyleheaf joins
19:34:28<jodizzle>rewby: I've heard that the "I" in IRC stands for "idle", so just staying connected is a good idea
19:34:56<jodizzle>Also note that there's another channel, #archiveteam-dev, that's specifically for software development
19:35:01<rewby>I was planning to stay connected, yes. But I'll probably remain mostly quiet.
19:37:56truthful quits [Remote host closed the connection]
19:49:55<EggplantN_>rewby as TMT?
19:50:08<rewby>Yes
19:56:44<AK>What's the first number before the equals sign in the log messages? (e.g the 8 in: 8=429 https://www.instagram.com/accounts/login/)
19:58:08<jut>the number of url grabbed = HTTPcode URL
19:58:29<TheFluffyCat>How do you upload files to FOS?
19:59:17<@arkiver>to all new people - we have a mediafire project coming up
19:59:22<@arkiver>#mediaonfire
20:00:11<AK>Ahh thank you jut
20:01:52kelsair joins
20:06:20saizai joins
20:06:33<saizai>how can I make a request for archival?
20:07:02<thuban>saizai: right here will do; what did you have in mind?
20:07:05<saizai>and, can your tools handle browser emulation?
20:07:19<thuban>the short version is it depends
20:07:33<saizai>heh. ofc
20:07:41<saizai>https://aad.archives.gov/aad/series-description.jsp?s=5057&cat=all&bc=sl
20:07:43kelsair quits [Ping timeout: 244 seconds]
20:08:55<saizai>limits: 1k per dl; must be binned by DOB year, starting 1800 [yes srsly]; DL link has a one time token, unsure precisely how generated
20:09:14<atphoenix>Kaz__, what makes that chart you linked to?
20:09:28<zyphlar>saizai any reason to suspect this data will go away any time soon?
20:09:55<saizai>not go away, but unavailable in useful bulk for research
20:10:25<saizai>this is NARA AAD after all
20:10:46<Craigle>atphoenix: Looks like htop
20:12:53<zyphlar>it seems theoretically possible to archive every page like https://aad.archives.gov/aad/record-detail.jsp?dt=3080&cat=all&tf=F&q=a&bc=sl,sd&rpp=10&pg=1&rid={INTEGER} where {INTEGER} is 1 thru some max
20:13:14<zyphlar>have you tried asking them for a full dump? i know websites like Ancestry seem to have programmatic access
20:14:03<thuban>also, which download link are you referring to? i'm not seeing one
20:15:49benjins quits [Read error: Connection reset by peer]
20:16:17<Kaz__>atphoenix: it is indeed htop
20:17:02<saizai>thuban: e.g. https://aad.archives.gov/aad/display-partial-records.jsp?dt=3059&sc=29592%2C29578%2C29580%2C29596%2C29583%2C29584%2C29585%2C29608%2C29612&cat=all&tf=F&bc=sl%2Cfd&q=&btnSearch=Search&as_alq=&as_anq=&as_epq=&as_woq=&nfo_29592=V%2C9%2C1900&op_29592=0&txt_29592=&nfo_29578=V%2C15%2C1900&op_29578=0&txt_29578=&nfo_29580=V%2C15%2C1900&op_29580
20:17:02<saizai>=0&txt_29580=&nfo_29596=V%2C20%2C1900&op_29596=0&txt_29596=&nfo_29583=V%2C2%2C1900&cl_29583=08&nfo_29584=V%2C2%2C1900&cl_29584=&nfo_29585=N%2C4%2C1900&op_29585=3&txt_29585=1865&txt_29585=&nfo_29608=V%2C12%2C1900&op_29608=0&txt_29608=&nfo_29612=V%2C3%2C1900&cl_29612= download results top right
20:17:07Mineroboter_ joins
20:17:10Mineroboter quits [Read error: Connection reset by peer]
20:17:15<zyphlar>and then dt=3060 is the series, like 3060 is C-D, 3080 is S-T
20:17:20benjins joins
20:17:24<saizai>I have it documented
20:17:36mutantmonkey (mutantmonkey) joins
20:17:40<saizai>scripted it already. limit is IP blocking.
20:18:10<zyphlar>if you request an out-of-bounds RID like rid=9000000 it just returns a blank page
20:19:03<atphoenix>thanks for confirming htop
20:19:13<saizai>zyphlar: seems possible but high burden on server vs getting <1k per req
20:19:16<@arkiver>can we move htop talk to #archiveteam-ot please
20:19:20<saizai>in csv
20:19:21<@arkiver>saizai: interesting
20:19:52<saizai>I'd like the whole thing in csv. mind that each dt has different fields
20:20:18<thuban>saizai: upload the script so we can look?
20:20:44<saizai>sure. gimme a bit; it's on remote shell
20:20:51<zyphlar>oh i see
20:20:52<zyphlar>"You searched for: DATE OF BIRTH (MONTH) = 08; DATE OF BIRTH (YEAR) = 1865; STATE OR FOREIGN COUNTRY OF BIRTH = download results top right"
20:21:07<zyphlar>good way of doing that, and then they give you a download. cool
20:21:30<saizai>mind you need to check number of rows. it does partial cutoff sometimes
20:21:36isthisthedarkweb quits [Remote host closed the connection]
20:21:42<@hook54321>zyphlar: ancestry oftentimes has deals with governments to get special access to data, sometimes exclusively even if the data is supposed to be public.
20:21:49isthisthedarkweb joins
20:22:07<zyphlar>sure just saying a FOIA "give me all the SS death indexes digitally" should also take care of it
20:22:16<saizai>hook54321 is right. I have legal process running on this but want this version for now
20:22:32<saizai>FOIA filed months ago. they want to charge
20:22:40<saizai>a few k
20:22:43<zyphlar>eesh
20:23:08<zyphlar>even for not csv just raw whatever they store it in?
20:23:11<saizai>no bulk dl avail, only this search or enumeration
20:23:27<saizai>I precisely asked for their native format. :/
20:23:32<zyphlar>what dicks
20:24:13<saizai>SSA's fault probably. will be publishing about it soon.
20:25:05<saizai>"free", "public", but only available to Ancestry
20:26:09<zyphlar>you're right this is a good candidate for AT Warrior since we can get it all slowly from many IPs
20:26:38<atphoenix>saizai, 'The best government per dollar available anywhere'
20:26:49<saizai>srsly right
20:27:40<saizai>though there'll be more info on that in my publication. some interesting chicanery.
20:28:23LordThanatos quits [Ping timeout: 244 seconds]
20:28:40<zyphlar>do these not have death dates?
20:29:24<saizai>the ss5 and claim don't
20:29:31<saizai>so needs to bin by DOB not DOD
20:29:41<zyphlar>gotcha
20:31:00<saizai>https://pastebin.com/RRE6XXHv
20:35:37weezel quits [Ping timeout: 244 seconds]
20:36:49<saizai>https://pastebin.com/3PQFYxp8
20:37:04<saizai>quick redaction of some server specific stuff but should work
20:37:42<saizai>some cruft from prior iterations also but those might be useful
20:38:24<saizai>due to fake no-record downloads and aborts, it does need to scrape the number records claimed by search
20:39:59<saizai>don't know how your distributed image works so figured I'd punt that to you :p
20:41:00flashfire42 (flashfire42) joins
20:41:04s-crypt (s-crypt) joins
20:41:28kiska (kiska) joins
20:41:28@ChanServ sets mode: +o kiska
20:44:29Mineroboter_ quits [Remote host closed the connection]
20:44:39Mineroboter_ joins
20:44:58<saizai>fyi in case you do this, I'm https://s.ai / https://keybase.io/saizai — would appreciate a ping, since I'm not usually on IRC
20:49:01<thuban>d00d i like your conlangs :O
20:49:12<saizai>o.o you know them?
20:49:46pgup (pgup) joins
20:49:53<thuban>as of about five minutes ago, yes
20:50:06LordThanatos joins
20:50:17<saizai>heh. thought this might be a small world thing :p
20:50:28<saizai>but thanks. /curtsy
20:50:53<saizai>UNLWS is my favorite, though other people seem enamored of the idea of Gripping
20:51:09<@JAA>Off-topic stuff in #archiveteam-ot please.
20:51:20<saizai>JAA sorry
20:54:01<saizai>anywho, for your consideration. it'd be extremely useful but only in bulk
20:57:56happypotato leaves
20:59:25<mgrandi>Archivebot request: journalist might be deleting twitter https://twitter.com/ABDanielleSmith , see https://twitter.com/ABDanielleSmith/status/1348754741073874945
21:03:32<zyphlar>i think you can just go to #archivebot
21:05:15ccafe (ccafe) joins
21:05:17<@JAA>mgrandi: Done
21:05:50<mgrandi>Archivebot is spammy and I don't think anyone looks there for requests
21:06:09<zyphlar>yeah i don't request i just !ao it
21:06:23<zyphlar>saizai you might want to spearhead the project to see it through, everyone's in this project for their own reasons. maybe thuban or others can help transform it into a Warrior project. as a first step, making a wiki page for it might be good with these links
21:07:31<saizai>zyphlar I'm not familiar with how y'all work, esp. for things requiring decentralized scrape, so thought it'd be better to ask.
21:07:44<zyphlar>absolutely!
21:08:20<zyphlar>idk who's in charge of warrior projects and storage and stuff but i do know the person who cares most about this project will be you
21:08:34<saizai>how do you go about adding it to the queue for that?
21:08:49<mgrandi>In my experience, if you have a giant list of URLs, we can set up a warrior project easily
21:09:10<mgrandi>The hard part are the urls and any pitfalls (ie urls that need to be ignored)
21:09:12<zyphlar>so generating the list of URLs from the table you've got would be a first step
21:09:32<mgrandi>Is this for government sites.
21:09:33<mgrandi>?
21:10:35<saizai>zyphlar problem is that the urls can't be generated w/out interaction
21:10:55<zyphlar>are you able to interact slowly enough to get the urls in a reasonable time without ban?
21:10:58<saizai>because of the token addition and need to validate against total counts
21:11:00<zyphlar>yes mgrandi, national archives
21:11:16LordThanatos quits [Ping timeout: 244 seconds]
21:11:28<saizai>zyphlar 6 months or so at fastish rate on one box
21:12:14<saizai>so… not very
21:12:42<zyphlar>well wouldn't a first step be saving the pages returned by a search for each birth month?
21:13:09<zyphlar>and then a second pass could be drilling down into those pages' links?
21:13:13<saizai>the expected counts? yes. have done most of that into a json
21:13:51<thuban>we can do distributed projects with interaction (inspect the results of the original request and use it to make new requests) using wget-lua
21:13:52<saizai>the links w/ temp ids are temporary though and only after going through that dl interface
21:14:05<thuban>so you wouldn't need to pregenerate everything
21:14:16<saizai>I've not tried wget-lua before
21:14:51<thuban>here's an example, the lua script for out current reddit project: https://github.com/ArchiveTeam/reddit-grab/blob/master/reddit.lua
21:15:16<saizai>will take a read once not on fire
21:15:25<@arkiver>saizai: what is the size of this
21:15:51<@arkiver>rough estimate
21:15:53<saizai>arkiver est 150M records, probably 10gb?
21:16:28<@arkiver>right so size is not the problem
21:16:34<@arkiver>i didnt read all backlog
21:16:35<saizai>it'd be trivial if they didn't choke it
21:16:39<@arkiver>you say IPs are the problem?
21:16:53<saizai>yeah. and the 1k records per dl max
21:17:09<@arkiver>and this is all publicly accessible correct
21:17:16<saizai>100%
21:17:29<saizai>public records by law in fact
21:17:42<saizai>but just through that stupid search
21:17:53<@arkiver>might be nice to get
21:17:57<saizai>or the per ID iteration, I guess
21:18:17<@arkiver>you could make a wikipage and add details to that
21:18:23<saizai>though I think that scraping per ID would kill the server
21:23:52<saizai>I guess that's the other aspect I wanted to ask about: how do you do warrior w/out DDOS?
21:24:09<thuban>rate limiting by tracker
21:25:36<saizai>does the client generate URLs, and just ask tracker for something like its modulus and a rate bucket?
21:25:56LeGoupil quits [Client Quit]
21:26:21<thuban>the tracker distributes 'items', which the client uses to generate corresponding urls, according to a rate bucket
21:26:24nico_32 quits [Ping timeout: 240 seconds]
21:27:06<thuban>items are individually tracked, for requeueing etc
21:28:19<saizai>that seems hard if it's dynamic
21:28:41<thuban>i'm not sure what you mean
21:29:00<saizai>eg here I segment by year, then month, then day - depending on count in each, to reduce server load
21:29:35spirit quits [Client Quit]
21:29:48<saizai>don't get jan 1 1801 if 1801 entirely was <1k records
21:30:28<saizai>I guess the wget-lua would handle the interaction for token part
21:30:55<thuban>did you say you have the expected counts already? could use those to generate an item for each range and feed that into the tracker
21:31:29<thuban>^ the other problem would be handling the javascript interaction, since the warrior uses wget rather than selenium or similar. you (or someone) will need to do some poking in the network interaction to see which requests/params you really need
21:31:54<saizai>not all, but many yes. so an item is some sort of json object eg to give the script?
21:32:37<saizai>yeah, I tried. couldn't figure out how to get it to respond to wget/curl, even when replicating chrome headers precisely
21:32:48<saizai>so said fuckit and went with selenium
21:32:52<@arkiver>lets move this to a channel
21:32:56<@arkiver>saizai: channel name ideas?
21:33:18<saizai>numident? /shrug
21:33:19<thuban>#foiables
21:33:23<@arkiver>nice
21:33:31<atphoenix>paSSNion?
21:33:33<@arkiver>lets go #foiables
21:33:40<saizai>lulz.
21:33:53<saizai>atphoenix also cute
21:34:11<atphoenix>foibles is good too
21:34:19<atphoenix>or
21:34:33<atphoenix>NAARAD (norad)
21:34:39<@arkiver>it is #foiables
21:34:44<@arkiver>all are there
21:35:20<@kiska>Whats this project, I haven't read backlog
21:37:06<saizai>https://aad.archives.gov/aad/series-description.jsp?s=5057&cat=all&bc=sl
21:37:49<thuban>ssn info for dead people, legally public but only accessible through pita search form (ie not in bulk)
21:38:19<saizai>plus various useful demograhics, genealogy, etc
21:38:28<@kiska>SSN as a social security number?
21:38:38<saizai>in the ss5/claim records especially
21:38:42<saizai>kiska yes
21:38:47<@kiska>I see
21:39:29LordThanatos joins
21:39:48nico_32 joins
21:47:26LordThanatos quits [Ping timeout: 244 seconds]
21:48:14ave quits [Remote host closed the connection]
21:49:16Mineroboter_ quits [Read error: Connection reset by peer]
21:49:28limb (limb) joins
22:06:27Mineroboter joins
22:07:00AK quits [Client Quit]
22:07:51AK joins
22:09:38Mineroboter_ joins
22:10:41Mineroboter__ joins
22:10:51Mineroboter_ quits [Client Quit]
22:10:54Mineroboter__ quits [Remote host closed the connection]
22:11:43TheFluffyCat quits [Ping timeout: 244 seconds]
22:13:04vivid-noble quits [Ping timeout: 240 seconds]
22:20:35<purplebot>Parler edited by Esobrev (+71, Added a source for Google's removal …) just now -- https://www.archiveteam.org/?diff=46158&oldid=46153
22:20:35<purplebot>Gab created by Nintendofan885 (+371, start) just now -- https://www.archiveteam.org/?diff=46159&oldid=0
22:25:44saizai quits [Remote host closed the connection]
22:30:06AK quits [Client Quit]
22:30:16saizai joins
22:30:53AK joins
22:31:30sierra_wright joins
22:31:50<sierra_wright>hola! is the tracker down or did i screw something up
22:32:26<thuban>we have many trackers. which project?
22:32:53<sierra_wright>um i was working on urlteam 2, but https://tracker.archiveteam.org/ is timing out for me
22:33:59<sierra_wright>the warrior says "Error communicating with tracker: HTTPSConnectionPool(host='tracker.archiveteam.org', port=1338): Max retries exceeded with url: /api/get (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9f98857b70>, 'Connection to tracker.archiveteam.org timed out. (connect timeout=60)'))."
22:48:45<@arkiver>sierra_wright: we're working on it
22:51:57testing joins
22:53:13testing quits [Remote host closed the connection]
23:04:47AK quits [Client Quit]
23:06:52AK joins
23:11:13bruhmomento quits [Remote host closed the connection]
23:12:22chakrachaos joins
23:17:00ave (ave) joins
23:25:34chakrachaos quits [Remote host closed the connection]
23:30:16TheJobd quits [Ping timeout: 244 seconds]
23:35:17Arcorann (Arcorann) joins
23:43:14Arcorann_ joins
23:45:16Arcorann quits [Ping timeout: 265 seconds]
23:47:15hooway quits [Client Quit]
23:49:37Arcorann_ quits [Ping timeout: 265 seconds]

HomeSearchPrevious dayNext day