HomeSearchPrevious dayNext day

00:07:14<@arkiver>JAA: any thought on max URLs, size, time for .eu domains?
00:07:28<@arkiver>I'm thinking 100k URLs, 20 GB, 2 days
00:10:32<@arkiver>size is the WARCs size here
00:12:33<@arkiver>actually, lets use uncompressed downloaded size
00:12:40<@arkiver>in that case I'd say
00:12:45<@arkiver>150 GB?
00:18:30<@JAA>arkiver: Well, we have just under 3 days now before the UK-owned ones get frozen, so while it would be great, those numbers seem a bit high to me.
00:19:51<@arkiver>it's not that we have to reach 100k URLs before the deadline to accept the WARC
00:19:56<@arkiver>it's a maximum
00:20:05<@arkiver>or are you talking about workers?
00:20:23<@arkiver>I suspect a very large portion of the 10M domains we have are either dead, or very small in size
00:20:43<@JAA>Yeah, we might block workers with large sites that aren't going down anyway.
00:20:54<@JAA>I suspect lots of rabbit holes due to Drupal and similar stuff.
00:21:02<@arkiver>we'll see
00:21:05<@JAA>Session ID hell etc.
00:21:12<@arkiver>will use only default wget URLs extraction
00:21:30<@arkiver>also ping HCross kiska we also need targets here, might be a big one
00:22:29<@arkiver>almost have a project ready for .eu
00:22:32<@arkiver>domains-grab
00:22:47<@arkiver>and only URLs from .eu will be archived (so no external non-.eu domains)
00:24:26<@JAA>But links to other .eu domains are followed?
00:24:46<@arkiver>page requisites
00:25:00<@arkiver>only
00:25:00<@JAA>Ok, sounds good.
00:25:05<@arkiver>alright
00:25:10<@arkiver>testing now
00:31:21hooway quits [Client Quit]
00:43:03fuzzy802 is now known as fuzzy8021
00:43:20fuzzy8021 quits [Changing host]
00:43:20fuzzy8021 (fuzzy8021) joins
00:45:22<purplebot>File:.eu-logo.png uploaded by Arkiver (+0) 1 minute ago -- https://www.archiveteam.org/?diff=46006&oldid=0
00:51:16<Larsenv>Ryz, JAA, katocala, flashfire42: I wish AB was an actual job I would get paid for
00:51:59<flashfire42>Bro fucking same
00:53:08<@JAA>Same
00:54:10<Ryz>Mmm >#<;
01:13:07<@arkiver>JAA: have to use an ugly workaround, so FYI there will be no logs in the WARCs for the .eu domains project
01:13:57<@arkiver>because wget.actions.ABORT prevents wget.callbacks.before_exit from running
01:14:11<@arkiver>need to fix that in Wget-AT, but there's not enough time for that now.
01:14:24<@JAA>:-(
01:15:52<@JAA>arkiver: When is download_child_p called, on adding to the URL queue or before retrieval?
01:16:31<@JAA>I'm thinking whether it's possible to set a flag and ignore all URLs instead of using ABORT.
01:18:10<@arkiver>download_child_p is just about accepting to rejecting URLs wget extracted
01:18:24<@arkiver>so yes
01:18:31<@arkiver>we could stop queueing new URLs at the maximum
01:18:32<@arkiver>but
01:18:43<@arkiver>we won't be able to use that to stop at a certain number of bytes or seconds
01:18:46<@JAA>Right, so it's called on queueing, not on retrieving.
01:18:51<@arkiver>yeah
01:19:08<@arkiver>I'll see about keeping the log some other way
01:20:34<@JAA>Write wget.log to a resource record from Python maybe?
01:23:33<@arkiver>yep for example
01:34:42<@arkiver>JAA: going to write manually as -tail.warc.gz
01:35:32<@arkiver>'manually' with warcio
02:03:43Dallas (Dallas) joins
02:16:02<@arkiver>that's in
02:17:52<@JAA>Re MAGFest: Ew, Confluence. :-(
02:24:39<OrIdow6>arkiver: Change for SmackJeeves (skip certain errors) is ready. Can you give the team write access to the repo? (Or, if the current state is deliberate, I can make a PR)
02:25:23<@arkiver>right
02:25:41<OrIdow6>Changes are currently up on OrIdow6/smackjeeves-grab in any case
02:25:59<Arcorann>https://friendsofmag.com/ <-- re: MAGFest, I posted this in -ot, there's a link to a Google drive that should probably be archived as well
02:26:02<@arkiver>OrIdow6: write access is setr
02:26:03<@arkiver>set*
02:31:24<OrIdow6>Ok, thank you
02:31:24<@arkiver>i see its merged
02:31:29<@arkiver>bumping version
02:32:00<@arkiver>new minimum for smackjeeves is 20201229.01
02:32:54<OrIdow6>EggplantN_: ^ Should be ready to rerun failed items
02:41:01<@arkiver>Fusl: builds seem to be failing due to wasabi problem at the moment (see the new builds for grab-base-df and smackjeeves-grab)
02:41:09<@arkiver>is there some way we can fix this while wasabi is down?
02:43:02<@arkiver>kiska: can you start a docker build for domains-grab?
02:43:16<@arkiver>we'll use that for the .eu project
02:45:32<OrIdow6>Looks like this is all .eu domains whatsoever at present?
02:45:37<OrIdow6>*Sounds like
02:46:06<@arkiver>i believe we found some ~10 million
02:46:21<@arkiver>which should be a quite big chunk
02:46:42<@JAA>What will we do about HTTP/HTTPS and www/no-www?
02:46:55<OrIdow6>If it gets to big for IA, the WHOIS thing should be able to reduce it by 1 or 2 OOM
02:46:57<@arkiver>queue with both http and https
02:47:10<@arkiver>we won't add www
02:47:19<@arkiver>seems like that's already in the dumps i have here
02:47:32<@JAA>I know my dump doesn't have www.
02:47:34<@arkiver>numbers of domains in 2006 https://en.wikipedia.org/wiki/.eu#Parking_and_redirects
02:47:40<@arkiver>JAA: ah
02:47:46<@arkiver>will queue with www. as well
02:47:47<@arkiver>why not
02:47:49<@Kaz>arkiver: fyi - that change you made to grab-base
02:47:52<@Kaz>currently cannot build
02:47:59<@Kaz>oh you said that above, nvm
02:48:01<@arkiver>Kaz: yeah, I pinged fus l about it
02:48:02<@Kaz>I just got the alert
02:48:21<@arkiver>yeah wasabi is down pretty hard
02:48:30<@Kaz>we can fix temporarily by hosting the wgat-at binary elsewhere for now
02:48:35<@Kaz>if someone has the latest ver..
02:48:44<@Kaz>it's not pretty though
02:49:20<@JAA>Alert?
02:49:32<@Kaz>I got a dockerhub email alert for a failed build
02:49:54<@JAA>Oh, grab-base is still on Docker Hub, not on atdr?
02:50:04<@Kaz>well, it's on both
02:50:15<@Kaz>but dockerhub emails me, drone doesn't
02:50:22<@JAA>Right
02:50:35<@JAA>I was just thinking about how we can get these alerts here on IRC.
02:50:42<@Kaz>it doesn't _need_ to be on dockerhub these days
02:50:54<@Kaz>https://readme.drone.io/webhooks/overview/
02:51:10<@JAA>Yes
02:51:14<@JAA>Can integrate that easily into http2irc.
02:51:53<@JAA>Docker Hub has hooks as well, but meh.
02:54:30<Ryz>Help? Is there any way all of this can be archived? https://drive.google.com/drive/folders/1EcGeSJzAlewzG84z1QXb8Hw_aU4i5AvH
02:54:34<Ryz>Came from https://friendsofmag.com/
02:54:40<Ryz>Unsure if AB can be able to take it
02:55:11<Ryz>I'm not sure when the corrupt people at Magfest can just DMCA it for technicality reasons :/
03:00:04mutantmonkey quits [Ping timeout: 240 seconds]
03:08:21<@arkiver>domains-grab project (.eu domains) is ready, waiting for a target
03:09:42<@Kaz>channel?
03:09:44<@Kaz>I can spin a target
03:11:11<@arkiver>Kaz: no channel currently
03:11:15<@arkiver>we could think of one
03:11:33<@Kaz>ok
03:11:48<@Kaz>using these vars unless any complaints:
03:11:51<@Kaz>IA_ITEM_TITLE="Archive Team Domains:"
03:11:53<@Kaz>IA_ITEM_PREFIX="archiveteam_domains_"
03:11:55<@Kaz>FILE_PREFIX="domains_"
03:12:19<@arkiver>for title, can we do "Archive Team .eu Domains"?
03:12:26<@Kaz>yep
03:12:40<@Kaz>I'll m ake prefixes eudomains rather than just domains, to match
03:12:48mutantmonkey (mutantmonkey) joins
03:13:43<@arkiver>for the others you can keep just _domains_ (like it is now)
03:15:20<@Kaz>gaaaaaaah
03:15:22<@Kaz>grr
03:16:08<@Kaz>ok done
03:16:20<@Kaz>target is configured and added to tracker
03:16:35<@arkiver>awesome, thank you!
03:25:03<atphoenix>.eu domains-grab -> domainrot/eurid eur-out youreout/domainbrexit/eurouttahere/noteurdomain
03:25:48<atphoenix>not-eur-domain = not your domain
03:26:11<OrIdow6>Ryz: I've never seen anything to automatically save Google Drive folders
03:26:17<OrIdow6>As opposed to individual files
03:27:43<@Kaz>i like noteurdomain
03:28:01<@Kaz>(i have joined #noteurdomain)
03:31:48<@Kaz>no idea if this project has a wiki page but please update if necessary
03:31:52<@Kaz>I couldn't find it from a quick look
03:33:06<@arkiver>no wiki page yet
03:33:11<@arkiver>project is started!
03:33:15<@arkiver>all items queued
03:46:04nerdguy1138 quits [Ping timeout: 240 seconds]
03:51:51<fuzzy8021>can a docker image get built please
03:52:05<@JAA>Not until Wasabi gets their shit together.
03:52:17<fuzzy8021>ah k
03:52:46<@JAA>Apparently their domain got blocked because of malicious content.
03:56:26nerdguy1138 (nerdguy1138) joins
04:06:55<@kiska>Kaz: target still required for domains?
04:07:02<@Kaz>nah
04:07:06<@Kaz>also, #noteurdomain
04:08:48@kiskaWee1 is now known as @kiskaWeebChat
04:09:37thuban quits [Client Quit]
04:10:55thuban joins
04:39:00qw3rty_ joins
04:42:24qw3rty quits [Ping timeout: 240 seconds]
05:08:21etnguyen03 quits [Client Quit]
05:39:12Ryz quits [Remote host closed the connection]
05:39:55Ryz (Ryz) joins
06:11:40nerdguy1138 quits [Remote host closed the connection]
06:20:00nerdguy1138 (nerdguy1138) joins
06:28:19Terbium joins
06:44:46Mateon1 quits [Remote host closed the connection]
06:44:49Mateon1 joins
06:45:40Matthww6 joins
06:46:43Matthww quits [Ping timeout: 240 seconds]
06:46:44Matthww6 is now known as Matthww
06:47:55lukash7 joins
06:53:06<brad>So, I've discovered that my own private little mirror for https://community.fantasyflightgames.com/ has managed to download ~60GB worth of stuff so far, and it's still going on that one single wget thread. So, I'm curious to know how I might be able to share what I've downloaded, in case it is of use to anyone else in trying to put together a complete archive of the site?
06:59:03Matthww5 joins
07:00:03Matthww quits [Ping timeout: 240 seconds]
07:00:04Matthww5 is now known as Matthww
08:27:48<HP_Archivist>Can I crawl a Vimeo user's page and have the video captured or do I have to submit each one individually? https://vimeo.com/user29920803
08:28:37<atphoenix>Matthww, here's your chance to get a headstart on EggplantN_ . See #noteurdomain docker image this time is at warcforceone/domains-grab
08:29:37<atphoenix>HP_Archivist idk. hopefully someone else knows
08:30:33<HP_Archivist>atphoenix, hm alright
08:33:51<jodizzle>brad: You could consider uploading what you download as an archive.org item
08:35:24<brad>@jodizzle: I would be happy to do that, but I’m not sure how that process works. You’re creating a “WARC” file, right? So do you have a link to a wiki article on how to create one of those things manually?
08:38:17<brad>And since the source isn’t shutting down until something like February 1st, is there a good way to do another run to pick up just the changes since the last run?
08:48:36<jodizzle>brad: So saving WARCs would probably be ideal, but it's not a requirement for uploading to archive.org. People upload data in all sorts of different formats.
08:50:22<jodizzle>A WARC can't (and should never) be created "manually", in the sense of manufacturing a WARC's contents. The point of WARCs is to accurately record as much of the web traffic between client and server (request and response) as possible.
08:51:04<jodizzle>That unfortunately means that if your wget run isn't saving WARCs already, you would probably need to restart.
08:51:38<jodizzle>Here's a wiki page about WARCs: https://archiveteam.org/index.php?title=The_WARC_Ecosystem. There are probably several others on the wiki.
08:55:45<brad>Oh.
09:02:04<brad>Okay, so I’m going to have to restart.
09:02:48<jodizzle>If you want WARCs, yes.
09:03:06<brad>Is there an obvious way to do wget in a multi-threaded way, so that I can speed up the restart?
09:03:55<OrIdow6>In any case your data will probably not be getting into Wayback Machine
09:04:38<jodizzle>Yes, also that, if that's a concern
09:05:53<brad>Ahh. Found http://stackoverflow.com/questions/7577615/ddg#11850469 but that assumes that I already have a list of URLs to hit. Which I wouldn’t have until the crawl is complete.
09:06:15<jodizzle>brad: I don't know a ton about wget, but I think base wget can be limited and hard to work with. You could look through the other tools on that wiki page I linked.
09:06:32<jodizzle>grab-site is a pretty commonly used tool
09:07:20<OrIdow6>Yeah
09:15:50<brad>Ahh. I had started looking at wpull, but had not followed the next logical step of looking at what might build on top of it. It’s late tonight, so I’ll pick this back up tomorrow. But I will start with grab-site. Thanks!
09:36:11hooway joins
10:53:01<Matthww>Are the left over items for smackjeeves still going to be requeed?
10:53:41<Matthww>I mean they are already in the queue if I see it correctly just the tracker not giving them out anymore...
10:54:51Ryz quits [Remote host closed the connection]
10:55:32Ryz (Ryz) joins
11:12:58HackMii quits [Remote host closed the connection]
11:13:20HackMii (hacktheplanet) joins
11:30:55<OrIdow6>Yes, they were requeued
11:31:02<OrIdow6>Rate limit is at 0
11:32:07<OrIdow6>Should be started back soon (I'm not the person controlling that) - but you will need the new version
11:32:34<Matthww>Yep I already updated.
11:32:56<Matthww>Also Idk who is the lead for the domains-grab but I'm right now doing a site with ALOT of referral codes.
11:33:10<OrIdow6>#noteurdomain
11:33:33<Matthww>Thanks
12:09:40spirit joins
12:21:10<EggplantN_>Matthww I even removed the limit
12:21:18<EggplantN_>You should be beating me at this
12:21:19<EggplantN_>anyway
12:21:22<EggplantN_>I enabled it
12:21:38<EggplantN_>sorry arkive r forgot to remove the limit when he upped min version
12:24:35<OrIdow6>As before, please tell me if any fail
12:25:32<OrIdow6>Glad that it looks like this one will get finished
12:26:00<EggplantN_>will do Orldow6 thank you!
12:27:30<OrIdow6>Yep - and thank you for restarting it
12:38:24<EggplantN_>looking pretty good orldow6 not sure there will be many items left if any
12:41:24ragu_ joins
12:45:12ragu quits [Ping timeout: 265 seconds]
12:45:59ragu__ joins
12:49:24ragu_ quits [Ping timeout: 240 seconds]
13:05:57<OrIdow6>Good
13:33:44<@arkiver>right
13:44:23<EggplantN_>Orldow6 were done will requeue and weed out the errors just at the brewery's
14:16:04Arcorann quits [Ping timeout: 265 seconds]
15:00:27<EggplantN_>Orldow6 you around?
15:01:03<EggplantN_>https://www.irccloud.com/pastebin/WOoQvfSM/Error1
15:02:27<EggplantN_>similar items comic:300011 comic:300029 comic:300007
15:02:42<EggplantN_>or dunno if you are able to handle arkiver ^
15:02:48<EggplantN_>just so we can close this project up :P
15:09:22<@arkiver>can check later, but OrIdow6 is more into this one
15:11:31<EggplantN_>understood
15:11:35<EggplantN_>only 70 items left it seems
15:26:55ragu_ joins
15:29:44ragu__ quits [Ping timeout: 240 seconds]
16:07:04honno joins
16:51:22<purplebot>Yahoo! Groups edited by Betamax (+0, Forgot to mark Yahoo! Groups as …) just now -- https://www.archiveteam.org/?diff=46007&oldid=45933
17:12:13<@arkiver>HCross: #noteurdomain
17:23:35systwi_ (systwi) joins
17:25:04systwi quits [Ping timeout: 240 seconds]
17:35:45honno quits [Remote host closed the connection]
17:59:40lukash7 quits [Client Quit]
18:02:07lukash7 joins
18:07:35Matthww quits [Ping timeout: 265 seconds]
18:11:23Matthww joins
18:22:34spirit quits [Read error: Connection reset by peer]
18:36:11VADemon joins
18:50:38<@arkiver>oh hey VADemon
19:40:02sembiance quits [Remote host closed the connection]
20:13:19Nick55 joins
20:13:56Nick55 quits [Client Quit]
20:28:02<Sanqui>how 2 archive wikis on https://w.atwiki.jp
21:40:41xit quits [Client Quit]
21:41:49xit joins
21:46:20<EggplantN_>anyway Orldow6 lmk the update
21:46:32<EggplantN_>all items that are left are purely that error above
21:56:04tzt quits [Ping timeout: 240 seconds]
22:08:36Arcorann (Arcorann) joins
22:19:41godane (godane) joins
22:35:33nerdguy1138 quits [Remote host closed the connection]
22:36:39nerdguy1138 (nerdguy1138) joins
23:11:21<brad>Hmm. So, I'm trying to install grab-site according to the instructions at https://github.com/ArchiveTeam/grab-site#install-on-macos but I've run into an error:
23:12:21<brad>Error message when I try to install grab-site https://www.irccloud.com/pastebin/moEkTnwa/grab-site.err
23:39:40<brad>I have filed an issue on this at https://github.com/ArchiveTeam/grab-site/issues/174

HomeSearchPrevious dayNext day