HomeSearchPrevious dayNext day

00:08:02hooway quits [Client Quit]
00:10:23qw3rty_ quits [Ping timeout: 265 seconds]
00:14:08<@arkiver>so-net project coming up (credits to OrIdow6^2 )
00:16:11Arcorann (Arcorann) joins
00:22:19godane1 joins
00:23:07qw3rty joins
00:23:22godane quits [Read error: Connection reset by peer]
00:43:24qw3rty quits [Ping timeout: 240 seconds]
00:54:57sliccricc quits [Remote host closed the connection]
00:57:49sliccricc (sliccricc) joins
02:10:14<EggplantN>Excellent :D
02:33:39<Jake>Do we have a channel for that?
02:39:44@dxrt quits [Ping timeout: 265 seconds]
02:41:49<ALTROSKY_mystique>arkiver: what's so-net about?
02:47:25<@JAA>Jake: No(t yet?).
02:47:29<@JAA>ALTROSKY_mystique: See Deathwatch
03:22:53<OrIdow6^2>If all goes as expected, this will be simple enough as to not warrant a channel
03:26:46<Jake>Excellent :) Just wanted to make sure I wasn't missing one
03:42:00<purplebot>File:So-net-logo.png uploaded by Arkiver (+0) just now -- https://www.archiveteam.org/?diff=46236&oldid=0
03:59:12<@arkiver>EggplantN: kiska: can one of you please setup a target for
03:59:39<@kiska>For?
03:59:43<@kiska>Bar?
03:59:46<@kiska>Var*
03:59:53<@arkiver>archiveteam_sonet-upageplus_
03:59:53Eighty quits [Remote host closed the connection]
04:00:02<@kiska>Lemme get out of bed
04:00:03<@arkiver>_sonet-upageplus
04:00:26<@arkiver>Archive Team So-Net U-Page+:
04:00:32<@arkiver>kiska: thank you!
04:00:34<@JAA>sonet-upageplus_ ?
04:01:18<@JAA>(That's the file prefix I assume?)
04:01:50<@arkiver>i'm open to suggestions
04:02:06<@JAA>I'm just confused by your putting the underscore as the first char.
04:02:09<@kiska>rsync://rsync.hel1.kiska.pw/sonet/:downloader/
04:02:27<@arkiver>ah yeah its a prefix
04:02:31<@arkiver>for the WARC filename
04:03:30<@arkiver>kiska: awesome!
04:04:24<@kiska>If you overwhelm the target I'll go and get a CPX31
04:06:27Gaelan quits [Quit: ZNC 1.8.1 - https://znc.in]
04:06:45<@arkiver>this is really not large at all
04:07:24Eighty (Eighty) joins
04:08:58Gaelan (Gaelan) joins
04:11:30<@arkiver>OrIdow6^2: could you please briefly look over the changes i made to the Lua?
04:11:38<@arkiver>should be good
04:11:46<@arkiver>if all looks good will queue items and start project
04:13:15<OrIdow6^2>Looking
04:17:20<Jake>(I got only got 404s on my first few jobs, I assume that's normal?)
04:21:06<OrIdow6^2>arkiver: yeah, looks good
04:26:11<OrIdow6^2>Hm, though it looks like if it gets another So-Net index page as an outlink, httploop_result will set item_host and item_user_dir accordingly - this is unlikely (I can see it as happening due to a typo) but I may put in a PR anyway arkiver
04:26:51<OrIdow6^2>Jake: Yes, most(?) of the pages that once existed have gone away (I think due to people not paying their fees)
04:27:09<Jake>Ah, makes sense. Just wanted to be sure there wasn't some issue!
04:31:18<@arkiver>OrIdow6^2: good point on outlink on that
04:31:32<@arkiver>will fix now
04:34:28<@arkiver>OrIdow6^2: fixed, what do you think?
04:34:48<@arkiver>also added a note multi item size should stay 1, else this fix will not work
04:38:50<OrIdow6^2>Glad to help
04:38:52<OrIdow6^2>Looking
04:39:59<@arkiver>items loaded
04:40:11<@arkiver>if you give the go we'll start granting items
04:40:20<OrIdow6^2>Looks good
04:40:49<Jake>what concurrency is recommended?
04:40:57<@arkiver>started!
04:41:04<@arkiver>i'm off now for the night
04:41:08<@arkiver>(5.40 AM)
04:41:30<Jake>have a good night!
04:41:32<@arkiver>if something turns out to be not going well we'll requeue items after a fix, this is small anyway
04:41:35<@arkiver>Jake: thanks
04:42:01<OrIdow6^2>I suppose this isn't really a "multi-item" setup anymore, but oh well, not a big project
04:42:10<OrIdow6^2>Should get done anyhow
04:42:25<OrIdow6^2>Sun rises & sets at strange times in the Netherlands, evidently
04:42:31<@arkiver>OrIdow6^2: yeah, but I want to make the multi items pipeline.py the new standard
04:42:51<OrIdow6^2>Oh, that's what I was sort of suspecting
04:43:00<@arkiver>still requires some changes though, but will make this in upcoming projects
04:43:13<@arkiver>sun is absolutely weird here :)
04:43:23<@arkiver>OrIdow6^2: where are you again?
04:43:29<OrIdow6^2>California
04:44:00<@arkiver>ah nice!
04:44:12<@arkiver>good day to you :)
04:44:25<OrIdow6^2>Good day/night
04:45:17<OrIdow6^2>:)
04:49:15dxrt joins
04:49:17dxrt quits [Changing host]
04:49:17dxrt (dxrt) joins
04:49:17@ChanServ sets mode: +o dxrt
04:56:31BlueMaxima quits [Client Quit]
05:01:06Mineroboter joins
05:01:37Mineroboter quits [Client Quit]
05:09:48<fuzzy8021>anyone around to enable so-net docker image?
05:12:37<Craigle>kiska You available? ^
05:14:56<OrIdow6^2>By the way, please don't suddenly go crazy on So-Net, but instead gradually escalate - I think it's still just about 20 Apache servers
05:15:19<OrIdow6^2>"Still" as it was in 2005 or so
05:16:02Krownest quits [Quit: Leaving]
05:16:27<OrIdow6^2>Looks like a surprisingly high amount is coming out of backfeed
05:16:39<OrIdow6^2>Enough that queued is still going up
05:18:14<Craigle>It looks like it's holding up ok. I see a handful of CONERROR and 403's, but mostly everything looks like it's running smoothly
05:19:25<OrIdow6^2>Copious errors are expected on outlinks
05:20:26<OrIdow6^2>As long as it's not upp.so-net.ne.jp having problems, it's good
05:22:33<OrIdow6^2>Out of curiosity, if any of you have easy access to your logs, what is being discovered (lines start with "found item")? Even if it's all garbage, the script and site can handle it, just interested
05:23:32<Jake>found item userdir:lib003/cgi-bin
05:23:32<Jake>found item userdir:www003/-
05:23:32<Jake>found item userdir:www003/eMzf
05:24:27<Jake>https://pastebin.com/raw/SHtNnNiS
05:25:08<Craigle>Nothing I see giving errors right now is so-net. Looks like outlinks
05:25:31<OrIdow6^2>Yeah, something tells me most of these aren't valid users :P
05:25:45<Jake>https://pastebin.com/raw/YjtccVJT
05:25:54<Jake>Yup, most don't look like valid users but 🤷
05:26:11<Jake>(And I agree, mostly outlinks throwing errors, so-net seems fine)
05:26:21<OrIdow6^2>Good
05:32:00<purplebot>So-net U-Page+ created by JustAnotherArchivist (+554, Created page with "{{Infobox project …) just now -- https://www.archiveteam.org/?diff=46237&oldid=0
05:33:00<purplebot>Deathwatch edited by JustAnotherArchivist (-41, /* 2021 */ Link So-net U-Page+ …) just now -- https://www.archiveteam.org/?diff=46238&oldid=46214
05:45:25taka joins
05:46:17HP_Archivist quits [Quit: Leaving]
05:49:59<OrIdow6^2>taka: So-Net is running - see https://tracker.archiveteam.org/so-net-u-page-plus/
05:54:27<thuban>i take it we're not going to need a load of workers on this?
05:56:24<taka>OrIdow6^2 Thanks! But...
05:56:31<taka>I found a service called "Web Space" operated by So-net, which is being discontinued at the same time as So-net U Page+.This stopped accepting new applications a long time ago, but you can see some of them by doing a Google search for "site:domain".There seem to be two types of domains, ****.gate01.com/** and ****.gyao.ne.jp/***.I had mistakenly
05:56:32<taka>thought that it was the same service as U Page+, but it seems to be a different service.
05:57:17<taka>https://www.so-net.ne.jp/info/2020/op20200908_0039.html "WebSpace" shutdown announce
05:57:31<OrIdow6^2>thuban: Doesn't look like it (but see above - will update if this changes)
05:58:19<taka>https://www.so-net.ne.jp/option/gyao/webspace/ service detail
06:07:05jhg quits [Ping timeout: 265 seconds]
06:13:54<OrIdow6^2>taka: Thank you for telling us about this. This should be straightforward to add to the program that is already running. Come back in 10 hours or so and hopefully we will have it added.
06:14:22<OrIdow6^2>Do you know if there is a list of all sites?
06:16:58<taka>Probably not. However, the number of remaining pages seems to be much smaller than that of U Page+.
06:19:05<OrIdow6^2>Yes.
06:19:28<atphoenix>OrIdow6^2, there isn't a project channel for So-net is there? E.g. sonnet? sayonarasonnet? sayonaranet?
06:19:41<OrIdow6^2>No
06:20:49<OrIdow6^2>Feel free if you want to make one, I will keep talking to ark iver here until he wakes up, though
06:22:19<atphoenix>I've opened all three of my suggestions as channels in case one of them is of interest for use
06:31:28<OrIdow6^2>I[ve gone to #sayonaranet
06:34:20<atphoenix>I've done all the standard channel setup steps so AT ops should be granted to the regular ops.
06:58:14<OrIdow6^2>arkiver: Here are my changes to support the other domains/nominally-separate-but-really-identical-product: https://github.com/OrIdow6/so-net-u-page-plus-grab/commit/fe52429ad0faeda537166c218b4b9b16efcd123a
07:03:50godane2 joins
07:04:31<OrIdow6^2>I was thinking that if the 404s become overwhelming, multi-items could be supported by rejecting outlinks to the domains that trigger item switching, and just relying on backfeed to get whatever's there
07:06:32godane1 quits [Ping timeout: 265 seconds]
07:27:26<@kiska>Craigle: Building
07:36:00<purplebot>Yahoo! Groups edited by Switchnode (+869, the big it's-dead edit) just now -- https://www.archiveteam.org/?diff=46239&oldid=46007
07:54:43qw3rty joins
07:57:00Craigle0 (Craigle) joins
07:57:01Craigle quits [Ping timeout: 252 seconds]
07:57:01Craigle0 is now known as Craigle
08:05:24lukash7 quits [Ping timeout: 240 seconds]
08:05:40lukash7 joins
08:11:16spirit joins
08:30:00<purplebot>Yahoo! Groups/Archiving Project FAQ for group members edited by Switchnode (+564, it's dead 2: electric FAQ) just now -- https://www.archiveteam.org/?diff=46240&oldid=43595
08:36:16hooway joins
08:57:17<thuban>how are we on 9lives (jan 31) and the fantasy flight games forum (feb 1)?
08:57:24<@HCross>Do we need to start backing up The Netherlands?
08:59:00<thuban>^ per brad and jodizzle's conversation of 16 jan the latter should be finished (between https://archive.fart.website/archivebot/viewer/job/5l4qk and https://archive.fart.website/archivebot/viewer/job/erulq)
08:59:37<thuban>i see a couple of finished jobs for 9lives.be (https://archive.fart.website/archivebot/viewer/domain/www.9lives.be); does that cover everything?
08:59:56<jodizzle>The 9lives.be job is still running, and is almost done
09:01:50<thuban>ah, good to know--didn't realize split warcs appeared on the viewer site before their job finished
09:02:30<jodizzle>I suppose there is a question of whether we want to try and get posts that were made after the job started, since the job has been running for a while and the newest posts may have been skipped over
09:02:57<jodizzle>But I don't really know the best way to do that either.
09:03:11<jodizzle>Same story with fantasy flight games
09:03:51<thuban>possibly, yes--brad suggested it would be "a game of catch-up"
09:04:54<thuban>someone who works with archivebot more pls comment, thanks, goodnight
09:05:02<Sanqui>we can run another job with --no-offsite
09:05:23<Sanqui>it should be able to more-or-less catch up on the latest stuff
09:06:06<Sanqui>no guarantees made but that's the nature of the game
09:06:21<jodizzle>Yeah, that would be the lowest-effort solution, just run another job until the content disappears
09:06:46<thuban>sounds good (tho it will probably need the same large delays as the original one)
09:07:04Arcorann quits [Ping timeout: 240 seconds]
09:07:20<jodizzle>Yeah, there are some 403s from that that I will ideally clean up
09:07:21<Sanqui>if you think about the breadth-first approach, it's going to reach some links leading to latest posts, so they'll be queued up in the first few rows.
09:09:18<brad>Much appreciated, folks!
09:09:39<thuban>Sanqui: hm, yeah. does ab have a maxdepth parameter, so that we could keep running relatively small jobs that would still get the newest stuff? docs suggest no but that's a bit surprising
09:09:57<Sanqui>I've never seen the maxdepth parameter in use
09:10:23<Sanqui>oh, that's because there is none, yeah
09:10:42<brad>I know one copy of an archive for the community.fantasyflightgames.com has already gone up at another site, but there’s no telling how complete or accurate it is, so the more complete we can make this one, the better.
09:10:50<Sanqui>we can always abort, but I don't think there's a real point in doing more than one more of these
09:11:15<Sanqui>but in general, we have a bit of spare job space atm, so it can't hurt.
09:11:36<thuban>i guess we could always do a ~manual scrape of the various front pages and grab urls from the results as !ao
09:11:51<thuban>will look at it later if no one else does
09:11:54<brad>I have pointed people at the WARCs that have already been created by the ArchiveBot, but I haven’t heard where their effort is on getting a copy of those up and running on an actual community-owned site.
09:12:13<thuban>(zzz for real this time)
09:12:26<brad>Me too! It’s 3:12am....
09:12:37<Sanqui>brad: a little bird told me avoozl is working on tools for scraping forum databases from warcs... ;)
09:12:59<Sanqui>and, good morning from europe, go to bed!
09:13:02<brad>Sanqui: Yay!
09:13:40<brad>I used to live in Belgium for almost eight years, and moved back to the US in 2006. My body clock still hasn’t quite caught up.
09:14:00<Sanqui>permanent jet lag :D
09:14:24<brad>I’ve always been a night owl, but it seems to just get worse as I get older.
09:14:32<brad>Anyway, thanks again and good night, all!
09:14:41<Sanqui>nini and thanks for working on preservation!
09:33:41<avoozl>I have some very basic things running now, but the io load on search is quite heavy
09:42:57Arcorann (Arcorann) joins
10:14:23nimaje joins
10:18:41LeGoupil joins
10:19:32LeGoupil quits [Client Quit]
10:43:07aleph quits [Ping timeout: 252 seconds]
10:47:57hexa- quits [Ping timeout: 600 seconds]
10:51:28hexa- (hexa-) joins
11:17:25aleph joins
11:21:44aleph quits [Ping timeout: 240 seconds]
12:46:35aleph joins
13:06:06LeighR joins
13:10:46<@arkiver>OrIdow6^2: right, it'll not extract items anymore from the already archived items
13:10:54<@arkiver>are you running CDX scans for the other two?
13:11:30<@arkiver>Jake: OrIdow6^2: on items like cgi-bin, -, etc. those are pretty generic and likely often discovered, but they are only queued once through backfeed
13:11:53<@arkiver>if I look at the items going by on the tracker they look reasonable to me
13:11:58<Jake>Ah, so once they are found once they should only be put through backfeed once, instead of constantly being submitted?
13:13:15<@arkiver>they are resubmitted yes
13:13:21<@arkiver>but backfeed does deduplication
13:13:33<@arkiver>well based on bloom filter with 1/million false positive rate
13:13:38<Jake>ah, that does make sense, should be fine then!
13:14:20<OrIdow6^2>arkiver: I downloaded the CDX, still need to process it, but I will do that tomorrow (/ in 12 hours or so) if no one else gets to it (it's straightforward enough, and the commands I used previously are in a commit message in -items)
13:16:14<OrIdow6^2>The queue is still growing
13:16:36<OrIdow6^2>ANyhow channel that has largely been settled on for this is #sayonaranet
13:20:44LeighR leaves
14:23:59isthisthedarkweb quits [Remote host closed the connection]
14:37:24Arcorann quits [Ping timeout: 240 seconds]
14:51:30rsn_ quits [Ping timeout: 265 seconds]
14:53:59rsn joins
15:10:27hexa- quits [Ping timeout: 600 seconds]
15:19:33hexa- (hexa-) joins
15:35:08isthisthedarkweb (isthisthedarkweb) joins
16:12:26LeighR joins
16:35:29britmob23 joins
16:35:44britmob2 quits [Ping timeout: 240 seconds]
16:35:45britmob23 is now known as britmob2
16:36:52britmob quits [Remote host closed the connection]
16:37:11britmob joins
16:54:13LeGoupil joins
17:50:20godane1 joins
17:53:14godane2 quits [Ping timeout: 265 seconds]
18:04:24sliccricc quits [Ping timeout: 240 seconds]
18:14:23LeighR quits [Ping timeout: 244 seconds]
18:55:59<purplebot>Webzdarma edited by Sanqui (-63, /* ArchiveBot jobs */ webzdarma_subdomainfinder_04 …) just now -- https://www.archiveteam.org/?diff=46241&oldid=46218
19:32:03DogsRNice (Webuser299) joins
19:34:51<Ryz>Hmm, the consideration of archiving radio stations (well, there can be websites; unsure about archiving audio): https://twitter.com/LifeLogik/status/1352260681941151752
19:40:52<EggplantN>arkiver ^
19:41:30<@JAA>#radio-archive
20:11:01<avoozl>JAA, Sanqui: some progress... https://imgur.com/a/MM2O86w hope to have something to live demo soon
20:11:31<@JAA>Nice :-)
20:18:34<@JAA>My CNET forums grab ran a bit faster than expected as the response times were significantly better for a few hours (daily cycle of visitors?). ETA is now 3-5 hours.
20:28:18spirit quits [Client Quit]
20:33:21<Sanqui>I'm grabbing https://www.flashflashrevolution.com/vbz/ without offsite links right now
20:33:29<Sanqui>should be a goood testbed too when it's done
20:33:55<Sanqui>ETA is maybe a week
20:34:44Merzius joins
20:37:16etnguyen03 (etnguyen03) joins
21:07:11spirit joins
21:12:21LeGoupil quits [Client Quit]
21:13:33spirit quits [Client Quit]
21:15:00<purplebot>Deathwatch edited by JustAnotherArchivist (+136, /* 2021 */ Add StoryFire) just now -- https://www.archiveteam.org/?diff=46242&oldid=46238
21:15:00<purplebot>Discourse edited by Themadprogramer (+49, /* Active Discourses */) just now -- https://www.archiveteam.org/?diff=46243&oldid=46003
21:24:53prebala-publici joins
21:32:45<EggplantN>Storyfire looks a fun one
21:33:03<EggplantN>With it being video we will need to act fast to get it done
21:34:06<@JAA>I only clicked around briefly but saw video content being loaded from Vimeo on my sample size of 2. So maybe we can get away with not saving the actual videos until later.
22:13:38<@jrwr>https://twitter.com/silascutler/status/1354166757729202177
22:13:47<@jrwr>someone is looking to track down archives of a few twitter accounts
22:13:54<@jrwr>do you know if we have coverage of these at all
22:15:35Merzius leaves
22:19:35prebala-publici quits [Remote host closed the connection]
22:39:19<benjins>jrwr: I believe those accounts were all suspended before Google published their story, so it's pretty unlikely
22:39:50<@jrwr>JAA: Mind looking under the rug to see if it got archived at all?
22:48:58Merzius joins
22:57:34<@arkiver>checking storyfire
22:58:10<@arkiver>Sanqui: what is that?
22:58:22<Sanqui>https://archiveteam.org/index.php?title=Webzdarma
22:58:35<Sanqui>Czech freehost. I've done discovery & been going over it over the past year
22:58:40<Sanqui>using archivebot
22:58:45<Sanqui>but I need to move faster now
22:59:05<Sanqui>For starters, I think --no-offsite is a good idea
22:59:32nothere quits [Quit: Leaving]
22:59:55<Sanqui>I'm running 4 jobs concurrently at the moment, but it goes slowly because of loads of spam and offsite links in general
23:00:13<Sanqui>tips to speed it up welcome
23:01:08Arcorann (Arcorann) joins
23:02:26<Sanqui>Gonna start the next batch with --no-offsite and see where it takes us, I suppose
23:05:55nothere joins
23:06:00<purplebot>Webzdarma edited by Sanqui (+38, starting webzdarma_subdomainfinder_05) just now -- https://www.archiveteam.org/?diff=46244&oldid=46241
23:07:42BlueMaxima joins
23:20:21mutantmonkey quits [Remote host closed the connection]
23:20:41mutantmonkey (mutantmonkey) joins
23:21:34<OrIdow6^2>Sanqui: Are you saying that you think deletion of inactive accounts is accelerating?
23:21:36OrIdow6^2 is now known as OrIdow6
23:52:07<@JAA>My CNET forums grab finished. :-)
23:54:54Lord_Nightmare quits [Quit: ZNC - http://znc.in]
23:58:04Lord_Nightmare (Lord_Nightmare) joins

HomeSearchPrevious dayNext day