HomeSearchPrevious dayNext day

00:14:17<fuzzy8021>arkiver - https://pastebin.com/FYbLznUY
00:14:34<fuzzy8021>seems odd the error'ed items download fine from my home internet
00:15:56<@JAA>SketchTheCow: Is the lack of progress on teamarchive1 also due to an S3 breakage? There must be a bit of a pile of AB data on there, and nothing's been uploaded from it in about 2 weeks I think.
00:16:13<fuzzy8021>and if i wget it manually it seems to pull down ok as well on that same box
00:17:39<@arkiver>fuzzy8021: WARC_ERR indicates something on your machine
00:17:44<@arkiver>you have enough space?
00:18:08<fuzzy8021>yep only 54% full
00:18:48<fuzzy8021>odd because 1 of 4 concurrency is running fine on that box
00:29:54hooway quits [Client Quit]
00:36:25<Craigle>arkiver: Seeing a bunch of jobs come through with 13.5x kb file sizes. Could something be up? https://transfer.notkiska.pw/6Un8E/fresh.JPG
00:37:53Barto quits [Ping timeout: 252 seconds]
00:41:36<Craigle>Nevermind, just figured out that's a "no video" page
00:44:16<Craigle>Or possibly not. Just realized I can't see the username to check the video
00:55:53Barto (Barto) joins
01:01:04<@arkiver>Craigle: yeah it means it doesn't exist
01:01:19<@arkiver>I'll be making an update later that will get us only the highest quality video
01:01:35<@arkiver>if we're still going slow then we'll reduce the quality
01:10:37wizwiz quits [Remote host closed the connection]
01:37:06<tech234a>Is there a way to list the images on Fusl's Docker repo? https://atdr.meo.ws/v2/_catalog/ gives a 404 for me.
01:41:37<Craigle>Awesome, just wanted to make sure it wasn't an issue with the grab
02:00:02Dallas quits [Client Quit]
02:04:12Dallas (Dallas) joins
02:10:24Arcorann_ quits [Ping timeout: 240 seconds]
02:19:54<EggplantN_>i set a lil box off going with this
02:31:03<SketchTheCow>JAA: Oh likely, I'll go look.
02:31:20<@JAA>Thanks
02:33:21<SketchTheCow>Weird, whatever happened, my watchtower stopped. Maybe they reboot the box or something. It's on it now.
02:46:20<EggplantN_>okay i deployed some yolo-ness
02:46:26<EggplantN_>lets see how fucked this website rally is
02:49:08<@arkiver>EggplantN_: damn
02:49:27<@arkiver>i was going to release an update to ignore some stuff, but I'll wait for now
02:49:37<@arkiver>EggplantN_: note that some items are multiple GBs
02:49:42<EggplantN_>yep
02:49:54<EggplantN_>thats fine these VMs are doing 60 concurrent with 240G disks
02:50:13<EggplantN_>i'd go harder but the target will fall over
02:50:18<@arkiver>some items can be very large
02:50:28<@arkiver>and I might make the update anyway tomorrow
02:50:58<@arkiver>also I removed the limit of 200 items/min
02:53:14<@arkiver>actually pushing the update anyway
02:53:45<EggplantN_>ah i didnt deploy watchtower
02:53:50<EggplantN_>so i'll do it tomrorow
02:54:12<@arkiver>yeah updated
02:54:19<@arkiver>please update asap
02:54:23<@arkiver>I won't increase version
02:54:28<@arkiver>minimum version *
02:56:52@arkiver is off to bed
02:57:09<@arkiver>this latest update will decrease time by half for large items (which are the ones holding things up)
02:57:29<@arkiver>another update is coming tomorrow if this is not enough that will strip this even more
03:21:54<@kiska>Has the site died?
03:23:36<OrIdow6>Viewing it in a request, I get errors
03:23:44<OrIdow6>*in a browser
03:24:09<OrIdow6>Now it's gone
03:24:18<OrIdow6>Complaint about an XHR request timing out
03:24:36<@kiska>Oh so we are already causing it to break wonderful :D
03:31:02<OrIdow6>エラーが発生しました , newline, ホームに戻る ; title "XMLHttpRequest timeout"
03:32:11<OrIdow6>Going to try to figure out if it gives a non-200 or not
03:38:48<OrIdow6>Now it doesn't seem to be happening anymore
03:39:59<OrIdow6>And not sure if that would happen with the /dummy/id approach that the script uses - XHR problem suggests that it's only when I go from a category (or suggested) list to a video and it does it with JS instead of with a normal page load
03:52:16<fuzzy8021>i dont think the update to freshlive code got pushed to the docker iamge
03:53:23<@kiska>-_-
04:14:25qw3rty__ quits [Read error: Connection reset by peer]
04:14:37qw3rty__ joins
04:16:38xit5 joins
04:16:44voltagex quits [Ping timeout: 240 seconds]
04:16:50anelki quits [Remote host closed the connection]
04:16:53jlu5 quits [Remote host closed the connection]
04:16:57sec^nd quits [Ping timeout: 240 seconds]
04:16:59anelki (anelki) joins
04:17:02voltagex joins
04:17:10jlu5 (jlu5) joins
04:17:44xit quits [Ping timeout: 240 seconds]
04:17:44xit5 is now known as xit
04:18:04cadence quits [Ping timeout: 240 seconds]
04:18:27sec^nd (second) joins
04:21:23cadence (cadence) joins
04:33:34<@kiska>fuzzy8021: Build in progress
04:34:15<@kiska>Done
04:51:50qw3rty_ joins
04:55:50qw3rty__ quits [Ping timeout: 252 seconds]
05:12:56<fuzzy8021>thanks
05:46:17<OrIdow6>So it has occured to me that though the .eu domain crawl itself should be able to happen fairly quickly (because it's so parallel), the discovery cannot
05:46:45<OrIdow6>So I think I'll try to get that going (or, at least, written) in the next few hours
05:48:20<OrIdow6>Though it might be a privacy/spam concern if some spammer finds the collection on IA (though from the hundreds of views obscure domains' information gets, they're there already), I think I'll do it in warcs
05:48:31<OrIdow6>If there are problems, they can be excluded from the WBM
05:48:46<@JAA>What discovery are you thinking of exactly?
05:48:50<OrIdow6>Scraping EURid
05:48:56<@JAA>Ah, right, nice!
05:49:23<OrIdow6>So, really, discovery part II
05:51:23Doranwen quits [Ping timeout: 252 seconds]
05:52:59<OrIdow6>: filtering
05:53:22<@JAA>Yeah
05:55:05<@JAA>I wonder if it's worth trying to run the homepages through AB since #// wouldn't grab page requisites. I foresee a lot of anger and headaches though.
05:57:51<OrIdow6>Yeah
05:58:01<OrIdow6>From what I've seen, you'd be getting a lot that just time out
05:58:32<@JAA>That's fine.
05:58:58<@JAA>My concerns are rather about Drupal and other script hells.
05:59:04Doranwen joins
05:59:30<@JAA>Where it'd go down annoying rabbit holes of poorly linked <script> tags or JS-extracted URLs.
06:01:37<OrIdow6>Oh
06:03:28Doranwen quits [Ping timeout: 252 seconds]
06:03:33Doranwen joins
06:33:34<Ryz>There's a bunch of space right now, though mainly running small jobs from me ignoring websites for a job to finish faster after being stuck for so long
06:38:39AeonG joins
06:48:00<OrIdow6>In retrospect, I probably should have based this on the URLs grab, considering it's basically that with some more nuanced delay handling
06:54:01<OrIdow6>arkiver: What does this line in the URLS grab pipeline do?: wget_args.append('item-name://'+url)
06:54:57HP_Archivist quits [Quit: Leaving]
08:02:49Arcorann_ joins
08:28:43wickedplayer494 quits [Remote host closed the connection]
08:36:38wickedplayer494 joins
08:53:02hooway joins
09:10:28AeonG quits [Read error: Connection reset by peer]
09:21:25AeonG joins
10:02:29<@HCross>arkiver: I have pressed the "go faster button"
10:03:11<OrIdow6>Macron "has been diagnosed positive" for coronavirus - obviously not big of a news story as it would have been earlier this year
10:03:19<OrIdow6>*not as big
10:07:34<OrIdow6>Oh, that's made the rate go up a lot
10:08:45<@Kaz>post vars
10:17:58HP_Archivist (HP_Archivist) joins
11:43:16godane (godane) joins
12:14:07<@HCross>arkiver: we've killed it
12:14:28<@HCross>freshlive is now very slow, and producing 5XX errors
12:21:43<EggplantN_>arkiver we keep getting 502
12:21:48<EggplantN_>i've set limit to 0
12:21:55<EggplantN_>finish jobs that are out first
13:49:00Silica100 joins
14:03:11Silica100 quits [Remote host closed the connection]
14:28:41<grawity>are the miniclip.com flash games getting archived somewhere?
14:39:36<thuban>grawity: flashpoint has some of them; i don't know about a full scrape https://bluemaxima.org/flashpoint/datahub/Game_Master_List#Flash
14:51:04Arcorann_ quits [Ping timeout: 240 seconds]
15:39:10godane quits [Client Quit]
15:45:25<@arkiver>EggplantN: thanks
15:45:52<@arkiver>HCross: well the go fast button did speed things up
15:46:12<@HCross>before it broke
15:46:27<@arkiver>OrIdow6: it's an addition to Wget-AT to set item names. that way we can do multiple items with a single Wget-AT session
15:47:10<@HCross>I have several hundred GB on my disks already
15:47:15<@HCross>in in-flight items
15:50:16<@HCross>arkiver: could we auto abort big files (and check before somehow if they're big), and then we can requeue those on different servers with more disks?
15:57:46<EggplantN_>^
15:58:00<@HCross>so that we can rattle through the smaller videos quickly
15:58:03<@HCross>and get them out the way
15:59:14@kiska breathes on freshlive
15:59:20<@kiska>Freshlive has issues :D
16:34:21<atphoenix>I guess more helpers are not needed for freshlive? I don't see info on the wiki on where the docker image or other code is located...
17:01:28<Craigle>atphoenix: Not sure if we are still paused at the moment due to website issues
17:01:39<Craigle>But, if you want to spin something up, code is here: Scripts https://github.com/ArchiveTeam/freshlive-grab
17:02:06<Craigle>And docker image is here: atdr.meo.ws/archiveteam/freshlive-grab
17:03:07<Craigle>But the site is slow/dying, so we can't hammer it like on fun jobs, unfortunately
17:08:53<@HCross>arkiver: its dead
17:08:59<@HCross>its throwing 404s
17:15:11<atphoenix>:(
17:15:45<EggplantN_>She's dead
17:16:49<atphoenix>https://atdr.meo.ws/ArchiveTeam/freshlive-grab returns a 'nope'
17:17:16<atphoenix>but it's a moot point if the site be freshdead
17:19:08<@kiska>No no use docker cli
17:19:15<atphoenix>I guess they said closes on Friday 12/18. It's 2 am Friday 12/18 in Japan now.
17:19:18<purplebot>Webzdarma edited by Sanqui (+25, /* ArchiveBot jobs */ job 16) just now -- https://www.archiveteam.org/?diff=45920&oldid=45886
17:40:12<SketchTheCow>JAA: Cleared out. Interestingly, my thing doesn't understand a world where new data isn't coming.
17:40:36<SketchTheCow>So it WILL hold onto certain data until it's pushed through. I'm not sure of a solution to this.
17:50:02<@JAA>SketchTheCow: Right, same with the DPoS targets and other stuff. That's fine.
17:50:08<@JAA>Thanks!
17:57:27<Sanqui>media fired?
17:57:36<@kiska>:D
17:57:46<Sanqui>alternatively
17:57:47<Sanqui>media on fire
17:57:50<@kiska>:D
17:57:59<@kiska>I vote for mediaonfire
17:58:02<Sanqui>(these are getting easier and easier...)
18:07:40<SketchTheCow>It was my favorite part of archive team work
18:10:15<Sanqui>I *am* calling my new discord archival tool... discard
18:10:49<thuban>hah
18:11:22<Sanqui>(follow #discard if interested, I just dropped a preview readme)
18:17:22<@JAA>mediaonfire was also my first thought.
18:21:39<atphoenix>or #mediabonfire
19:07:02VerifiedJ quits [Quit: The Lounge - https://thelounge.chat]
19:08:02VerifiedJ (VerifiedJ) joins
20:13:37second (second) joins
20:16:27sec^nd quits [Ping timeout: 240 seconds]
20:16:28second is now known as sec^nd
20:44:22Ryz quits [Remote host closed the connection]
20:47:33HP_Archivist quits [Client Quit]
20:55:06HP_Archivist (HP_Archivist) joins
21:03:40Ryz (Ryz) joins
21:56:04qw3rty_ quits [Ping timeout: 252 seconds]
22:16:04<SketchTheCow>mediaonfire or mediabonfire are both excellent
22:16:15<SketchTheCow>except mediabonfire sounds like cinnabon's up and coming millenial comeptitor
22:17:37<Doranwen>I like mediaonfire
22:18:04HP_Archivist quits [Ping timeout: 240 seconds]
22:38:41spirit quits [Client Quit]
22:44:11<@JAA>#mediaonfire has been created already (as announced in #archiveteam).
22:51:16Arcorann_ joins
22:54:18<purplebot>Running Archive Team Projects with Docker created by Tech234a (+12683, Create draft page with Docker instructions) just now -- https://www.archiveteam.org/?diff=45921&oldid=0
22:54:35<tech234a>^ work in progress draft
22:54:48<tech234a>wanted to save what I had so far
23:03:35yanome quits [Quit: Ping timeout (120 seconds)]
23:03:46yanome (yano) joins
23:10:12lennier2 joins
23:13:05lennier1 quits [Ping timeout: 252 seconds]
23:13:12lennier2 is now known as lennier1
23:18:20<Lord_Nightmare>is my reading of tlller's proposed copyright felony(?) law correct? I made a tweet-thread about it here: https://twitter.com/Lord_Nightmare/status/1339706621514149896
23:18:37<Lord_Nightmare>... actually this should be in -ot, sorry
23:38:33qw3rty joins
23:39:17<purplebot>Running Archive Team Projects with Docker edited by Tech234a (+2608, Add more instructions) 19 minutes ago -- https://www.archiveteam.org/?diff=45922&oldid=45921
23:43:10<@arkiver>HCross: what is the contact on the zeelandnet WARCs you uploaded?

HomeSearchPrevious dayNext day