HomeSearchPrevious dayNext day

04:04:12geezabiscuit quits [Ping timeout: 255 seconds]
04:06:29geezabiscuit (geezabiscuit) joins
04:41:37michaelblob_ (michaelblob) joins
04:45:09michaelblob quits [Ping timeout: 255 seconds]
05:20:49Iki quits [Remote host closed the connection]
05:20:56Iki joins
05:24:13Iki quits [Remote host closed the connection]
05:24:16Iki joins
05:30:54qwertyasdfuiopghjkl quits [Remote host closed the connection]
06:04:50<@OrIdow6^2>How's your download going Aoede
06:04:51<@OrIdow6^2>?
06:18:14qwertyasdfuiopghjkl joins
06:46:47Maakuth|m joins
07:37:53michaelblob (michaelblob) joins
07:39:15Maturion joins
07:41:16michaelblob_ quits [Ping timeout: 240 seconds]
08:28:13<Maakuth|m>any ideas about mbnet?
08:32:22<@OrIdow6^2>Working on it
08:32:37<@OrIdow6^2>Maakuth|m: May I ask what is special about koti.mbnet.fi? Just redirects to www. for me
08:33:18<Maakuth|m>the content is behind userdirs like koti.mbnet.fi/~username
08:33:46<Maakuth|m>it's a free hosting service provided to subscribers of a finnish computer magazine
08:33:48<@OrIdow6^2>Oh, I see
08:34:05<@OrIdow6^2>Is it the same there and on username.mbnet.fi?
08:34:11<@OrIdow6^2>That's what I've been working with
08:34:16<Maakuth|m>yes, I think so
08:35:33<Maakuth|m>yeah: http://koti.mbnet.fi/~flux responds with 302 redirect to http://flux.mbnet.fi
08:36:48<@OrIdow6^2>Guess we can get both
08:37:06<@OrIdow6^2>Since this looks to be fairly small in overall file size
10:03:18<@OrIdow6^2>Alright Jake Maakuth|m here you are https://pad.notkiska.pw/p/mbnet
10:04:55<Maakuth|m>okay let's see
10:06:04<@OrIdow6^2>Yes I realize there are a million issues with this but I'd like to get to sleep
10:06:30<Maakuth|m>thank you sir. I'll try it out right away
10:08:38<@OrIdow6^2>Thank you for running it\
10:08:53<Maakuth|m>hmmh, wget-at: --regex-type: Invalid value 'pcre'.
10:09:34<Maakuth|m>this is what wget-at --help says: --regex-type=TYPE regex type (posix)
10:09:50<Jake>I'm off to bed right now, but I will get going in the morning assuming anything is left.
10:10:12<Maakuth|m>I built it with the bundled Dockerfile assuming that would get correct stuff in, but maybe not?
10:10:50<@OrIdow6^2>Jake: Well, at the current rate that will be abundantly true
10:11:03<@OrIdow6^2>Maakuth|m: Huh, I didn't know that was possible, looking at it
10:11:05<Jake>haha true
10:11:09<Jake>maybe I should get started.... one sec
10:11:44<Maakuth|m>ah, the rejects file seems complicated, so I suppose i'm gonna need pcre
10:12:57<@OrIdow6^2>If the docker version doesn't have PCRE it can probably be switched to POSIX
10:13:15<@OrIdow6^2>My amazing trick there has been to put all the grab-site ignores into one giant regex
10:13:21<Maakuth|m>let's see what happens if I do that
10:13:24<@OrIdow6^2>But it's mostly nosortedindex that's needed
10:13:47<@OrIdow6^2>Do what?
10:14:20<Maakuth|m>just change the flag to posix. clearly that doesn't work without chaging the regexp itself
10:15:57<@OrIdow6^2>If you do make changes to the script please confirm they're OK before letting it upload BTW, though obviously you're kind of doing that here laready
10:16:01<@OrIdow6^2>*already
10:16:15<@OrIdow6^2>What happened when you changed it?
10:16:41<Maakuth|m>Invalid regular expression ...
10:16:54<Maakuth|m>I think i'll build the binary again
10:17:01<Maakuth|m>adding libpcre2
10:19:11<@OrIdow6^2>Thank you for doing that
10:19:13<Jake>(looks like I didn't compile with it either)
10:19:26<Jake>recompiling.
10:20:55<Maakuth|m>i'm adding debian packages: libpcre2-dev libpcre2-posix0
10:24:59<@OrIdow6^2>Thank you both for doing this, I was not aware pcre was a nonstandard part of my local setup
10:29:25<Jake>so NETWORK_FAIL and "unlink: no such file" are both fine?
10:30:37<@OrIdow6^2>Yeah
10:31:08<@OrIdow6^2>Both are just side effects of users without subdomains
10:31:19<@OrIdow6^2> /that do not exist
10:31:49<Jake>👍
10:37:26<Maakuth|m>hmmh Lua error: script.lua:4: module 'socket.url' not found:
10:37:32<Maakuth|m>I suppose that's another debian package
10:38:57<Maakuth|m>yep, that was lua.socket
10:39:21<Maakuth|m>now running
10:39:44<Jake>wow a whole wordpress blog
10:39:44<Jake>http://juvar.mbnet.fi/blog/
10:40:13<Jake>(sorry, forgot to mention, I've been running for about 10-ish minutes)
10:40:48<Maakuth|m>seems okay. once I see how big they tend to get I'll probably start combining more ranges
10:47:31<@OrIdow6^2>Right now I have it set up to use wget's default recursion depth of 5
10:47:40<@OrIdow6^2>As a cheap way to prevent infinite calendars etc.
10:48:06<Jake>it got a little stuck on wordpress
10:48:09<Jake>but it is fine now
10:48:10<@OrIdow6^2>If we have time maybe we can analyze the big ones near the end to make sure we have everything
10:48:29<Jake>what's the time limit on this site?
10:51:47<Jake>oh it auto-uploads?
10:51:49<Jake>cool!
10:55:01<@OrIdow6^2>If only there were some way to automate the claim-finish and warc-packing processes as well haha
10:56:54<Jake>you could probably script warc-packing hah
10:57:07<Jake>claim-finish less easy without something like the tracker :-)
11:16:17<Aoede>OrIdow6^2: Around ~20% done maybe. Wget got stuck on a loop last night
11:31:08<Maakuth|m>i'm running on two machines, seem to work fine in bunches of five ranges from the document
12:06:27<Maakuth|m>hmm, something went wrong with the upload
12:07:29<Maakuth|m>http://paste.debian.net/1256536/
12:08:06<Jake>got a 502 on upload
12:08:11<Maakuth|m>yeah
12:08:16<Jake>yeah me too lol
12:08:20<Maakuth|m>oh
12:08:29<Jake>hm uhh
12:08:34<Maakuth|m>too bad the script delets the warcs
12:08:42<Maakuth|m>even in this case
12:08:47<Jake>hopefully it doesn't uhh
12:08:47<Jake>yeah
12:08:51<Maakuth|m>maybe someone in -bs knows
12:09:04<Jake>we may have hit size limit or transfer is just down
12:09:39<Maakuth|m>hmm yeah, size limit could be it
12:09:50<Jake>odds are very good we hit the size limit
12:09:57<Maakuth|m>I imagine the range:size mapping is very fuzzy
12:10:35<Jake>oh yeah your earlier one was 1.2GB.
12:10:39<Jake>We def hit the size limit
12:10:50<Maakuth|m>as it depends on how much each user has filled their userdir
12:11:02<Jake>I just looked at the clock so I have to run
12:11:15<Jake>but I'd keep ranges lower based on how this works with uploading for now
12:12:53<Maakuth|m>ok
12:27:37<ThreeHM>Looking at run.bash, I think there's a mistake that causes ranges to be way larger than specified
12:27:54<ThreeHM>Pretty sure that `tail -n +$1 MBNet_usernames_combined.txt | head -n $2` gives you $2 lines starting at offset $1
12:27:55fuzzy8021 quits [Read error: Connection reset by peer]
12:28:16<ThreeHM>So, giving the range 4000-5000 would try to download 5000 items
12:29:17fuzzy8021 (fuzzy8021) joins
12:29:51<Maakuth|m>oop
12:29:53<Maakuth|m> * oops
12:34:15<Maakuth|m>so head should get the difference between $1 and $2
12:35:25<Maakuth|m>or alternatively we need to input the ranges differently
12:37:39<ThreeHM>Should be `head -n $2 MBNet_usernames_combined.txt | tail -n +$1` if I'm not mistaken
12:39:13<Maakuth|m>I don't think it's that easy. because after 'head' the line numbers are different
12:39:54Maturion quits [Ping timeout: 255 seconds]
12:40:07<Maakuth|m>oh... they are not because it's first $2 lines. bad thinking
12:41:18<ThreeHM>Yeah, I'll wait for approval from OrIdow6 before touching anything
12:52:27Maturion joins
13:01:15Iki quits [Ping timeout: 265 seconds]
14:47:15datechnoman quits [Ping timeout: 255 seconds]
15:00:37datechnoman (datechnoman) joins
15:07:27TheTechRobo quits [Client Quit]
16:19:20TheTechRobo (TheTechRobo) joins
17:24:58ThreeHM_ (ThreeHeadedMonkey) joins
17:26:04ThreeHM quits [Ping timeout: 240 seconds]
19:21:07Iki joins
19:30:45Webuser611 joins
20:17:28<@OrIdow6^2>Yeah I forgot I changed the format
20:33:00<@OrIdow6^2>Link in the pad updates Maakuth|m Jake
20:33:03<@OrIdow6^2>Thank you
20:51:32Webuser611 quits [Ping timeout: 265 seconds]
21:04:28<Jake>just checking, does that fix both the range and upload issue?
21:42:06<Jake>OrIdow6^2: still hit the size limit on upload to transfer.
21:45:45Maturion quits [Ping timeout: 255 seconds]
21:55:39Maturion joins
22:21:26Maturion quits [Ping timeout: 265 seconds]
22:49:03wickedplayer494 quits [Remote host closed the connection]
23:00:26<@OrIdow6^2>Jake: IDK what to do about that
23:01:46<@OrIdow6^2>I have thought that I don't need to insist on having control here
23:02:26wickedplayer494 joins
23:02:45<Jake>As long as it's counting correctly, I guess smaller ranges?
23:02:48<@OrIdow6^2>So feel free to change up the script, the procedures, etc., just keep things in sync with each other and obviously don't fake warcs
23:03:36<@OrIdow6^2>Probably the easiest way
23:03:54<Jake>👍I'll see what I can do
23:10:30<ThreeHM_>I've claimed a fairly big range near the end to run over night (assuming my automation works) - Feel free to reclaim them in case progress gets there before I'm back tomorrow
23:10:34ThreeHM_ is now known as ThreeHM
23:18:21DiscantX joins
23:33:24<@OrIdow6^2>Have you accounted for the uploading issue ThreeHM?
23:37:03<ThreeHM>I've split it into batches of 200 items each. If anything still ends up over the size limit, I can re-run that manually with a smaller batch size

HomeSearchPrevious dayNext day