HomeSearchPrevious dayNext day

00:11:20no112 quits [Quit: Ping timeout (120 seconds)]
00:11:30no112 joins
00:12:43Nicole quits [Client Quit]
00:16:58Jonboy345 quits [Read error: Connection reset by peer]
00:18:10Jonboy345 joins
00:18:44Jonboy345 quits [Read error: Connection reset by peer]
00:22:02Jonboy345 joins
00:25:39<SketchTheCow>I mean, that's the site
00:37:59no112 quits [Client Quit]
00:38:13daalekz quits [Client Quit]
00:38:13no112 joins
02:04:21BlueMaxima quits [Read error: Connection reset by peer]
02:26:07<thuban>are we planning to update the warrior vm appliance to run the new warrior dockerfile?
02:26:32<@EggplantN>we are in due time
02:26:55<thuban>ok, cool
02:27:30<@EggplantN>a working one was first
02:27:40<@EggplantN>we're looking at the images likely with chfoo later :D
02:30:11<thuban>viel erfolg!
02:51:32BlueMaxima joins
03:02:15BlueMaxima quits [Client Quit]
03:20:34Arcorann_ quits [Ping timeout: 252 seconds]
03:49:02isthisthedarkweb quits [Ping timeout: 244 seconds]
04:02:34DLoader quits [Quit: DLoader]
04:07:03qw3rty_ joins
04:10:37qw3rty__ quits [Ping timeout: 252 seconds]
04:31:58Ajay quits [Remote host closed the connection]
04:32:13Ajay joins
04:32:22Ajay quits [Remote host closed the connection]
04:32:42Ajay joins
04:37:01Wayward quits [Ping timeout: 252 seconds]
04:38:07Wayward (wayward) joins
04:44:56DogsRNice quits [Read error: Connection reset by peer]
04:49:57etnguyen03 quits [Client Quit]
04:52:52godane (godane) joins
04:54:44godane1 quits [Ping timeout: 240 seconds]
05:01:53Wayward quits [Ping timeout: 244 seconds]
05:03:02Wayward (wayward) joins
05:34:06Mee joins
05:55:06Mee quits [Ping timeout: 244 seconds]
06:08:04klg quits [Ping timeout: 240 seconds]
06:08:18klg (klg) joins
06:23:06HP_Archivist quits [Client Quit]
06:26:26godane1 joins
06:26:40godane quits [Read error: Connection reset by peer]
06:59:40DLoader joins
07:15:12Mateon1 quits [Remote host closed the connection]
07:16:04Mateon1 joins
07:40:08Mee joins
07:40:26godane1 quits [Read error: Connection reset by peer]
07:44:52<Mee>NaNoWriMo forum archives are getting retired on 25th February, and it'd be great to get them archived before then. The forum threads themselves are only visible to a logged-in user. We have a username and password so we can access them, but aren't sure how to go about archiving the whole thing.
07:51:41<OrIdow6>Mee: The basic method is usually to get a login cookie and then put that into whatever crawler you're using
07:53:29<Mee>got any crawlers you could recommend?
07:56:38<OrIdow6>wget (may cause problems if you want to output to a warc file, and play back using certain software), wpull
07:58:56<OrIdow6>If you want something easy, use htttrack
07:59:45<OrIdow6>This is very broad, because I don't know how much background knowledge you have, or what you specifically want to do
08:00:15<OrIdow6>Also, if you want public copies, you can probably wait here and have someone do it for you
08:02:20HP_Archivist (HP_Archivist) joins
08:07:38<Mee>public copies would be great, but the forums aren't accessible without being logged in there.
08:08:17<Mee>we've used httrack a bit before, but not too much.
08:19:07hooway joins
08:35:22<OrIdow6>httrack is easy to use if you don't have the requisite technical knowledge for the other two
08:35:49<OrIdow6>If someone here was willing to do it, you could send them either your login credentials or your login cookie
08:54:16Arcorann_ joins
09:05:07BookBat joins
09:09:28<yawkat>okay, wrt bintray: jcenter does actually have a literal file index at https://jcenter.bintray.com/
09:14:56<yawkat>im going to do an explorative wget --recursive. no clue how big the repo actually is (100s of gbs? tbs? 10s of tbs?)
09:25:02skrzyp joins
09:26:58<yawkat>from what i can tell looking at the documentation ( https://www.jfrog.com/confluence/display/BT/Working+With+Supported+Package+Formats ), jcenter seems to be their only truly centralized repos. the other repos are /user/project-namespaced, im not sure how they could be scraped
09:38:59<yawkat>maybe scanning github for maven and gradle usages of repos could work
09:42:38HP_Archivist quits [Client Quit]
09:54:44<BookBat>Any Pearl experts around? I'm trying out the NanoWrimo-specific "scrapers" from http://www.naperwrimo.org/archivenanomail/, but get an "Error GETing https://archive.nanowrimo.org/sign_in: Can't connect to archive.nanowrimo.org:443 (hostname verification failed) at archivenovels.pl"
09:55:05LeGoupil joins
09:55:18<BookBat>Is it a Perl, SSL or some other bit of setup version mismatch?
09:56:41<SpikedCola>if i visit https://archive.nanowrimo.org/sign_in i get an SSL error in chrome, ERR_CERT_COMMON_NAME_INVALID
09:57:15<SpikedCola>not a perl dev, but in theory the error is valid then?
09:57:49<SpikedCola>there's probably a flag to disable ssl verification, which would let perl carry on.
09:58:24<BookBat>Fascinating, my Firefox doesn't bat an eye - but doesn't light up an SSL lock, either, so it just reverts to http
09:58:50<SpikedCola>it looks like the error is caused by the ssl cert being for "nanowrimo.org" , not "archive.nanowrimo.org" or "*.nanowrimo.org"
09:58:55<skrzyp>Chomikuj.pl will be probably shut down by Polish supreme court order: https://spidersweb.pl/2021/01/chomikuj-sad-najwyzszy-final.html (goolag translate does the job pretty well)
10:00:29<skrzyp>While the article tells about movie piracy, it's the minority of their content IMO, most of it is unique user's content, like rare book scans, magazines, various datasheets, service manuals and software which is pretty much hard to find elsewhere - for example car software, maps, stock phone firmwares and service utilities (often for older devices too), etc.
10:00:35<SpikedCola>ssl labs also complains: https://www.ssllabs.com/ssltest/analyze.html?d=archive.nanowrimo.org
10:00:40<SpikedCola>surprised firefox doesnt actually
10:01:04<SpikedCola>possible solution here: https://stackoverflow.com/a/6795057/1058081
10:01:11<skrzyp>But I'm quite woried about how much the whole chomikuj.pl weights in data, even after deduplication
10:01:50<skrzyp>as this is the biggest polish "shared storage" website and it's running for, like, 15 years?
10:01:59<BookBat>Ah, of course, they've moved the website to "archive", but didn't change the certs. I've got an ancient Firefox, I guess. It just drops the SSL and continues to http://, unless I explicitly say "https://"
10:02:34<BookBat>10-15 years seems to be an average lifespan for a website these days :/
10:03:18<skrzyp>but the content, except these polish movies might be also highly relevant for foreigners, as I found many English books which I wasn't able to find even on libgen, same for software
10:04:00<skrzyp>the trick is that they're rate-limiting the transfer and using per-user quotas to make profit and balance the site traffic
10:04:01HP_Archivist (HP_Archivist) joins
10:04:02<skrzyp>BUT
10:04:06<skrzyp>it's pretty much flawed
10:04:35<skrzyp>for example, if you use a mobile application (which no one uses), you can "preview" the files and this feature is not limited by filesize
10:04:42<skrzyp>the "preview" does not count to the user quota
10:05:01<skrzyp>people quickly did what was needed, scraped out the HTTP communication and extracted URL schemas
10:05:35<skrzyp>there are several "direct link generators" out in the wild and the schema is also explained in a few places
10:06:46<skrzyp>also, people were able to get more quota by uploading files, each upload added some quota back to you and each download of your file also added extra traffic to your profile
10:07:03<BookBat>I'm so awed by what people manage to figure out )
10:57:21Wayward quits [Ping timeout: 244 seconds]
11:14:09LeGoupil quits [Client Quit]
11:20:36Mee quits [Ping timeout: 244 seconds]
11:43:46sliccricc quits [Remote host closed the connection]
12:17:08yanome quits [Client Quit]
12:18:51qw3rty_ quits [Read error: Connection reset by peer]
12:19:06qw3rty_ joins
12:19:12no112 quits [Client Quit]
12:19:23no112 joins
12:37:14luckcolors quits [Remote host closed the connection]
12:37:14Jonimus quits [Ping timeout: 265 seconds]
12:37:41Jonimus joins
12:38:21luckcolors (luckcolors) joins
12:39:39ave quits [Quit: Ping timeout (120 seconds)]
12:40:04ave (ave) joins
12:40:57ragu_ joins
12:43:44ragu quits [Ping timeout: 240 seconds]
13:32:46etnguyen03 (etnguyen03) joins
13:34:17systwi_ (systwi) joins
13:36:41systwi quits [Ping timeout: 265 seconds]
13:47:56jamiew (jamiew) joins
14:49:46godane joins
15:08:02jamiew quits [Ping timeout: 265 seconds]
15:20:35jamiew_ (jamiew) joins
15:24:03jamiew_ quits [Client Quit]
15:27:05Arcorann_ quits [Ping timeout: 240 seconds]
15:37:29ragu__ joins
15:39:26godane1 joins
15:41:23ragu_ quits [Ping timeout: 265 seconds]
15:42:21godane quits [Ping timeout: 265 seconds]
15:43:34ragu_ joins
15:47:12ragu__ quits [Ping timeout: 244 seconds]
15:47:41ragu__ joins
15:48:09ragu_ quits [Ping timeout: 265 seconds]
15:58:12<nepeat>i figure i'd drop some of my input as someone that spun up 160 servers under my own user
15:59:52<nepeat>It worked but having the uptake of keeping the setup consistent and healthy alone and while I'm at work or asleep was a pain so I spun down and gave control of everything to elsewhere
16:47:38godane (godane) joins
16:50:10godane1 quits [Ping timeout: 252 seconds]
17:27:49godane1 joins
17:30:08godane quits [Ping timeout: 265 seconds]
17:47:38LeGoupil joins
17:55:57isthisthedarkweb (isthisthedarkweb) joins
18:05:28godane2 joins
18:08:19godane1 quits [Ping timeout: 265 seconds]
18:58:57DogsRNice (Webuser299) joins
19:01:44Craigle quits [Ping timeout: 240 seconds]
19:01:58thuban quits [Ping timeout: 265 seconds]
19:29:56Craigle (Craigle) joins
19:37:05thuban joins
19:40:07igloo22225 quits [Ping timeout: 252 seconds]
19:43:48igloo22225 (igloo22225) joins
19:59:38sliccricc (sliccricc) joins
20:03:47sliccricc quits [Remote host closed the connection]
20:38:20BlueMaxima joins
20:50:43jut quits [Ping timeout: 265 seconds]
20:51:20jut (jut) joins
20:58:45jut quits [Ping timeout: 244 seconds]
21:01:08jut (jut) joins
21:12:14<yawkat>wget --recursive seems to be working well for jcenter, but is there a way to paralellize it? i only got about 1G download since this morning, too slow to back up the whole thing
21:12:40<yawkat>i could build a tool to do it but id rather avoid that if theres already a solution :)
21:20:58jut quits [Ping timeout: 244 seconds]
21:22:08LeGoupil quits [Client Quit]
21:24:16BlueMaxima quits [Read error: Connection reset by peer]
21:26:54jut (jut) joins
21:38:48daalekz (daalekz) joins
21:44:26ThreeHeadedMonkey quits [Quit: WeeChat 2.9]
21:44:39ThreeHeadedMonkey (ThreeHeadedMonkey) joins
21:45:30OrIdow6 quits [Quit: Quitting.]
21:45:56OrIdow6 (OrIdow6) joins
21:46:28<programmerq>yawkat▸ check out wpull with its --concurrent option
21:47:36<yawkat>Ah, thanks
21:50:44<thuban>skrzyp: can you link us to the schema explanation/"direct link generator"? (does using one of these generators mean you can download full files without needing an account?)
21:51:22<@EggplantN>kinda a high priority issue
21:51:25<yawkat>programmerq: is there a "sane default" config for it? i assume i need --database, anything else?
21:51:28<@EggplantN>but when can the tracker have a dark mode
21:53:41<yawkat>and is it preferred to save to warc? jcenter is purely static after all, so does it really matter?
21:58:48Dj-Wawa quits [Quit: Dj-Wawa]
21:59:05Dj-Wawa joins
22:00:16mutantmonkey quits [Remote host closed the connection]
22:00:17HackMii quits [Write error: Broken pipe]
22:00:18Jonboy345 quits [*.net *.split]
22:00:18Matthww quits [*.net *.split]
22:00:18katocala quits [*.net *.split]
22:00:18lennier1 quits [*.net *.split]
22:00:18Ruthalas quits [*.net *.split]
22:00:18lunik1 quits [*.net *.split]
22:00:18Larsenv quits [*.net *.split]
22:00:18benjins quits [*.net *.split]
22:00:18jeff-kao-propublica-org quits [*.net *.split]
22:00:18divi quits [*.net *.split]
22:00:18yano quits [*.net *.split]
22:00:18britm0b quits [*.net *.split]
22:00:18rsn quits [*.net *.split]
22:00:18xit quits [*.net *.split]
22:00:18SpikedCola quits [*.net *.split]
22:00:18VerifiedJ quits [*.net *.split]
22:00:18Atom-- quits [*.net *.split]
22:00:18lukash7 quits [*.net *.split]
22:00:18treora quits [*.net *.split]
22:00:18space quits [*.net *.split]
22:00:18monoxane quits [*.net *.split]
22:00:18Jake quits [*.net *.split]
22:00:18balrog quits [*.net *.split]
22:00:18@arkiver quits [*.net *.split]
22:00:18BPCZ quits [*.net *.split]
22:00:18summerisle quits [*.net *.split]
22:00:18voltagex quits [*.net *.split]
22:00:18limb quits [*.net *.split]
22:00:18Zerote quits [*.net *.split]
22:00:18colona quits [*.net *.split]
22:00:18pgup quits [*.net *.split]
22:00:18avoozl quits [*.net *.split]
22:00:18Jon quits [*.net *.split]
22:00:18@kiska quits [*.net *.split]
22:00:18s-crypt quits [*.net *.split]
22:00:18atomicthumbs quits [*.net *.split]
22:00:18archzz quits [*.net *.split]
22:00:18HotSwap quits [*.net *.split]
22:00:18rewby quits [*.net *.split]
22:00:18kpcyrd quits [*.net *.split]
22:00:18cadence quits [*.net *.split]
22:00:18EggplantNBot quits [*.net *.split]
22:00:18FalconK quits [*.net *.split]
22:00:18@kiskaWeebChat quits [*.net *.split]
22:00:18@Kaz quits [*.net *.split]
22:00:18mupf quits [*.net *.split]
22:00:18maxfan8 quits [*.net *.split]
22:00:19@AlsoJAA quits [*.net *.split]
22:00:19programmerq quits [*.net *.split]
22:00:19phuzion quits [*.net *.split]
22:00:19step quits [*.net *.split]
22:00:19nyany quits [*.net *.split]
22:00:19MrKen quits [*.net *.split]
22:00:19mazet quits [*.net *.split]
22:00:19kiskaLogBot quits [*.net *.split]
22:00:19kallsyms quits [*.net *.split]
22:00:19sknebel quits [*.net *.split]
22:00:19brabo quits [*.net *.split]
22:00:19Suika quits [*.net *.split]
22:00:19jodizzle quits [*.net *.split]
22:00:19pie_ quits [*.net *.split]
22:00:19Hecz quits [*.net *.split]
22:00:20AlsoJAA joins
22:00:20treora joins
22:00:20mazet joins
22:00:20AlsoJAA quits [Changing host]
22:00:20AlsoJAA (JAA) joins
22:00:20MrKen joins
22:00:21kiskaLogBot joins
22:00:22atomicthumbs joins
22:00:23@ChanServ sets mode: +o AlsoJAA
22:00:25BPCZ joins
22:00:28HotSwap joins
22:00:28Jon joins
22:00:28archzz joins
22:00:29pie_ joins
22:00:30Atom-- joins
22:00:32lunik1 joins
22:00:32mupf joins
22:00:33colona joins
22:00:33xit joins
22:00:33Matthww joins
22:00:34brabo joins
22:00:35step joins
22:00:35Ruthalas joins
22:00:36rsn joins
22:00:37Jonboy345 joins
22:00:37katocala joins
22:00:38avoozl joins
22:00:39lukash7 joins
22:00:40Zerote joins
22:00:40space joins
22:00:40phuzion (phuzion) joins
22:00:42rewby joins
22:00:45FalconK joins
22:00:47pgup joins
22:00:48balrog joins
22:00:48programmerq joins
22:00:49Kaz joins
22:00:51kpcyrd joins
22:00:54brabo quits [Changing host]
22:00:54brabo (brabo) joins
22:00:54maxfan8 joins
22:00:55kiskaWeebChat joins
22:01:00Hecz joins
22:01:03limb joins
22:01:05summerisle (summerisle) joins
22:01:25Suika joins
22:01:32lennier1 joins
22:01:45Hecz quits [Changing host]
22:01:45Hecz (Hecz) joins
22:01:49benjins joins
22:01:49kiskaWeebChat quits [Client Quit]
22:01:52jodizzle (jodizzle) joins
22:01:53britmob joins
22:01:53kallsyms joins
22:01:54SpikedCola (SpikedCola) joins
22:01:55divi (divi) joins
22:01:59arkiver (arkiver) joins
22:01:59nyany (nyany) joins
22:02:00voltagex joins
22:02:01@ChanServ sets mode: +o arkiver
22:02:07yano (yano) joins
22:02:11kiskaWeebChat (kiska) joins
22:02:12@ChanServ sets mode: +o kiskaWeebChat
22:02:26cadence (cadence) joins
22:02:27sknebel (sknebel) joins
22:02:44Church joins
22:02:44Larsenv (Larsenv) joins
22:02:57jlu5 quits [Ping timeout: 240 seconds]
22:03:12kiska (kiska) joins
22:03:12@ChanServ sets mode: +o kiska
22:04:11aleph quits [Ping timeout: 265 seconds]
22:04:56<yawkat>alright, ima keep this running overnight, will see if i get blocked...
22:11:35<thuban>EggplantN: https://transfer.notkiska.pw/rxjVf/at-tracker-dark.css
22:12:12LeighR (LeighR) joins
22:13:14VerifiedJ (VerifiedJ) joins
22:15:13Barto quits [Ping timeout: 252 seconds]
22:16:11<mgrandi>@phuzion: the VGPC might take those
22:19:37Kaz quits [Changing host]
22:19:37Kaz (Kaz) joins
22:19:37@ChanServ sets mode: +o Kaz
22:27:06EggplantNBot joins
22:27:29godane2 quits [Read error: Connection reset by peer]
22:27:30HackMii (hacktheplanet) joins
22:28:17godane (godane) joins
22:29:46Barto (Barto) joins
22:30:23jlu5 (jlu5) joins
22:38:28Barto quits [Ping timeout: 244 seconds]
22:38:44Barto (Barto) joins
22:45:27HackMii quits [*.net *.split]
22:45:27EggplantNBot quits [*.net *.split]
22:46:14daalekz quits [Client Quit]
22:47:21HackMii (hacktheplanet) joins
22:47:21EggplantNBot joins
22:50:03hooway quits [Client Quit]
22:56:32jlu5 quits [Client Quit]
22:59:41jlu5 (jlu5) joins
23:08:59britmob quits [Read error: Connection reset by peer]
23:09:53LeighR leaves
23:11:06britmob joins
23:13:40mutantmonkey (mutantmonkey) joins
23:14:43Arcorann_ joins
23:39:14<phuzion>mgrandi: thanks, I’ll look into it
23:39:51<mgrandi>They use discord, I can get a link
23:40:41<mgrandi>https://www.preservegames.org/p/contact-donate.html
23:53:00<@JAA>yawkat: Note that --concurrent is broken on wpull 2.x. It always runs at 1 concurrency (unless you set the concurrency from a plugin).
23:53:43Dj-Wawa quits [Client Quit]
23:54:25Dj-Wawa joins

HomeSearchPrevious dayNext day