HomeSearchPrevious dayNext day

00:03:03nertzy quits [Client Quit]
00:03:45nertzy (nertzy) joins
00:26:27Arcorann_ quits [Ping timeout: 244 seconds]
00:29:42Arcorann (Arcorann) joins
00:52:48ThreeHeadedMonkey quits [Ping timeout: 244 seconds]
00:54:59ThreeHeadedMonkey (ThreeHeadedMonkey) joins
01:02:26Matthww quits [Client Quit]
01:02:42Matthww joins
01:15:54Jake (Jake) joins
02:51:38BlueMaxima joins
03:22:07Arcorann quits [Ping timeout: 252 seconds]
03:59:46godane quits [Read error: Connection reset by peer]
03:59:48godane1 joins
04:04:22DLoader quits [Client Quit]
04:06:03qw3rty__ joins
04:10:04qw3rty_ quits [Ping timeout: 265 seconds]
04:23:22atphoenix quits [Read error: Connection reset by peer]
04:24:03atphoenix (atphoenix) joins
04:45:48DogsRNice quits [Read error: Connection reset by peer]
04:55:29marked1957 quits [Quit: Ping timeout (120 seconds)]
04:55:46marked1957 joins
05:13:04<mgrandi>What is the irc channel for the halo project going on now?
05:13:10<mgrandi>i don't see it in the deathwatch page
05:14:03<@hook54321>mgrandi: #yolohalo
05:23:31etnguyen03 quits [Client Quit]
05:44:18isthisthedarkweb quits [Remote host closed the connection]
06:17:57hooway joins
06:19:58lennier2 joins
06:23:59lennier1 quits [Ping timeout: 244 seconds]
06:24:09lennier2 is now known as lennier1
07:10:12<mgrandi>ty
07:52:06DLoader joins
08:01:25Arcorann (Arcorann) joins
08:10:24<skrzyp>thuban: there's an instance hosted at http://premki.space (u: generator@epremki.com p: epremki123), but I'm already in the process of getting the current extraction flow and contacting with authors of the extraction library
08:11:25godane1 quits [Ping timeout: 252 seconds]
08:21:15HackMii quits [Remote host closed the connection]
08:21:28LeighR (LeighR) joins
08:21:35HackMii (hacktheplanet) joins
08:22:53HackMii quits [Remote host closed the connection]
08:23:13HackMii (hacktheplanet) joins
08:25:51HackMii quits [Remote host closed the connection]
08:26:09HackMii (hacktheplanet) joins
08:44:20<yawkat>JAA: ah, annoying. that explains some of the logs though. is there a patch?
08:45:19<yawkat>this night it was another ~800mb, so just as slow as with wget
08:48:26<yawkat>ah... https://github.com/ArchiveTeam/wpull/pull/393
08:59:06Arcorann quits [Ping timeout: 265 seconds]
09:11:05spirit joins
09:28:18LeGoupil joins
09:55:23Arcorann (Arcorann) joins
09:55:49BookBat quits [Ping timeout: 244 seconds]
11:11:24LeighR quits [Client Quit]
11:36:02BlueMaxima quits [Client Quit]
12:49:43yawkat quits [Ping timeout: 252 seconds]
12:52:25yawkat (yawkat) joins
13:16:33LeighR (LeighR) joins
13:16:35LeighR quits [Remote host closed the connection]
13:19:11ragu_ joins
13:22:29ragu__ quits [Ping timeout: 244 seconds]
13:27:21Arcorann quits [Ping timeout: 265 seconds]
13:34:11etnguyen03 (etnguyen03) joins
14:23:19godane joins
15:03:54isthisthedarkweb (isthisthedarkweb) joins
16:28:38El-D joins
16:45:32El-D quits [Remote host closed the connection]
16:45:41britmob quits [Read error: Connection reset by peer]
16:46:44britmob joins
17:00:00<purplebot>Webzdarma edited by Sanqui (+25, /* ArchiveBot jobs */ webzdarma_mwlinkscrape_10) just now -- https://www.archiveteam.org/?diff=46286&oldid=46264
17:02:28lennier1 quits [Client Quit]
17:10:25lennier1 (lennier1) joins
17:38:17<yawkat>okay even with fixed --concurrent, jcenter is still pretty slow right now. i guess they do throttle by ip. whats the easiest way to spread the job? i do have shell access to a few ips if necessary, or is there existing infra for this sort of thing?
17:38:56<yawkat>and how do you actually distribute this? is wpull with a http proxy list fine, or do i need to build something custom?
17:49:01fuzzy8021 quits [Read error: Connection reset by peer]
17:49:17fuzzy8021 joins
17:49:18fuzzy8021 quits [Changing host]
17:49:18fuzzy8021 (fuzzy8021) joins
18:24:49britm0b joins
18:24:49britmob quits [Read error: Connection reset by peer]
18:41:43DogsRNice (Webuser299) joins
19:00:06Eighty quits [Quit: Lost terminal]
19:00:41Eighty (Eighty) joins
19:33:31<revi>wondering if this is sort of thing ArchiveTeam can help? https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(miscellaneous)#Mass_archiving_of_the_Hong_Kong_Daily_Press (direct link to HK library breaks because the link is.... too long)
20:12:07mutantmonkey quits [Ping timeout: 264 seconds]
20:24:31mutantmonkey (mutantmonkey) joins
20:54:16Ryz quits [Ping timeout: 252 seconds]
21:12:59<thuban>looks like the viewer is javascript-based, so not an archivebot job (at least for the actual scans), but seems crackable
21:15:36LeGoupil quits [Client Quit]
21:18:14Ryz (Ryz) joins
21:20:58<thuban>inspect raw source -> find "var htmlURL" in the script -> extract item number and encToken from query parameters -> hit https://mmis.hkpl.gov.hk/ebook/viewer?resource=xml&item=<ITEM>&encToken=<TOKEN> for xml with page image urls -> for each page url, hit https://mmis.hkpl.gov.hk/ebook/viewer?resource=page&item=<ITEM>&ref=<PERCENT_ENCODED_URL>&encToken=<TOKEN>
21:24:35<thuban>i think i'll write a script to grab the raw images later; anyone have opinions on that and/or doing some sort of archivebot run on the site itself for provenance reasons?
21:25:36<thuban>(url structure suggests the latter would get hairy, but i expect i'll need to generate a list of the item pages anyway, so we could use that with !ao)
21:32:01BlueMaxima joins
21:33:46Ryz quits [Remote host closed the connection]
21:34:14<jodizzle>We did do a run of that site back in 2019, but had to de-facto abort it (set an ignore for the main domain) because it was too much. It was also blocking pipeline maintenance.
21:34:54Ryz (Ryz) joins
21:34:55<jodizzle>I considered doing something custom for it, but never did unfortunately.
21:35:22<jodizzle>cc thuban, revi ^
21:35:52<revi>thanks, I'll probably tell them there was some efforts
21:36:44<thuban>jodizzle: yeah, figures. do you think the 22606 results are doable in !ao?
21:36:58<jodizzle>Definitely, yes
21:38:26<jodizzle>Also, to be clear, you're right that the problem is the URL structure. It's too much for a naive !a job (though I don't remember all the details of why).
21:38:37<jodizzle>Also an annoying language-switch cookie, iirc
21:39:59<thuban>cool :) the only question then is whether/how frequently the tokens change (i.e., could i do the extraction from the archivebot results, or would it have to be a redundant scrape on the fly)
21:40:59<thuban>only one way to find out, i guess; i'll see about generating that list
21:41:19nertzy quits [Ping timeout: 265 seconds]
21:42:06isthisthedarkweb quits [Ping timeout: 244 seconds]
21:43:46nertzy (nertzy) joins
21:49:30Jonboy3451 joins
21:50:52isthisthedarkweb (isthisthedarkweb) joins
21:51:30<revi>replied to them: https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(miscellaneous)&diff=1005077812&oldid=1005037769
21:53:28Jonboy345 quits [Ping timeout: 244 seconds]
21:53:49sliccricc (sliccricc) joins
21:54:30Jonboy3451 quits [Ping timeout: 244 seconds]
21:54:46<@EggplantN>arkiver: just wondering any projects upcoming
22:00:32sliccricc quits [Remote host closed the connection]
22:00:51sliccricc (sliccricc) joins
22:08:30Jonboy345 joins
22:12:01Jonboy3451 joins
22:15:41jonboy3452 joins
22:15:42Jonboy345 quits [Ping timeout: 244 seconds]
22:17:37Jonboy345 joins
22:18:47Jonboy3451 quits [Ping timeout: 244 seconds]
22:19:49jonboy3452 quits [Ping timeout: 244 seconds]
22:22:18sliccricc quits [Ping timeout: 264 seconds]
22:50:25@dxrt quits [Quit: ZNC - http://znc.sourceforge.net]
22:52:06dxrt joins
22:52:08dxrt quits [Changing host]
22:52:08dxrt (dxrt) joins
22:52:08@ChanServ sets mode: +o dxrt
22:55:49nerdguy1138 quits [Ping timeout: 252 seconds]
23:11:12spirit quits [Client Quit]
23:28:27Arcorann (Arcorann) joins
23:49:43ThreeHeadedMonkey quits [Ping timeout: 244 seconds]
23:51:38ThreeHeadedMonkey (ThreeHeadedMonkey) joins
23:57:58OrIdow6 quits [Ping timeout: 252 seconds]
23:58:19OrIdow6 (OrIdow6) joins

HomeSearchPrevious dayNext day