HomeSearchPrevious dayNext day

00:15:39Arcorann_ joins
00:17:40<Jake>(Amazon Sidewalk has a interesting security whitepaper https://m.media-amazon.com/images/G/01/sidewalk/final_privacy_security_whitepaper.pdf )
00:51:26Frogging101 joins
00:52:28<Frogging101>What tool should one use to save URLs to WARC? Not a full crawler but something to call from, say, a bash/python script on a list of URLs
00:52:34<Frogging101>wpull looks kind of abandoned
00:56:31<@OrIdow6>Wpull is still used in ArchiveBot
00:57:09<@OrIdow6>Wget also works, minus the angle bracket issue, an ambiguity in the warc specification that means wget's output isn't read anywhere
00:57:13<@OrIdow6>*everywhere
00:57:29<Frogging101>angle bracket issue?
00:58:55<@OrIdow6>See second paragraph of http://fileformats.archiveteam.org/wiki/WARC (first result on DuckDuckGo, evidently it's a small world)
00:59:16<Frogging101>hmm
00:59:21<@OrIdow6>Basically, some software won't read warcs produced by wget
00:59:25<Frogging101>so if I use wget how much of an issue will this be in practice
00:59:28<@OrIdow6>Though support has gotten much better recently
00:59:29<Frogging101>like how much software is broken
01:00:03<Frogging101>And does Archive.org care
01:02:49<@OrIdow6>A lot of older software is broken, IIRC WebRecorder works around it
01:03:14dm4v_ joins
01:03:43<@OrIdow6>I know that at least in the past archive.org would fail with this wget output; I remember there was talk that it could have been fixed, but I don't know if that ever happened
01:04:16<@OrIdow6>But if you're really concerned about CDXs, wget can make them itself
01:04:19dm4v quits [Ping timeout: 258 seconds]
01:04:19dm4v_ is now known as dm4v
01:04:19dm4v quits [Changing host]
01:04:19dm4v (dm4v) joins
01:05:12<@OrIdow6>Since presumably this won't be going into the Wayback Machine
01:05:32<Frogging101>yeah, probably not, but just curious how borked a wget-based WARC grab would be
01:05:32<@OrIdow6>Also, you can use wget-at/wget-lua (github.com/archiveteam/wget-lua), a fork used in projects
01:05:51<@OrIdow6>Which obviously uses the old/normal format
01:06:05<Frogging101>ah, neat
01:07:16<Wayward>OrIdow6: I thought warc output was fixed in wget
01:07:52<Wayward>I stick JAA in #wget on freenode and spoke ancient words
01:08:08<Wayward>*stuck
01:13:35onetruth joins
01:13:36<@OrIdow6>Wayward: Still seeing the brackets on 1.21 (packaged with Debian)
01:14:05<@OrIdow6>Which isn't the most recent version, though (which, incidentally, came up with the Shift_JIS issue)
01:14:48<@OrIdow6>(Well, maybe it's the highest number, but there have been commits since the basis Debian uses)
01:15:00<Frogging101>this bullshit that I'm archiving is just something where I can count up the URLs, regex some stuff out of it to get more URLs, and then download the lot
01:15:13<Frogging101>count up meaning /foo/1/, /foo/2/, etc.
01:15:43<Frogging101>so I don't need any fancy crawling or excessive downloading of outbound links
01:16:02<Frogging101>I think wget's page-requisites should be enough
01:16:17<Frogging101>though I've wondered how robust that is
01:29:30<Frogging101>hmm... can wget append to a WARC instead of overwriting it?
01:29:57<Frogging101>When I try to use --continue or --no-clobber with it, it says it's not supported with WARC output
01:50:42<Frogging101>maybe i'm being stupid and should just use the lua support to add the URLs to the grab list
01:50:48<Frogging101>instead of calling wget on every one
01:53:21<@OrIdow6>--input-file
01:54:19<Frogging101>yeah, there's that. I also want to parse the results for more URLs (it's a topic listing). Could do two passes.
01:58:23<Frogging101>Example URL: https://www.misterpoll.com/forums/129
01:58:28Iki joins
02:00:16<Frogging101>I'd want to grab all the /forums/x and parse that for /forums/x/topics/n, then grab those. And then there are pages within forums and topics
02:00:40<Frogging101>this site is absolute trash but it holds nostalgia for me for some reason. it's very, very old
02:00:40Iki1 quits [Ping timeout: 258 seconds]
02:02:18driib1 (driib) joins
02:03:26<Frogging101>Could even put it on Wayback, but I don't know Sketch's policy about accepting WARCs from people
02:04:22driib quits [Ping timeout: 250 seconds]
02:04:22driib1 is now known as driib
02:08:01HP_Archivist quits [Read error: Connection reset by peer]
02:08:25HP_Archivist (HP_Archivist) joins
02:18:38<Frogging101>OrIdow6: Is this a mistake? --domains=voat.co in bintray-grab https://github.com/ArchiveTeam/bintray-grab/blob/master/pipeline.py#L187
02:29:25Larsenv quits [Quit: ZNC 1.8.2+deb1+focal2 - https://znc.in]
02:30:21Larsenv (Larsenv) joins
02:39:18<@OrIdow6>Frogging101: It doesn't affect anything
02:39:32<@OrIdow6>Since I override get_urls without paying attention to wget's verdict
02:39:43<@OrIdow6>But you should probably do it the right way in yours, anyway
02:45:06ThreeHM quits [Ping timeout: 250 seconds]
02:46:21<@JAA>Wayward, OrIdow6: Guess I'll poke them again about that WARC patch. I asked darnir to bump it to WARC/1.1, which resolves the angle bracket issue, but apparently that didn't make it into 1.21.
02:46:39<Wayward>++
02:47:10ThreeHM (ThreeHeadedMonkey) joins
02:47:11<Wayward>joe
02:47:15<Wayward>oop
02:49:53<@JAA>Also reminds me yet again that we still didn't fix all the file operation issues properly and need to upstream that afterwards.
03:17:41qw3rty_ joins
03:21:33qw3rty__ quits [Ping timeout: 258 seconds]
03:33:18Larsenv quits [Remote host closed the connection]
03:36:46Larsenv (Larsenv) joins
03:57:15DogsRNice quits [Read error: Connection reset by peer]
04:05:28<Frogging101>OrIdow6: ah, okay
04:20:31HP_Archivist quits [Read error: Connection reset by peer]
04:20:55HP_Archivist (HP_Archivist) joins
04:21:53lukash7 quits [Client Quit]
04:22:25lukash7 joins
04:29:33BlueMaxima_ joins
04:33:37BlueMaxima quits [Ping timeout: 258 seconds]
05:13:39MaxG quits [Ping timeout: 244 seconds]
05:35:14fuzzy8021 quits [Killed (NickServ (GHOST command used by fuzzy802!~fuzzy8021@173-224-26-244.ptcnet.net))]
05:35:21fuzzy8021 (fuzzy8021) joins
05:52:26MaxG joins
06:24:42fuzzy8021 quits [Killed (NickServ (GHOST command used by fuzzy802!~fuzzy8021@173-224-26-244.ptcnet.net))]
06:24:49fuzzy8021 (fuzzy8021) joins
07:06:28BlueMaxima__ joins
07:10:24BlueMaxima_ quits [Ping timeout: 258 seconds]
07:21:53BlueMaxima_ joins
07:25:44BlueMaxima__ quits [Ping timeout: 258 seconds]
07:58:49fuzzy8021 quits [Killed (NickServ (GHOST command used by fuzzy802!~fuzzy8021@173-224-26-244.ptcnet.net))]
07:58:56fuzzy8021 (fuzzy8021) joins
08:20:33HP_Archivist quits [Ping timeout: 258 seconds]
08:25:57BlueMaxima_ quits [Read error: Connection reset by peer]
10:01:02Tweebie joins
10:01:14Tweebie leaves
10:22:26fuzzy8021 quits [Killed (NickServ (GHOST command used by fuzzy802!~fuzzy8021@173-224-26-244.ptcnet.net))]
10:22:33fuzzy8021 (fuzzy8021) joins
10:42:14noteness quits [Remote host closed the connection]
10:42:30noteness (noteness) joins
11:26:16LeGoupil joins
11:43:40LeGoupil quits [Client Quit]
12:22:48LeGoupil joins
13:54:49Iki quits [Ping timeout: 258 seconds]
14:31:28hexa- quits [Quit: WeeChat 3.1]
14:32:30hexa- (hexa-) joins
15:40:14Arcorann_ quits [Ping timeout: 258 seconds]
15:51:36dm4v quits [Read error: Connection reset by peer]
16:30:43hexa- quits [Quit: WeeChat 3.1]
16:30:57hexa- (hexa-) joins
16:31:28hexa- quits [Quit: WeeChat 3.1]
16:31:47hexa- (hexa-) joins
17:16:08ave quits [Read error: Connection reset by peer]
17:17:28ave (ave) joins
17:32:55Ryz quits [Remote host closed the connection]
17:34:09Ryz (Ryz) joins
17:54:27spirit joins
17:55:53DogsRNice (Webuser299) joins
18:35:42HP_Archivist (HP_Archivist) joins
18:37:22spirit quits [Client Quit]
19:02:43noteness quits [Remote host closed the connection]
19:03:30noteness (noteness) joins
19:22:31LeGoupil quits [Client Quit]
19:58:41MaxG quits [Remote host closed the connection]
20:44:37Iki joins
21:13:58spirit joins
21:21:33HP_Archivist quits [Read error: Connection reset by peer]
21:21:58HP_Archivist (HP_Archivist) joins
22:03:42<Frogging101>too bad this never got merged https://github.com/ArchiveTeam/wpull/pull/360
22:08:38<@JAA>Huh, Travis deleted the old build logs. WTF?
22:09:19<@JAA>Oh
22:09:26<@JAA>'Please be aware travis-ci.org will be shutting down by end of May 2021.'
22:09:31<@JAA>RIP
22:13:22Wayward quits [Ping timeout: 250 seconds]
22:23:13sec^nd quits [Remote host closed the connection]
22:24:44sec^nd (second) joins
22:25:40<Frogging101>JAA: jesus christ, what
22:25:47<Frogging101>Wow I hadn't heard of this
22:31:03HP_Archivist quits [Read error: Connection reset by peer]
22:31:29HP_Archivist (HP_Archivist) joins
22:31:45benjins quits [Remote host closed the connection]
22:33:58benjins joins
22:34:15benjins quits [Remote host closed the connection]
22:35:48benjins joins
22:36:45benjins quits [Remote host closed the connection]
22:38:13benjins joins
22:38:35djsrv (djsrv) joins
22:42:59HP_Archivist quits [Remote host closed the connection]
22:43:07HP_Archivist (HP_Archivist) joins
22:46:02mgrandi quits [Write error: Connection reset by peer]
22:47:36godane1 joins
22:50:12godane quits [Ping timeout: 250 seconds]
22:52:00mgrandi (mgrandi) joins
23:20:59LordThanatos quits [Quit: WeeChat 2.3]
23:22:18<@JAA>'We’re aiming to sunset the travis-ci.org domain on May 31, 2021. Any accounts still on the travis-ci.org domain after that date will still have access to historical data for build logs and config information [...]'
23:22:22<@JAA>Uhuh
23:25:41<@JAA>Oh wait, this could be related to the repo rename from chfoo/wpull to ArchiveTeam/wpull.

HomeSearchPrevious dayNext day