HomeSearchPrevious dayNext day

00:27:00Lord_Nightmare quits [Quit: ZNC - http://znc.in]
00:41:58Lord_Nightmare (Lord_Nightmare) joins
00:44:19blankie (blankie) joins
00:46:09pabs (pabs) joins
00:48:32<pabs>hi all. whats the easiest way to extract a site from web.archive.org? its penta.debconf.org, which was static HTML with pages that shouldn't change after creation and where the server was lost and there are probably no backups
00:51:52<@OrIdow6>If it uses relative links, easy way is to do wget or similar with an allow regex for https://web.archive.org/[0-9]+id_/(URL matching site)
00:53:40Mineroboter_ joins
00:54:35Mineroboter quits [Ping timeout: 258 seconds]
00:57:15<pabs>probably the cdx API is a better option than that?
00:57:28<pabs>https://archive.org/help/wayback_api.php
00:59:16<@OrIdow6>Oh, that too
00:59:23<@OrIdow6>That may not be the "easiest"
00:59:42<@OrIdow6>If you are not adept with a CLI
01:01:31<@OrIdow6>curl "https://web.archive.org/cdx/search?url=query" | grep " 200 " | awk '{ printf("https://web.archive.org/web/%sim_/%s\n", $2, $3) }' | wget --no-directories --retry-on-http-error=429 --input-file=- is what I've used if you are
01:01:37<@OrIdow6>But recursive wget is the easiest
01:02:23<pabs>I was assuming there would be an existing command-line tool that would use the API and could download everything related to the domain
01:02:45<@OrIdow6>Not to my knowledge
01:02:59<@OrIdow6>But what I've put should do the same thing
01:03:18<@OrIdow6>Unless it's a very big site, in which case you'll have to use the CDX pagination API
01:03:20<anarcat>for completeness, i know of https://github.com/jsvine/waybackpack https://github.com/hartator/wayback-machine-downloader
01:03:33<@OrIdow6>OH
01:03:34<anarcat>which may or may not use those APIs
01:03:36<@OrIdow6>*Oh
01:03:40<anarcat>i suspect the answer is "not"
01:03:49<pabs>anarcat: could you say more about the WARC thing you mention on another channel?
01:04:12<anarcat>yes
01:04:21<pabs>IA has got the "backup" part of "backup the Internet" down, but they seem to be lacking in the "restore" part :)
01:04:40<anarcat>archiveteam operates a bot called archivebot which crawls websites and throws that into WARC files which is a "standard" (yeah those) for archiving websites
01:04:59<anarcat>if the site was crawled by archivebot, it's a good way to extract the entirety of that crawl
01:05:13<anarcat>it can also be used to perform authenticated crawls, wich proxies and plugins and so on
01:05:18<@OrIdow6>I wouldn't call this a deficiency on archive.org's part; it would not be possible for them to support all uses as niche as this
01:05:22<anarcat>because then you can reproduce a logged in user
01:05:37<anarcat>pabs: it's an "archive", not a "backup" :)
01:05:40<anarcat>anyways
01:05:45<anarcat>we're archiveteam here, not IA :)
01:05:49<pabs>:)
01:06:12<@OrIdow6>Yeah #internetarchive (also unofficial) might be best for future questions liek these
01:06:20<anarcat>right
01:06:23<anarcat>blame me
01:06:26<anarcat>i pointed pabs here :)
01:06:50<anarcat>then he blamed me for asking another question in the wrong channel, so we're even
01:06:55<anarcat>:)
01:06:57<anarcat>anyways, gn
01:07:08<pabs>later :)
01:10:16HP_Archivist (HP_Archivist) joins
01:14:00HP_Archivist quits [Client Quit]
01:15:26<@OrIdow6>I don't think (though I'm not of the strongest of opinions here) that it's wrong to put it in -ot, so much that it's on-topic and may as well be made more prominent in the right channel
01:26:07<Wayward->With the surface densities of a 1.44 MB 3-1/2" diskette, or 1.2 MB 5-1/4" floppy... how would they scale up to 12 TB capacity in inches?
01:27:04<Wayward->will also accept units in miles.
01:30:38Lord_Nightmare quits [Client Quit]
01:34:21Lord_Nightmare (Lord_Nightmare) joins
01:42:35<Doranwen>lol
01:49:50blankie quits [Remote host closed the connection]
02:04:49Lord_Nightmare quits [Client Quit]
02:06:40blankie joins
02:06:40blankie quits [Changing host]
02:06:40blankie (blankie) joins
02:07:09Lord_Nightmare (Lord_Nightmare) joins
02:13:10<G4te_Keep3r>yikes seek time. Really wish laserdisks would have taken off better-imagine one but with blueray density so holds...~200gb?
02:34:20Stilett0 quits [Ping timeout: 250 seconds]
02:56:57pabs leaves
03:04:00HP_Archivist (HP_Archivist) joins
03:20:11DogsRNice quits [Read error: Connection reset by peer]
03:25:49<@JAA>Don't worry. Holographic storage will be here soon!!1!
03:34:06<Wayward->any decade now
03:42:10<Wayward->an LD had ~6.25x the surface area of a CD/DVD/BR btw, so might have had LD-R's with a capacity of 312 GB
03:43:26blankie quits [Client Quit]
03:44:08blankie (blankie) joins
03:47:06qw3rty__ joins
03:50:55qw3rty_ quits [Ping timeout: 258 seconds]
04:02:06etnguyen03 quits [Client Quit]
04:28:23BlueMaxima_ joins
04:32:12BlueMaxima quits [Ping timeout: 250 seconds]
04:37:43blankie quits [Remote host closed the connection]
04:38:04Ruthalas quits [Ping timeout: 258 seconds]
04:38:14blankie joins
04:38:14blankie quits [Changing host]
04:38:14blankie (blankie) joins
04:43:34Krownest1 (Krownest) joins
04:47:16Krownest quits [Ping timeout: 258 seconds]
04:51:32Ruthalas (Ruthalas) joins
05:29:01Krownest1 is now known as Krownest
06:03:22blank_x joins
06:03:24blank_x quits [Client Quit]
06:03:33blankie quits [Client Quit]
06:04:00blankie (blankie) joins
06:40:25blankie quits [Remote host closed the connection]
06:45:59spelli joins
06:49:41blankie joins
06:49:41blankie quits [Changing host]
06:49:41blankie (blankie) joins
07:17:26LeighR (LeighR) joins
07:50:48Arcorann (Arcorann) joins
08:00:19sliccricc_ quits [Remote host closed the connection]
08:00:39sliccricc_ (sliccricc) joins
08:12:13nertzy_ joins
08:13:53nertzy__ quits [Ping timeout: 258 seconds]
08:23:58BlueMaxima__ joins
08:28:04BlueMaxima_ quits [Ping timeout: 258 seconds]
08:37:01hooway joins
08:37:17spelli quits [Remote host closed the connection]
08:39:30hooway_ joins
08:39:31hooway quits [Read error: Connection reset by peer]
09:25:05Ruthalas quits [Client Quit]
09:25:28Ruthalas (Ruthalas) joins
09:38:20NF885 (NF885) joins
10:43:34PlsNoJava quits [Ping timeout: 250 seconds]
10:52:52PlsNoJava (ROpdebee) joins
11:02:07BlueMaxima__ quits [Client Quit]
11:06:53yarrow (yarrow) joins
11:09:20tixma (tixma) joins
11:09:20VukkyWork (VukkyWork) joins
11:34:39VukkyWork quits [Remote host closed the connection]
11:39:28hooway joins
11:39:28hooway_ quits [Read error: Connection reset by peer]
11:51:20Zopolis4 quits [Ping timeout: 244 seconds]
11:52:08nertzy__ joins
11:54:41nertzy_ quits [Ping timeout: 258 seconds]
12:49:26etnguyen03 (etnguyen03) joins
13:15:02LeighR quits [Ping timeout: 244 seconds]
13:39:40Mateon1 quits [Remote host closed the connection]
13:39:56Mateon1 joins
14:31:23Starholme quits [Remote host closed the connection]
14:56:50LeighR (LeighR) joins
15:00:36sliccricc_ quits [Ping timeout: 258 seconds]
15:06:14hooway quits [Read error: Connection reset by peer]
15:06:30hooway joins
15:15:10mutantmnky quits [Ping timeout: 258 seconds]
15:24:52mutantmnky (mutantmonkey) joins
15:25:08Arcorann quits [Ping timeout: 258 seconds]
15:27:14rsn joins
15:29:25second (second) joins
15:30:55NF885 quits [Ping timeout: 244 seconds]
15:31:13sec^nd quits [Ping timeout: 255 seconds]
15:31:14second is now known as sec^nd
15:43:07IDK (IDK) joins
15:43:13<IDK>Hi
15:57:52NF885 (NF885) joins
15:59:04blankie quits [Remote host closed the connection]
16:00:35Vukky (Vukky) joins
16:30:41HP_Archivist quits [Ping timeout: 258 seconds]
16:44:48tixma quits [Ping timeout: 244 seconds]
16:55:08NF885 quits [Ping timeout: 244 seconds]
17:02:05Krownest1 (Krownest) joins
17:02:07HP_Archivist (HP_Archivist) joins
17:05:11Krownest quits [Ping timeout: 258 seconds]
17:06:44wickedplayer494 quits [Remote host closed the connection]
17:13:14wickedplayer494 joins
17:36:34cmlow quits [Quit: Connection closed for inactivity]
17:36:35Krownest1 is now known as Krownest
17:37:59serx joins
18:01:36Ruthalas5 (Ruthalas) joins
18:03:50Ruthalas quits [Ping timeout: 258 seconds]
18:03:50Ruthalas5 is now known as Ruthalas
18:33:07marked joins
19:08:09tixma joins
19:23:16DogsRNice (Webuser299) joins
19:45:04<AK>ping ping ping ping
19:47:20Inhonion joins
21:22:43<nyany>AK: ping ping pong
21:23:42hilda joins
21:26:37<m0nika>🏓
21:45:07serx quits [Remote host closed the connection]
21:51:26tixma quits [Remote host closed the connection]
22:29:30LeighR quits [Client Quit]
23:01:18Wayward- quits [Ping timeout: 258 seconds]
23:03:45hooway quits [Client Quit]
23:08:00BlueMaxima joins
23:15:42webdownload joins
23:15:45webdownload quits [Remote host closed the connection]
23:16:14webdownload joins
23:21:05webdownload quits [Ping timeout: 244 seconds]
23:28:17HP_Archivist quits [Client Quit]

HomeSearchPrevious dayNext day