| 02:21:30 | | sonick quits [Client Quit] |
| 03:46:02 | | Stiletto quits [Read error: Connection reset by peer] |
| 03:46:14 | | Stiletto joins |
| 04:26:21 | | qwertyasdfuiopghjkl joins |
| 07:08:26 | | Stiletto quits [Ping timeout: 240 seconds] |
| 07:25:06 | | Stiletto joins |
| 09:01:52 | | sonick (sonick) joins |
| 12:55:58 | | Sluggs quits [Quit: Going offline, see ya! (www.adiirc.com)] |
| 13:02:46 | | Arcorann quits [Ping timeout: 240 seconds] |
| 13:53:46 | | sonick quits [Client Quit] |
| 14:41:12 | | sonick (sonick) joins |
| 16:39:27 | | Sluggs joins |
| 18:57:41 | | Wayward quits [Ping timeout: 265 seconds] |
| 18:58:49 | | sallysue joins |
| 18:59:06 | | protondonor joins |
| 19:00:31 | <sallysue> | hello! |
| 19:01:00 | <sallysue> | i want to archive a large amount of webpages to the wayback machine, and i was wondering if there was a faster way than submitting individual pages |
| 19:01:27 | <sallysue> | my current strategy is to use the waybackpy library to iterate through lists of URLs and save them one by one |
| 19:01:33 | <britmob|m> | Is it a whole site? Or just a large collection of links from various sites? |
| 19:01:48 | <sallysue> | both |
| 19:02:38 | <sallysue> | i can usually get up to roughly five simultaneous threads running, but i am not sure what the actual rate limit is |
| 19:03:03 | <britmob|m> | if it’s a whole site that has some time sensitivity, archiveteam can handle that with archivebot. as far as random various links that’s likely as quick as you can go |
| 19:04:06 | <sallysue> | that is somewhat unfortunate to hear |
| 19:04:18 | <sallysue> | perhaps i should spin up a few more servers to increase my bandwidth |
| 19:04:55 | <@JAA> | Yeah, the WBM is not really made for mass archival like that. It's already overloaded frequently as well. |
| 19:06:05 | <sallysue> | would it be better to capture WARC files locally and submit them to the internet archive separately from the wayback machine? |
| 19:08:32 | <@JAA> | Well, they wouldn't go into the Wayback Machine like that. |
| 19:08:58 | <@JAA> | Only data from trusted parties is accepted into the WBM index because anyone could falsify history otherwise. |
| 19:09:03 | <sallysue> | i understand |
| 19:09:11 | <sallysue> | the goal is just to preserve as many webpages as possible |
| 19:14:40 | <@JAA> | If it's something that's valuable, it might be best to archive it through ArchiveTeam so it does go into the WBM but isn't horribly inefficient. |
| 19:15:08 | <sallysue> | efficiency is a rather mild concern |
| 19:15:12 | <sallysue> | i just like to save things |
| 19:15:35 | <@JAA> | Random WARCs are better than nothing, but they will almost definitely never be looked at by anyone. |
| 19:16:09 | <sallysue> | it sounds like the best thing for me to do is to just keep running my scripts |
| 19:16:14 | <sallysue> | actually, i have another question |
| 19:16:34 | <sallysue> | when i submit pages to be saved through https://web.archive.org/save, there is an option to save outlinks |
| 19:16:48 | <sallysue> | can this be done for automated submissions as well? |
| 19:45:50 | <@JAA> | That's only available when you're logged in. waybackpy doesn't seem to support that at all. |
| 19:50:15 | <@JAA> | Also, waybackpy uses SPN, not the SPN2 API (which isn't publicly documented). I'm pretty sure it wouldn't save images, scripts, etc., just the actual URL you're feeding it. |
| 19:50:33 | <@JAA> | Unless that got changed in the SPN backend since. |
| 19:55:07 | <sallysue> | sounds like i need to do some experimentation with the SPN2 API |
| 19:57:38 | <sallysue> | would capturing POST data and swapping out the URL work? |
| 20:06:43 | | sallysue quits [Ping timeout: 244 seconds] |
| 20:52:22 | | DogsRNice (Webuser299) joins |
| 20:56:15 | | xit78 quits [Ping timeout: 252 seconds] |
| 20:57:40 | | qwertyasdfuiopghjkl quits [Remote host closed the connection] |
| 21:09:13 | | sallysue joins |
| 21:14:12 | | qwertyasdfuiopghjkl joins |
| 21:26:39 | | qwertyasdfuiopghjkl quits [Remote host closed the connection] |
| 22:33:34 | | Arcorann (Arcorann) joins |
| 23:44:18 | | Arcorann quits [Ping timeout: 265 seconds] |
| 23:46:55 | | Arcorann (Arcorann) joins |