| 01:18:08 | | DopefishJustin quits [Remote host closed the connection] |
| 01:21:16 | | DopefishJustin joins |
| 01:21:16 | | DopefishJustin is now authenticated as DopefishJustin |
| 03:03:47 | | qw3rty_ joins |
| 03:07:39 | | qw3rty quits [Ping timeout: 258 seconds] |
| 03:33:59 | | DogsRNice_ quits [Read error: Connection reset by peer] |
| 04:13:14 | <@JAA> | Huh, interesting, https://archive.org/details/stackexchange_20210607 has <mediatype>texts</mediatype> *and* <mediatype>data</mediatype>. Wut? |
| 04:14:24 | <@JAA> | Well, let's see what happens when it all settles down. |
| 04:15:42 | <@JAA> | Might be a bug in the item creation on copying as I created that with: ia copy stackexchange/3dprinting.meta.stackexchange.com.7z stackexchange_20210607/3dprinting.meta.stackexchange.com.7z --metadata=mediatype:data --metadata=[...] |
| 04:29:26 | <@JAA> | There are also some other oddities. No idea where <creator>StackExchange</creator> comes from, for example. I didn't set that, and it's not in the source item either. |
| 05:54:04 | | swebb quits [Ping timeout: 250 seconds] |
| 05:54:54 | | swebb joins |
| 06:00:45 | | Ryz quits [Remote host closed the connection] |
| 06:01:56 | | Ryz (Ryz) joins |
| 07:45:49 | | spirit joins |
| 11:44:31 | | Matthww quits [Quit: Ping timeout (120 seconds)] |
| 11:45:06 | | Matthww joins |
| 13:44:49 | | IDK (IDK) joins |
| 13:45:19 | <IDK> | is IA down right now because it says 503 and 404 if I click on save page now |
| 16:50:58 | <@JAA> | Well, my item's processing is all done, and that metadata's still there. Guess I'll file an issue later. Very weird behaviour. |
| 18:40:33 | <OrIdow6> | Getting 503s across the WBM right now |
| 18:47:32 | <HCross> | OrIdow6: Switch failure, IA aware |
| 18:48:56 | <OrIdow6> | Ah thanks HCross |
| 18:49:09 | <HCross> | same thing that hit IDK earlier |
| 18:52:08 | <OrIdow6> | Oh, I assumed that was just SPN |
| 19:15:29 | | DogsRNice (Webuser299) joins |
| 19:36:58 | | HP_Archivist (HP_Archivist) joins |
| 19:44:25 | | DogsRNice_ (Webuser299) joins |
| 19:47:23 | | DogsRNice quits [Ping timeout: 258 seconds] |
| 20:49:09 | | spirit quits [Client Quit] |
| 21:20:58 | | HP_Archivist quits [Read error: Connection reset by peer] |
| 21:21:20 | | HP_Archivist (HP_Archivist) joins |
| 22:35:40 | | Wayward quits [Ping timeout: 258 seconds] |
| 23:23:44 | <HP_Archivist> | Is there a command to download multiple ia items at once through the command line vs one by one? |
| 23:24:03 | <HP_Archivist> | I don't want to download an entire account's uploads. But do multiple items at once |
| 23:24:43 | <@JAA> | Was that 'don't' a typo? |
| 23:25:24 | <@JAA> | But yes, you can, with GNU Parallel, xargs, etc. |
| 23:26:44 | <@JAA> | For example: ia search --itemlist 'uploader:foo@example.org' | xargs -P 8 -n 1 ia download |
| 23:27:16 | <HP_Archivist> | Not a typo JAA |
| 23:27:20 | <HP_Archivist> | Although would that be easier? |
| 23:27:24 | <HP_Archivist> | I don't know anything about xargs |
| 23:27:41 | <@JAA> | xargs just reads lines from stdin and supplies them as arguments to whatever follows. |
| 23:27:56 | <@JAA> | -P 8 = 8 parallel processes, -n 1 = one line/argument per command |
| 23:28:31 | <@JAA> | So that example runs 8 `ia download <identifier>` processes in parallel, using the item identifier list produced by `ia search`. |
| 23:29:03 | <HP_Archivist> | Well, under the my account I have 263 items |
| 23:29:04 | <@JAA> | You can also take that identifier list from somewhere else of course, e.g. a file: xargs -P 8 -n 1 ia download <identifiers |
| 23:29:08 | <HP_Archivist> | But yeah that makes sense |
| 23:29:24 | <@JAA> | Note that the files in each item are still downloaded sequentially with this. |
| 23:29:53 | <@JAA> | If you want to parallelise over that, you'll want something like this: https://github.com/jjjake/internetarchive/issues/412#issuecomment-857989394 |
| 23:30:20 | <@JAA> | Parallelising both items and files in items is a bit more complicated but can also be done. |
| 23:31:06 | <HP_Archivist> | Well what I was going to do was grab my item urls and download one at a time... but that's pretty tedious |
| 23:31:41 | <HP_Archivist> | Is it worth trying to just download all the entire account? or is it not that simple |
| 23:31:55 | <HP_Archivist> | all items from the entire account* |
| 23:32:01 | <@JAA> | I thought that's what you *don't* want to do? lol |
| 23:32:09 | <HP_Archivist> | Yes. I'm trying to make up my mind heh |
| 23:32:22 | <HP_Archivist> | The only reason why I'm hesitant is I'm worried I'll exceed my data cap, probably would |
| 23:32:26 | <@JAA> | It's easy. The search+download command above will do exactly that. |
| 23:33:02 | <@JAA> | Ah yes, one of those poor souls with a data cap. |
| 23:33:04 | <HP_Archivist> | For an extra $50 I can get unlimited with my ISP. But I already pay over $100 just to get decent upstream |
| 23:33:16 | <HP_Archivist> | Yup |
| 23:34:23 | <HP_Archivist> | So it would be ia download @harry_potter_archive ? |
| 23:34:29 | <HP_Archivist> | That doesn't look right |
| 23:34:41 | <HP_Archivist> | Oh you said search |
| 23:37:16 | <@JAA> | `ia download --search 'uploader:foo@example.org'` should work, but that'd be sequential. |
| 23:37:27 | <@JAA> | You need to use the email address, not the account name. |
| 23:39:09 | <HP_Archivist> | I'm just thinking here..if my connection were to timeout or drop for example, can I start off on the same item or I would I have to start all over? I would think it would retry the last tried item |
| 23:39:20 | <HP_Archivist> | There's like 3+TBs under my account |
| 23:39:29 | <HP_Archivist> | And noted for the use of my email vs account name |
| 23:40:34 | <@JAA> | I think `ia download` has an option to skip existing files or even does that by default, no? |
| 23:41:56 | <HP_Archivist> | I have to check the --help again but yeah I think you're right. I'll see what I can do. |
| 23:42:01 | <HP_Archivist> | Thanks JAA |
| 23:42:13 | <@JAA> | There's --checksum, but not sure what the default behaviour is. |
| 23:42:58 | <@JAA> | I think the order is `ia download identifier --option` though, so that needs some extra stuff on xargs. |
| 23:43:54 | <@JAA> | xargs -P 8 -n 1 -i ia download {} --checksum |
| 23:46:41 | <HP_Archivist> | I'll have a look through. Really, what makes the data so large is 'video archive' items e.g., game footage and long play walkthroughs. Those items top out at 100GBs each or a little less and there are 31 items like that |
| 23:47:00 | <HP_Archivist> | So if I could figure a way to exclude those for now and download the rest of the items I think that would be more ideal |
| 23:47:28 | <HP_Archivist> | Which is why I was going the one item at a time route. But that's gonna take forever and a day |
| 23:49:08 | <HP_Archivist> | I lost source on those videos due to a drive failure about 6 months ago. I have most source on the rest of the items. But I have updated items on Archive and didn't replicate those updates locally. Things become messy after a while. |
| 23:49:38 | <@JAA> | Can you filter them by mediatype or some other field? |
| 23:50:24 | <HP_Archivist> | Well video items are under 'Video Game Replay' collection, wonder if I could filter using that |
| 23:51:08 | <@JAA> | Try adding a -collection:game_replays filter to your search. |
| 23:52:30 | <HP_Archivist> | Using that will exclude items under that collection from being downloaded? |
| 23:52:56 | <@JAA> | Yep |
| 23:52:58 | <HP_Archivist> | I want to eventually download all of them. But the video items can wait. |
| 23:54:42 | | HP_Archivist quits [Read error: Connection reset by peer] |
| 23:55:06 | | HP_Archivist (HP_Archivist) joins |
| 23:55:24 | <HP_Archivist> | Weird. I was kicked. |
| 23:55:31 | <HP_Archivist> | Connection dropped |
| 23:56:03 | <HP_Archivist> | JAA is this correct? ia download --search uploader:harrypotterarchival@gmail.com -collection:game_replays |
| 23:59:11 | <@JAA> | HP_Archivist: Quotes around the search query: ia download --search 'uploader:harrypotterarchival@gmail.com -collection:game_replays' |