| 00:33:01 | | dxrt_ is now known as dxrt |
| 00:33:02 | | dxrt is now authenticated as dxrt |
| 00:33:02 | | dxrt quits [Changing host] |
| 00:33:02 | | dxrt (dxrt) joins |
| 00:52:41 | | dxrt quits [Client Quit] |
| 00:52:59 | | dxrt joins |
| 00:53:01 | | dxrt is now authenticated as dxrt |
| 00:53:01 | | dxrt quits [Changing host] |
| 00:53:01 | | dxrt (dxrt) joins |
| 01:10:43 | | prebala-publici joins |
| 01:32:03 | | themadpro joins |
| 03:54:41 | <tech234a> | I'm almost ready to upload the YouTube Community Contributions data to IA. It has 1,361,998 files across 406,394 folders, and is 9.46 GB in size (11.7 GB size on disk). I assume putting that number of files and folders uncompressed into one item isn't advisable. How should I package the data for upload? |
| 03:59:39 | <OrIdow6^2> | 9.46 is plenty small enough for a single item |
| 03:59:45 | <OrIdow6^2> | Just put it in a tar or something |
| 04:00:02 | <OrIdow6^2> | *9.46 GB |
| 04:00:29 | <tech234a> | Alright, I was more worried about the folder count |
| 04:00:34 | <OrIdow6^2> | If you want people to be able to access the files individually, use an uncompressed tar, or something like zip that has an index at the beginning |
| 04:01:17 | <@JAA> | Is there some reasonable way to split it up into smaller groups? One tar or zip with that many files and directories is not going to work very well I think. |
| 04:01:36 | <@JAA> | I mean, it'll be fine for upload and download, but the viewer on IA won't like it most likely. |
| 04:01:55 | <tech234a> | It looks like I can divide it by first two characters of a the ID and it will be ~400 folders per grouping |
| 04:02:49 | <tech234a> | Probably it would be one item with ~4032 zip files in it then |
| 04:03:14 | <@JAA> | Hmm, that on the other hand sounds like a bit too many files in one item again. |
| 04:03:15 | <OrIdow6^2> | Oh, you suppose the viewer displays |
| 04:03:18 | <@JAA> | Maybe just one character? |
| 04:03:24 | <OrIdow6^2> | Whoops, did not mean to hit return |
| 04:04:35 | <OrIdow6^2> | *Oh, you're right, suppose the viewer displays them all at once |
| 04:04:53 | <tech234a> | One character per item would be ~64 zips per item |
| 04:05:24 | <tech234a> | And I guess the item IDs can be sequential and I would be able to request a collection be created? |
| 04:06:55 | <@JAA> | That would work I guess. I meant all in one item, but one tar/zip per character. That'd work out to ~6.5k directories per tar. Not sure how much the viewer can handle, but if the files are small enough, that might be fine. |
| 04:08:27 | <@JAA> | Could always try that first and then split it up further if it doesn't suffice. |
| 04:09:36 | <OrIdow6^2> | I think the viewer can handle enough that it's nearly crashed my browser in the past |
| 04:09:59 | <tech234a> | Yeah I think browser crashing is the problem I've had with large items previously |
| 04:10:10 | | qw3rty_ joins |
| 04:10:36 | <tech234a> | Can the viewer display the contents of zip files within zip files? |
| 04:10:50 | <tech234a> | I'd like to be able to link to specific files from the index search page |
| 04:13:24 | | qw3rty quits [Ping timeout: 240 seconds] |
| 04:14:26 | <tech234a> | Well, I'll try doing a zip within a zip within one item |
| 04:15:07 | <@JAA> | Just found an item with a zip in a tar, and that doesn't work. |
| 04:15:13 | <@JAA> | So I assume zip in zip doesn't either. |
| 04:17:42 | <tech234a> | Hmm... |
| 04:17:54 | <@JAA> | That tar has 17k files and opens fine in the viewer, although the browser doesn't like it all that much. |
| 04:18:21 | <@JAA> | For reference, it's this: https://archive.org/download/ftp-70.58.96.236 |
| 04:20:11 | <OrIdow6^2> | https://archive.org/download/youtubeannotations_21/V5.tar looks to have about 300k files, and it lists, though it takes up about 1.2 GB of memory in browser |
| 04:20:22 | <@JAA> | That's somewhat comparable to when you'd split it up into one tar per character prefix, which would end up at 21k files or so (and 6.5k dirs). |
| 04:21:19 | <tech234a> | Perhaps I’ll just do one zip per first character then |
| 04:22:05 | <tech234a> | That should generally mean less than 2,000 files per zip |
| 04:32:49 | <tech234a> | Looked at a different character and found a much high file count... I think I'll try one folder for each first character, and one zip for each first two characters |
| 05:39:35 | | DogsRNice quits [Read error: Connection reset by peer] |
| 08:49:16 | | @arkiver quits [Quit: .] |
| 08:49:30 | | arkiver (arkiver) joins |
| 08:49:30 | | @ChanServ sets mode: +o arkiver |
| 14:37:26 | | Arcorann quits [Ping timeout: 244 seconds] |
| 14:43:05 | | dxrt quits [Client Quit] |
| 14:43:23 | | dxrt joins |
| 14:43:25 | | dxrt is now authenticated as dxrt |
| 14:43:26 | | dxrt quits [Changing host] |
| 14:43:26 | | dxrt (dxrt) joins |
| 15:13:51 | | themadpro is now known as themadpro_ |
| 15:14:02 | | themadpro_ is now known as themadpro |
| 15:15:10 | | themadpro is now authenticated as themadpro |
| 19:47:55 | | prebala-publici quits [Client Quit] |
| 19:48:15 | | prebala-publici joins |
| 19:52:26 | | prebala-publici quits [Client Quit] |
| 19:52:45 | | prebala-publici joins |
| 19:57:30 | | prebala-publici quits [Client Quit] |
| 19:57:50 | | prebala-publici joins |
| 20:06:56 | | prebala-publici quits [Client Quit] |
| 20:07:44 | | prebala-publici_ joins |
| 20:11:46 | | prebala-publici_ quits [Read error: Connection reset by peer] |
| 20:12:07 | | prebala-publici joins |
| 21:58:45 | | prebala-publici quits [Read error: Connection reset by peer] |
| 23:22:20 | <tech234a> | Alright, files are now uploading after fixing some problems with case-insensitivity. I had to make the data directory case sensitive and use the `-ssc` option for 7-Zip. (Apparently Windows allows making individual directories case-sensitive using `fsutil file setCaseSensitiveInfo . enable`.) |
| 23:23:03 | <tech234a> | The upload is going rather slowly... is there any way to perhaps upload multiple files at once? Right now I |
| 23:23:08 | <tech234a> | Right now I |
| 23:23:17 | <tech234a> | Right now I'm using `ia upload youtube-community-contributions .` |
| 23:23:48 | <tech234a> | (whoops hit enter instead of the apostrophe) |
| 23:30:17 | <@JAA> | Imagine using Windows... ;-) |
| 23:31:10 | <@JAA> | You can run multiple `ia upload`, each uploading some distinct subset of files, but it's generally recommended against because it can cause problems. It's probably fine in this case though. |
| 23:31:34 | <tech234a> | Alright, thanks, will consider it |