HomeSearchPrevious dayNext day

00:01:11dm4v_ joins
00:01:11dm4v quits [Read error: Connection reset by peer]
00:01:23dm4v_ is now known as dm4v
00:01:25dm4v quits [Changing host]
00:01:25dm4v (dm4v) joins
00:07:17BlueMaxima joins
00:50:36Arcorann (Arcorann) joins
01:02:48dm4v quits [Read error: Connection reset by peer]
01:03:00dm4v joins
01:03:02dm4v quits [Changing host]
01:03:02dm4v (dm4v) joins
01:52:16bitbit (bitbit) joins
01:57:37<bitbit>Hi. I've read the documentation of the ia cli and internetarchive python library, but I haven't found an answer -- when I upload a multiple files to the same item identifier, can I upload them at separate times? like one now and one in an a couple of hours from now?
02:02:49<wolfin>bitbit: I've done this before as long as it's the same identifier. I have elevated permissions but I believe all accounts can do this as long as you use the same account for each operation.
02:03:00qwertyasdfuiopghjkl quits [Client Quit]
02:03:45qwertyasdfuiopghjkl joins
02:04:14<bitbit>thanks
02:33:03<bitbit>another question -- can a single identifier hold 100,000 files?
03:15:15<@OrIdow6>Maybe, but it's a bad idea
03:16:05<wolfin>I feel there's an xy problem here ( xyproblem.info )
03:16:15<wolfin>What are you trying to do, bitbit?
03:17:55<@OrIdow6>https://xyproblem.info/ - guide to writing an annoying SO answer
03:22:50knecht420 quits [Client Quit]
03:23:02knecht420 (knecht420) joins
03:40:29qw3rty_ joins
03:43:58qw3rty__ quits [Ping timeout: 252 seconds]
03:46:58Mayk78 quits [Ping timeout: 250 seconds]
03:48:31Mayk78 joins
04:14:01<bitbit>I want to slice a large dataset by its row (every X rows) and columns, this would produce 100k files. I want to hold onto only key certain columns on my machine, in order to query them from time to time. this query would produce a list of relevant rows. I want to then download only certain sliced column files, and only those that correspond to the relevant rows
04:14:39<bitbit>I suppose I can trim it down to 30k files or maybe less, I was just being lazy and also wondering if there's a limit. would 30k be reasonable?
04:15:56<bitbit>s/by its row (every X rows)/by its rows (every N-N+2000 rows)/
04:17:38<bitbit>OrIdow6: wolfin: ^ :)
04:19:23<wolfin>Aha, and this is knowledge you also want to give freely to the world? I want to make sure you're not using archive.org as a cheap personal cloud XD
04:21:39<bitbit>yes :) it's a youtube dataset so I think it's pretty useful but it's current version on IA is 1TB it takes like a week to download
04:22:13<bitbit>and most people, myself included, can't spare 1TB for something they use only once in a while
04:23:00<wolfin>That makes a lot of sense ^.^ So even though this isn't ideal it would work, but if you can sort your dataset at all I'd encourage you to subset it further into multiple identifiers
04:23:26<wolfin>Maybe a youtubedataseta, youtubedatasetb; or by the first 1 or 2 letters of some hex value
04:24:06<wolfin>the reason why is the scripts that run archive.org don't do well when a single identifiers gets really large, and I'd worry about breakage or data rot if it struggle to check the integrity of a large set.
04:24:23<wolfin>Archive.org is...largely a bunch of php held together with bailing wire.
04:25:41<wolfin>I'd try and target no one identifier being bigger than 50GB, if possible.
04:26:48<bitbit>haha alright. well I think if I'm not being lazy I can batch the files by max size (e.g. 200MB) instead of slicing it by a fixed number of rows. that would make it way smaller
04:27:44<bitbit>no bigger than 50GB? that is strange. so what of this 1TB dataset on IA? it's in danger of data rot?
04:28:52<bitbit>there's actually another which is 750GB that is important to me as well. are you sure it's accurate that archive.org scripts can't check integrity of larger files than ~50GB?
04:29:05<bitbit>s/750GB/720GB/
04:29:07<wolfin>That should work! I'd also reach out to https://archive-beta.ics.uci.edu/ml/donation-policy
04:30:24<wolfin>bitbit, the concern of having 1TB in a single identifier is it's all going to rest on one petabox, and if that machine becomes unhealthy a set that large will be harder to move away or recover for the automated tools that keep the lights on.
04:30:33<bitbit>or maybe you meant no one file should be larger than 50GB, rather than that all files in a certain identifier should be max 50GB
04:30:54<bitbit>oh I see
04:34:34<bitbit>thanks wolfin I'll see what I can do!
04:35:43<wolfin>You're welcome!
05:04:09HP_Archivist (HP_Archivist) joins
05:20:35qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds]
05:25:22jut quits [Quit: Goodbey]
05:57:29bitbit quits [Client Quit]
06:22:24Earendil (Cobalt17) joins
07:20:11<@OrIdow6>Donation banner fairly similar to IA and WMF https://web.archive.org/web/20201127075503/https://www.law.cornell.edu/supct/html/03-1454.ZS.html
07:41:47BlueMaxima_ joins
07:45:46BlueMaxima quits [Ping timeout: 244 seconds]
08:53:04Matthww8 quits [Client Quit]
09:02:25Matthww8 joins
09:02:47BlueMaxima__ joins
09:06:46BlueMaxima_ quits [Ping timeout: 250 seconds]
09:18:02HP_Archivist quits [Ping timeout: 250 seconds]
09:43:36swebb quits [Ping timeout: 250 seconds]
09:45:26swebb joins
09:56:19BlueMaxima__ quits [Client Quit]
11:30:36Mateon2 joins
11:32:34Mateon1 quits [Ping timeout: 252 seconds]
11:32:34Mateon2 is now known as Mateon1
12:11:04swebb quits [Ping timeout: 252 seconds]
12:20:30IDK quits [Client Quit]
12:25:25swebb joins
12:48:03<nyany>OrIdow6: I was going to say, obnoxiously large and takes up almost an entire page of content?
12:48:10<nyany>in re WMF
12:48:52<nyany>This one actually isn't horrible, it's only half of a page for me, but WMF ones are typically massive here
13:02:04flashmeow quits [Ping timeout: 250 seconds]
13:36:44@arkiver quits [Ping timeout: 250 seconds]
13:36:56arkiver (arkiver) joins
13:36:56@ChanServ sets mode: +o arkiver
13:57:58Video quits [Ping timeout: 250 seconds]
15:04:16Arcorann quits [Ping timeout: 250 seconds]
15:48:20britm0b quits [Read error: Connection reset by peer]
16:15:55Craigle quits [Quit: Ping timeout (120 seconds)]
16:19:04Craigle (Craigle) joins
16:20:16britmob joins
16:21:21Craigle quits [Client Quit]
16:24:27britmob quits [Read error: Connection reset by peer]
16:24:46Craigle (Craigle) joins
16:25:51@arkiver quits [Client Quit]
16:26:28arkiver (arkiver) joins
16:26:28@ChanServ sets mode: +o arkiver
16:27:07britmob joins
16:27:22@arkiver quits [Max SendQ exceeded]
16:27:40nothere_ quits [Quit: Leaving]
16:27:46Craigle quits [Client Quit]
16:28:01arkiver (arkiver) joins
16:28:01@ChanServ sets mode: +o arkiver
16:29:48@arkiver quits [Max SendQ exceeded]
16:30:44arkiver (arkiver) joins
16:30:44@ChanServ sets mode: +o arkiver
16:31:09@arkiver quits [Max SendQ exceeded]
16:35:52arkiver (arkiver) joins
16:35:52@ChanServ sets mode: +o arkiver
16:36:29@arkiver quits [Max SendQ exceeded]
16:37:05arkiver (arkiver) joins
16:37:05@ChanServ sets mode: +o arkiver
16:37:33Earendil quits [Read error: Connection reset by peer]
16:37:44Earendil (Cobalt17) joins
16:38:35nothere joins
16:41:32@arkiver quits [Max SendQ exceeded]
16:42:50nothere quits [Client Quit]
16:46:47arkiver (arkiver) joins
16:46:47@ChanServ sets mode: +o arkiver
16:49:05nothere joins
16:49:53@arkiver quits [Max SendQ exceeded]
16:51:57arkiver (arkiver) joins
16:51:57@ChanServ sets mode: +o arkiver
16:52:03nothere quits [Client Quit]
16:52:25@arkiver quits [Max SendQ exceeded]
16:53:24arkiver (arkiver) joins
16:53:24@ChanServ sets mode: +o arkiver
16:53:27nothere joins
16:55:33@arkiver quits [Max SendQ exceeded]
16:59:52nothere quits [Client Quit]
17:08:51sembiance quits [Remote host closed the connection]
17:10:33Craigle (Craigle) joins
17:14:16Core8769 (Cobalt17) joins
17:17:13Earendil quits [Ping timeout: 244 seconds]
17:40:14LeGoupil joins
18:15:09HP_Archivist (HP_Archivist) joins
18:34:51Earendil (Cobalt17) joins
18:38:19Core8769 quits [Ping timeout: 244 seconds]
19:29:17systwi (systwi) joins
19:46:06LeGoupil quits [Client Quit]
20:07:43Hackerpcs quits [Quit: Hackerpcs]
20:08:58Hackerpcs (Hackerpcs) joins
20:15:58systwi quits [Ping timeout: 244 seconds]
20:20:36Stilett0 quits [Ping timeout: 250 seconds]
20:21:20Stiletto joins
20:24:00systwi (systwi) joins
20:32:50Core7424 (Cobalt17) joins
20:34:22Stilett0 joins
20:34:52Stiletto quits [Ping timeout: 252 seconds]
20:36:12Earendil quits [Ping timeout: 250 seconds]
20:40:29Stiletto joins
20:40:46Stilett0 quits [Ping timeout: 244 seconds]
20:44:40Earendil (Cobalt17) joins
20:44:52Stiletto quits [Ping timeout: 250 seconds]
20:45:35Stiletto joins
20:48:00Core7424 quits [Ping timeout: 244 seconds]
21:00:24Core6852 (Cobalt17) joins
21:02:46Stilett0 joins
21:04:22Earendil quits [Ping timeout: 250 seconds]
21:05:14Stiletto quits [Ping timeout: 250 seconds]
21:58:47britmob quits [Read error: Connection reset by peer]
22:21:16thuban quits [Quit: What price Glory ?]
22:31:57driib1 (driib) joins
22:35:48driib quits [Ping timeout: 250 seconds]
22:35:48driib1 is now known as driib
22:35:49Earendil (Cobalt17) joins
22:37:58Core6852 quits [Ping timeout: 250 seconds]
22:39:35Ruthalas7 (Ruthalas) joins
22:40:39Ruthalas quits [Read error: Connection reset by peer]
22:40:39Ruthalas7 is now known as Ruthalas
23:00:35Core1232 (Cobalt17) joins
23:00:53Ruthalas0 (Ruthalas) joins
23:02:16Ruthalas quits [Ping timeout: 252 seconds]
23:02:16Earendil quits [Ping timeout: 252 seconds]
23:02:16Ruthalas0 is now known as Ruthalas
23:43:50lunik1 quits [Quit: :x]
23:44:19lunik1 joins

HomeSearchPrevious dayNext day