| 00:19:10 | | ArchivalEfforts quits [Ping timeout: 252 seconds] |
| 00:22:14 | | ArchivalEfforts joins |
| 00:27:48 | <fullpwnmedia> | can i increase the maximum amount of concurrent items above 6? |
| 00:31:57 | <@JAA> | fullpwnmedia: Not in the warrior. In the standalone project images, you can go up to 20. |
| 00:32:41 | <TheTechRobo> | (The limit is 20 because weird stuff happens when you go over, I've been told.) |
| 00:32:52 | <TheTechRobo> | (Though I would like details. :-) |
| 00:33:47 | <@JAA> | Something something race condition somewhere deep in seesaw. Nobody ever figured out *what* actually goes wrong I believe. |
| 00:34:14 | <TheTechRobo> | Weird how it only happens >20. Maybe it is happening and corrupting data as we speak... o_O |
| 00:36:36 | <fullpwnmedia> | yeah but how can i edit warrior to go to for example 15 |
| 00:36:45 | <@JAA> | I think that's just the point where the effects become too bad. |
| 00:37:00 | <TheTechRobo> | fullpwnmedia: You don't. |
| 00:37:06 | <fullpwnmedia> | damn |
| 00:37:07 | <TheTechRobo> | Run the docker images individually if you need to |
| 00:37:21 | <fullpwnmedia> | would 10 be like too much |
| 00:37:41 | <TheTechRobo> | (You might be able to, but modifying stuff is generally frowned upon here as you might accidentally introduce data integrity issues, etc.) |
| 00:37:43 | <@JAA> | There's probably a reason for the limit of 6, but it predates my presence here. |
| 00:37:46 | <tech234a> | docker instructions which enable up to 20 concurrent: https://wiki.archiveteam.org/index.php/Running_Archive_Team_Projects_with_Docker |
| 00:38:18 | <TheTechRobo> | JAA: Seems likely enough that people just figured that if you want more, just use the standalone stuff, since the Warrior is more a set it and forget it thing. |
| 00:38:30 | <@JAA> | Yeah, that as well. |
| 00:39:09 | <fullpwnmedia> | if its gonna cause like |
| 00:39:11 | <fullpwnmedia> | issues |
| 00:39:16 | <fullpwnmedia> | then nvm |
| 00:39:36 | <fullpwnmedia> | my archiving machine is a potato anyway |
| 00:40:00 | <tech234a> | JAA: I think I remember hearing something along the lines that people were running projects a high concurrency than necessary for a given project and/or running themselves into ratelimits for specific projects |
| 00:40:25 | <@JAA> | Ah yes, also a reasonable explanation. |
| 00:40:47 | <tech234a> | I think in addition to the splitting Warrior idea mentioned in #warrior, there should be a way to specify a maximum allowed concurrency for each project |
| 00:40:48 | <fullpwnmedia> | ooh yikeas |
| 00:41:12 | <@JAA> | Disk space when a project with large items becomes the default may also be a problem at higher concurrencies if the user isn't aware of it. |
| 00:41:24 | <TheTechRobo> | It'd definitely be nice if the Warrior could work on another selected project when there are no items available. |
| 00:41:30 | <fullpwnmedia> | how big is the hard disk anyway |
| 00:42:26 | <tech234a> | "The virtual machine has a hard limit of 60GB disk usage" from https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior#How_much_disk_space_will_the_Warrior_use? |
| 00:42:47 | <tech234a> | Docker doesn't have a limit though |
| 00:42:50 | <@JAA> | Depends how you run things. I think the VM appliance has something like 50 or 60 GiB by default? The Docker image has no restrictions and can probably fill the partition it's on. |
| 00:43:00 | <@JAA> | Damn ninjas! |
| 00:43:06 | <fullpwnmedia> | and im assuming after its done uploading from warrior it wipes it from the disk? |
| 00:43:17 | <@JAA> | Yes |
| 00:43:23 | <fullpwnmedia> | gotcha |
| 00:43:48 | <@JAA> | Though if you use the VM, the virtual disk file will not get trimmed automatically. |
| 00:45:12 | | wickedplayer494 quits [Ping timeout: 252 seconds] |
| 00:52:11 | | wickedplayer494 joins |
| 00:52:27 | | wickedplayer494 is now authenticated as wickedplayer494 |
| 01:05:09 | | Arcorann (Arcorann) joins |
| 01:48:19 | <fullpwnmedia> | we broke dpreview |
| 01:48:27 | <fullpwnmedia> | oops |
| 01:49:22 | <@JAA> | Project-specific talk should go in the project-specific channel. |
| 01:49:37 | <fullpwnmedia> | mb |
| 01:53:25 | <cm> | oi, is there an official generic term for an Internet Archive entry? |
| 01:53:53 | <@JAA> | 'Item' |
| 01:53:54 | <cm> | as in, the <something> that is referred to by https://archive.org/details/<something> |
| 01:54:18 | <cm> | so a collection of many files would still be an item? |
| 01:54:19 | <@JAA> | Ah, then 'identifier' |
| 01:54:38 | <cm> | well the <something> is the identifier, but it identifies an item? |
| 01:54:42 | <@JAA> | Yep |
| 01:54:47 | <@JAA> | And yes, collections are items. |
| 01:54:52 | <cm> | cool cool ty |
| 01:55:01 | <@JAA> | In the same way that directories are files on Unixoid systems. |
| 01:55:17 | <cm> | have to do the most writing a readme that will be written to tape lmao |
| 04:20:16 | | nicolas17 quits [Read error: Connection reset by peer] |
| 04:20:45 | | nicolas17 joins |
| 04:28:26 | <@OrIdow6> | I think it's a bad idea to have these Dvorak-keyboard style alternatives to WARC |
| 04:29:03 | <@OrIdow6> | Standards fragmentation is bad enough in stuff thousands of people have interest in now... would be horrible for something intended to last hundreds of years |
| 04:29:47 | <@OrIdow6> | Like if books changed their language every few decades |
| 04:29:58 | <@OrIdow6> | Though I suppose the non-archiving web area is doing that for us |
| 04:30:36 | <nicolas17> | what's wrong with WARC apart from the available tools? |
| 04:30:49 | <nicolas17> | the format itself seemed reasonable on a first look |
| 04:31:29 | <@OrIdow6> | That was in reply to backlog from yesterday |
| 04:31:52 | <@OrIdow6> | And yeah, I don't think it's so bad |
| 04:32:41 | <nicolas17> | I'm thinking of making my own script that will output WARC so I guess I'll find out if the format sucks |
| 04:34:35 | <nicolas17> | I want to archive thousands of POST requests/responses, I don't think I can coerce wget into doing that... wget does support POST but it would be one per call |
| 04:59:05 | | nicolas17 quits [Client Quit] |
| 05:09:12 | | Island quits [Read error: Connection reset by peer] |
| 05:15:48 | | dvd quits [Remote host closed the connection] |
| 05:17:26 | | dvd joins |
| 05:21:15 | | dvd_ joins |
| 05:21:23 | | dvd quits [Remote host closed the connection] |
| 05:44:33 | | BlueMaxima quits [Client Quit] |
| 06:14:33 | | umgr036 joins |
| 06:15:20 | | umgr036 quits [Remote host closed the connection] |
| 06:15:33 | | umgr036 joins |
| 07:43:02 | | Dj-Wawa quits [Remote host closed the connection] |
| 07:43:02 | | qwertyasdfuiopghjkl quits [Client Quit] |
| 07:43:52 | | Dj-Wawa (Dj-Wawa) joins |
| 08:21:30 | | Hackerpcs quits [Quit: Hackerpcs] |
| 08:24:40 | | Hackerpcs (Hackerpcs) joins |
| 09:44:34 | | dvd_ quits [Ping timeout: 252 seconds] |
| 09:56:18 | | qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins |
| 11:06:19 | | qwertyasdfuiopghjkl quits [Client Quit] |
| 12:38:43 | | HP_Archivist (HP_Archivist) joins |
| 14:03:04 | | Arcorann quits [Ping timeout: 252 seconds] |
| 14:46:23 | | HP_Archivist quits [Remote host closed the connection] |
| 14:46:32 | | HP_Archivist (HP_Archivist) joins |
| 14:55:49 | | monoxane (monoxane) joins |
| 16:25:21 | | umgr036 quits [Remote host closed the connection] |
| 16:25:35 | | umgr036 joins |
| 16:30:59 | | monoxane quits [Client Quit] |
| 16:31:19 | | HP_Archivist quits [Remote host closed the connection] |
| 16:51:00 | | qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins |
| 17:01:03 | <h2ibot> | JAABot edited CurrentWarriorProject (+0): https://wiki.archiveteam.org/?diff=49659&oldid=49658 |
| 17:02:03 | | dvd joins |
| 18:04:13 | | hitgrr8 joins |
| 18:05:06 | | Island joins |
| 18:15:06 | | qwertyasdfuiopghjkl quits [Remote host closed the connection] |
| 18:46:09 | | qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins |
| 19:02:42 | | tzt quits [Ping timeout: 265 seconds] |
| 19:33:06 | | Craigle quits [Quit: The Lounge - https://thelounge.chat] |
| 19:33:34 | | Craigle (Craigle) joins |
| 19:54:43 | | tzt (tzt) joins |
| 19:57:56 | | benjins2 joins |
| 20:14:12 | | hitgrr8 quits [Client Quit] |
| 21:25:07 | | Craigle quits [Client Quit] |
| 21:25:07 | | qwertyasdfuiopghjkl quits [Client Quit] |
| 21:25:27 | | Craigle (Craigle) joins |
| 21:27:17 | | onetruth joins |
| 21:35:43 | | qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins |
| 21:58:11 | | BlueMaxima joins |
| 22:38:24 | | nicolas17 joins |