HomeSearchPrevious dayNext day

00:41:40march_happy quits [Ping timeout: 265 seconds]
00:42:03march_happy (march_happy) joins
00:50:36lennier1 quits [Client Quit]
00:52:16lennier1 (lennier1) joins
00:58:49<h2ibot>JesseW created URLTeam/Dead (+39398, split off from [[URLTeam]] page): https://wiki.archiveteam.org/?title=URLTeam/Dead
00:59:49<h2ibot>JesseW edited URLTeam/Dead (+24, add header so the edit link works correctly): https://wiki.archiveteam.org/?diff=48555&oldid=48554
00:59:50<h2ibot>JesseW edited URLTeam (-39405, /* Dead or Broken */ split to subpage…): https://wiki.archiveteam.org/?diff=48556&oldid=48552
01:00:49<h2ibot>JesseW created URLTeam/Warrior (+40496, split off from [[URLTeam]] page): https://wiki.archiveteam.org/?title=URLTeam/Warrior
01:01:50<h2ibot>JesseW edited URLTeam (-40476, /* Warrior projects */ split off into…): https://wiki.archiveteam.org/?diff=48558&oldid=48556
01:04:11HP_Archivist (HP_Archivist) joins
01:08:19Arcorann (Arcorann) joins
01:08:48Arcorann quits [Remote host closed the connection]
01:15:45Arcorann (Arcorann) joins
01:21:41tbc1887 (tbc1887) joins
01:32:36Discant joins
01:39:48Discant quits [Ping timeout: 265 seconds]
01:44:37BPCZ quits [Client Quit]
01:46:04BPCZ joins
01:47:18adia3 (adia) joins
01:47:22BlueMaxima_ joins
01:47:27Craigle7 (Craigle) joins
01:47:44BlueMaxima_ quits [Remote host closed the connection]
01:48:00arkiver2 (arkiver) joins
01:48:00@ChanServ sets mode: +o arkiver2
01:48:02dm4v quits [Ping timeout: 286 seconds]
01:48:02Stiletto quits [Ping timeout: 286 seconds]
01:48:02le0n quits [Ping timeout: 286 seconds]
01:48:02@arkiver quits [Remote host closed the connection]
01:48:02celestial quits [Quit: ZNC 1.8.0 - https://znc.in]
01:48:02omglolbah quits [Remote host closed the connection]
01:48:03fuzzy8021 quits [Remote host closed the connection]
01:48:03yano quits [Remote host closed the connection]
01:48:03adamus1red quits [Quit: SigTerm]
01:48:03Iki1 quits [Remote host closed the connection]
01:48:03BlueMaxima quits [Remote host closed the connection]
01:48:03tbc1887 quits [Remote host closed the connection]
01:48:03adia quits [Client Quit]
01:48:03Craigle quits [Quit: Ping timeout (120 seconds)]
01:48:03driib08353 quits [Client Quit]
01:48:03VerifiedJ quits [Client Quit]
01:48:03IDK_ quits [Quit: Ping timeout (120 seconds)]
01:48:03Ryz quits [Quit: Ping timeout (120 seconds)]
01:48:03sepro quits [Quit: Ping timeout (120 seconds)]
01:48:03notbasetwo quits [Quit: o/]
01:48:03Sanqui quits [Quit: .]
01:48:03mikael quits [Quit: ZNC - http://znc.in]
01:48:03adia3 is now known as adia
01:48:03@chfoo quits [Write error: Broken pipe]
01:48:03@OrIdow6 quits [Write error: Broken pipe]
01:48:03geezabiscuit quits [Quit: ZNC - https://znc.in]
01:48:03HugsNotDrugs` quits [Quit: .]
01:48:03Craigle7 is now known as Craigle
01:48:07notbasetwo joins
01:48:07celestial joins
01:48:07mikael joins
01:48:07dm4v joins
01:48:08dm4v quits [Changing host]
01:48:08dm4v (dm4v) joins
01:48:08geezabiscuit joins
01:48:09VerifiedJ9 (VerifiedJ) joins
01:48:09tbc1887 (tbc1887) joins
01:48:09geezabiscuit quits [Changing host]
01:48:09geezabiscuit (geezabiscuit) joins
01:48:10adamus1red (adamus1red) joins
01:48:10IDK_ joins
01:48:12OrIdow6 (OrIdow6) joins
01:48:13HugsNotDrugs joins
01:48:13@ChanServ sets mode: +o OrIdow6
01:48:13sepro5 (sepro) joins
01:48:13driib083536 (driib) joins
01:48:29le0n (le0n) joins
01:48:33Sanqui joins
01:48:36Stiletto joins
01:48:43omglolbah joins
01:48:47chfoo (chfoo) joins
01:48:47@ChanServ sets mode: +o chfoo
01:48:49Ryz (Ryz) joins
01:48:51BlueMaxima_ joins
01:49:15BlueMaxima_ quits [Remote host closed the connection]
01:49:35yano (yano) joins
01:50:09@arkiver2 is now known as @arkiver
01:50:21BlueMaxima_ joins
01:50:44BlueMaxima_ quits [Remote host closed the connection]
01:51:51BlueMaxima_ joins
01:52:15BlueMaxima_ quits [Remote host closed the connection]
01:52:28BlueMaxima_ joins
01:57:16fuzzy8021 (fuzzy8021) joins
02:00:32tbc1887 quits [Client Quit]
02:01:22tbc1887 (tbc1887) joins
02:13:37benjins quits [Ping timeout: 252 seconds]
02:17:02benjins joins
02:33:35lennier1 quits [Client Quit]
02:35:48lennier1 (lennier1) joins
02:40:33lennier1 quits [Ping timeout: 252 seconds]
02:40:40lennier1 (lennier1) joins
02:41:37lennier1 quits [Client Quit]
02:43:29lennier1 (lennier1) joins
02:56:03DogsRNice quits [Read error: Connection reset by peer]
03:08:42wyatt8750 joins
03:09:05wyatt8740 quits [Ping timeout: 265 seconds]
03:45:57march_happy quits [Ping timeout: 265 seconds]
03:46:13<h2ibot>JustAnotherArchivist edited URLTeam/Dead (+51, Preparation for JAABot): https://wiki.archiveteam.org/?diff=48559&oldid=48555
03:46:23HP_Archivist quits [Client Quit]
03:46:44march_happy (march_happy) joins
03:49:14<h2ibot>JAABot edited URLTeam/Dead (+0): https://wiki.archiveteam.org/?diff=48560&oldid=48559
03:53:14<h2ibot>JustAnotherArchivist edited URLTeam/Dead (-31, Fix mangled entries): https://wiki.archiveteam.org/?diff=48561&oldid=48560
03:55:05DopefishJustin quits [Remote host closed the connection]
03:59:51BlueMaxima_ quits [Client Quit]
04:00:15<h2ibot>JAABot edited URLTeam/Dead (+0): https://wiki.archiveteam.org/?diff=48563&oldid=48561
04:15:51<tbc1887>Hi all, just got a quick question about running the projects in docker (such as urls-grab), is it possible to make them prefer to do their file operations in RAM instead of disk? The machine I have it running on has a slow HDD and I *would* put an SSD in, but I'm slightly concerned about the write wear that would entail. The machine has plenty of ram spare so is there a way to do this?
04:20:33DopefishJustin joins
04:29:48<@JAA>tbc1887: You can mount a tmpfs at /grab/data in the container.
04:31:35<tbc1887>I'm not the best with linux / docker, I'll google it but do you have any idea? @jaa
04:33:10<tbc1887>I can see some run variables on https://docs.docker.com/storage/tmpfs/ that look promising, based on what you said should I do --tmpfs /grab/data
04:34:30<tbc1887>I can also see size options. I currently run 20 threads per container, any advice there too? (Thanks for the help and sorry for the mild spam)
04:39:50<@JAA>hackint/#down-the-tube 2021-11-30 18:27:48 UTC < ThreeHM> Hmm, good idea to use tmpfs for reducing disk load. Turns out docker has an option for that: "docker run --mount type=tmpfs,tmpfs-size=2G,destination=/grab/data ..."
04:40:53<tbc1887>I see tmpfs-size=2G - the page I linked says that's in bytes. Does the grab section really only need to be that big?
04:41:06<@JAA>The required size will vary strongly by project, and I have no specific advice there.
04:41:22<h2ibot>Ryz edited URLTeam/Warrior (+221, /* Warrior projects */ Added https://lttr.ai/): https://wiki.archiveteam.org/?diff=48564&oldid=48557
04:42:08<tbc1887>urls-grab would have to store the entire set of urls there until upload is completed right? Also, if the mount fills up do you know if it will move to HDD?
04:42:41<@JAA>Yes, the data is kept locally until the upload is complete.
04:42:50<@JAA>There is no spillover to storage, this is purely in RAM.
04:43:12<tbc1887>Will the download threads wait for storage or fail?
04:43:52<@JAA>Pretty sure the file writes will fail with ENOSPC and then wget should crash with some sort of error.
04:44:31<tbc1887>If that happens will the threads automatically be restarted or will that thread remain permanently down?
04:45:25<@JAA>It will retry the same item(s) once and then move on to fetch new work from the tracker. The failed ones will eventually get handed out to someone else.
04:46:03<tbc1887>Perfect. Any idea if item(s) have a size limit I could base my size on?
04:47:04<@OrIdow6>Depends on what you're running
04:47:37<tbc1887>I'm running the urls project and I'm fairly sure I've seen *large* uploads before
04:48:02<@OrIdow6>Yeah, I think a lot of the big ongoing projects have long tails
04:48:29<tbc1887>Long tails?
04:49:33<@JAA>Imagine a graph of item size vs number of items at that size. Most items are smallish, but some are very large.
04:49:40<@OrIdow6>https://en.wikipedia.org/wiki/Long_tail
04:49:51<@JAA>I had to kill an item on GitHub recently that was at over 400 GiB.
04:50:11<tbc1887>bRUH
04:50:14<@JAA>Most items there are a few MB.
04:50:16<tbc1887>Did you see what it was?
04:50:26<@JAA>Yeah, I mentioned it in #gitgud at the time.
04:50:50<@JAA>Anyway, those outliers will typically fail many many times on various workers before they maybe end up on one that can handle it.
04:51:56<@JAA>You'll want to look at typical item sizes, then multiply that by your concurrency and a safety factor to account for larger items.
04:52:51<@JAA>You did say you have plenty of RAM though, didn't you? ;-)
04:53:33<tbc1887>Well, plenty in terms of currently unused vs used >:P
04:53:50<tbc1887>I can always yoink some sticks out of a dead machine'
05:15:26<h2ibot>Ryz edited URLTeam/Warrior (+209, /* Warrior projects */ Added https://lttr.ai/ -…): https://wiki.archiveteam.org/?diff=48565&oldid=48564
05:42:47michaelblob_ quits [Ping timeout: 265 seconds]
05:43:37michaelblob (michaelblob) joins
05:48:57<tbc1887>I'm currently waiting for my github container to stop and I have been for almost an hour now, it's up to over 5000 items on some threads, wtf?
05:58:06tbc1887 quits [Read error: Connection reset by peer]
06:39:06pabs quits [Client Quit]
06:39:32pabs (pabs) joins
06:46:40<h2ibot>Tech234a edited Chrome Web Store (+264, /* Chrome Apps */ Support extended for Chrome…): https://wiki.archiveteam.org/?diff=48566&oldid=47715
06:48:41<h2ibot>Tech234a edited Deathwatch (+205, /* 2022 */ Chrome Apps supported for…): https://wiki.archiveteam.org/?diff=48567&oldid=48553
06:50:41<h2ibot>Tech234a edited Deathwatch (+44, /* 2022 */ Git.io: only some links may be removed): https://wiki.archiveteam.org/?diff=48568&oldid=48567
07:18:44qwertyasdfuiopghjkl joins
07:28:58pabs quits [Client Quit]
07:34:51pabs (pabs) joins
07:40:51driib083536 quits [Ping timeout: 252 seconds]
08:36:30tbc1887 (tbc1887) joins
08:57:18wyatt8750 quits [Ping timeout: 252 seconds]
08:59:06wyatt8740 joins
09:45:42Arcorann quits [Ping timeout: 252 seconds]
09:49:01Arcorann (Arcorann) joins
10:01:07wyatt8750 joins
10:03:26wyatt8740 quits [Read error: Connection reset by peer]
10:16:44Terbium quits [Quit: http://quassel-irc.org - Chat comfortably. Anywhere.]
10:17:10Terbium joins
10:37:27spirit2 joins
10:37:57spirit2 quits [Client Quit]
11:13:32wyatt8740 joins
11:14:15wyatt8750 quits [Ping timeout: 252 seconds]
11:14:29qwertyasdfuiopghjkl quits [Ping timeout: 265 seconds]
11:30:19knecht42079 is now known as knecht420
11:33:52tbc1887 quits [Read error: Connection reset by peer]
11:41:33Discant joins
11:41:56Discant quits [Remote host closed the connection]
11:43:03Discant joins
11:43:26Discant quits [Remote host closed the connection]
11:44:32Discant joins
11:44:56Discant quits [Remote host closed the connection]
11:46:03Discant joins
11:46:26Discant quits [Remote host closed the connection]
11:47:33Discant joins
11:47:56Discant quits [Remote host closed the connection]
11:49:03Discant joins
11:49:26Discant quits [Remote host closed the connection]
11:50:32Discant joins
11:50:56Discant quits [Remote host closed the connection]
11:52:03Discant joins
11:52:26Discant quits [Remote host closed the connection]
11:52:48Discant joins
11:54:53dom joins
11:55:39dom quits [Remote host closed the connection]
12:15:41driib083536 (driib) joins
12:17:45wyatt8750 joins
12:19:07wyatt8740 quits [Ping timeout: 265 seconds]
12:22:44Discant quits [Remote host closed the connection]
12:23:06Discant joins
12:26:48TheTechRobo quits [Remote host closed the connection]
12:28:23TheTechRobo (TheTechRobo) joins
12:29:50TheTechRobo quits [Read error: Connection reset by peer]
12:30:27TheTechRobo (TheTechRobo) joins
12:30:43TheTechRobo quits [Remote host closed the connection]
12:31:05TheTechRobo (TheTechRobo) joins
12:33:08Discant quits [Ping timeout: 265 seconds]
12:47:38qwertyasdfuiopghjkl joins
12:52:15pabs quits [Client Quit]
12:52:42pabs (pabs) joins
12:57:14HP_Archivist (HP_Archivist) joins
13:07:35TheTechRobo quits [Ping timeout: 265 seconds]
13:10:33TheTechRobo (TheTechRobo) joins
13:49:30syntaxx quits [Read error: Connection reset by peer]
13:49:50syntaxx4 (syntaxx) joins
13:51:53adamus1red quits [Client Quit]
13:53:52syntaxx4 quits [Read error: Connection reset by peer]
13:54:20adamus1red (adamus1red) joins
14:00:09sepro5 is now known as sepro
14:00:43wyatt8750 quits [Read error: Connection reset by peer]
14:02:55wyatt8740 joins
14:16:06Arcorann quits [Ping timeout: 240 seconds]
15:33:51fuzzy8021 quits [Ping timeout: 252 seconds]
15:59:19<Ryz>Welp, Ragnarok Online private servers are being hunted down as per https://massivelyop.com/2022/05/06/gravity-is-apparently-cracking-down-on-ragnaroks-rogue-server-community/
16:20:03fuzzy8021 (fuzzy8021) joins
16:33:29Megame (Megame) joins
16:33:38Megame quits [Remote host closed the connection]
16:37:50march_happy quits [Ping timeout: 265 seconds]
17:09:43wyatt8750 joins
17:12:38wyatt8740 quits [Ping timeout: 265 seconds]
17:25:26LeGoupil joins
18:04:57wyatt8740 joins
18:06:09wyatt8750 quits [Ping timeout: 265 seconds]
19:18:15HP_Archivist quits [Ping timeout: 252 seconds]
19:28:09JackThompson05 quits [Ping timeout: 252 seconds]
19:51:05HP_Archivist (HP_Archivist) joins
19:51:22HP_Archivist quits [Remote host closed the connection]
19:51:49HP_Archivist (HP_Archivist) joins
19:52:53HP_Archivist quits [Client Quit]
20:24:48spirit quits [Quit: Leaving]
20:28:50<h2ibot>Sanqui edited Telegram (+161): https://wiki.archiveteam.org/?diff=48569&oldid=48501
21:28:12LeGoupil quits [Client Quit]
21:35:32Megame (Megame) joins
21:35:39Megame quits [Remote host closed the connection]
21:39:40Megame (Megame) joins
21:45:29march_happy (march_happy) joins
22:01:41onetruth joins
22:55:37wyatt8740 quits [Read error: Connection reset by peer]
22:56:16wyatt8740 joins
23:24:40Arcorann (Arcorann) joins
23:26:01qwertyasdfuiopghjkl quits [Remote host closed the connection]
23:30:57march_happy quits [Ping timeout: 265 seconds]
23:31:50march_happy (march_happy) joins
23:58:46qwertyasdfuiopghjkl joins

HomeSearchPrevious dayNext day