| 00:14:53 | | Guest69 joins |
| 00:15:19 | | atphoenix quits [Remote host closed the connection] |
| 00:16:31 | | atphoenix (atphoenix) joins |
| 00:16:46 | | atphoenix quits [Remote host closed the connection] |
| 00:17:25 | | atphoenix (atphoenix) joins |
| 00:18:31 | <atphoenix> | OVH and DriveSavers should have some conversations. |
| 00:18:33 | <atphoenix> | Data recovery from fire-damaged equipment is sometimes possible: https://drivesaversdatarecovery.com/company/museum-of-bizarre-disk-asters/ |
| 00:18:36 | <atphoenix> | But not always: https://drivesaversdatarecovery.com/blog/fire-damaged-devices-recoverable-or-not/ |
| 00:18:38 | <atphoenix> | OVH would also need to have details on which drive serial number was associated with which machine and customer to put things back together. |
| 00:18:53 | | hooway quits [Client Quit] |
| 00:19:03 | | Guest69 quits [Client Quit] |
| 00:21:58 | <@JAA> | For the cheap price of a few dozen million Euros, you, too, can get a DC worth of burnt HDDs recovered! |
| 00:22:35 | <atphoenix> | apply a bulk discount code to sve 25% |
| 00:22:56 | <@JAA> | SBG2 housed 30k servers, and each HDD costs on the order of €1-2k to recover, depending on the damage. So yeah... |
| 00:24:04 | <@JAA> | Not to mention that if anything of the internal structure collapsed during the fire, they might not even know which HDD belonged to which server. |
| 00:24:09 | <@JAA> | Also, some of them probably melted. |
| 00:24:58 | <Sylirana> | They *could* do it for less though. And it would definitely be some good PR for them as well. |
| 00:25:08 | <atphoenix> | Someone could get a lot of practice on fire-damaged-drive data recovery. Drive serial numbers should be extractable from each individual drive, assuming the drive isn't completely destroyed. |
| 00:25:51 | <atphoenix> | Then match drive serial to server to customer. If those records still exist. |
| 00:25:57 | <@JAA> | Yeah, if. |
| 00:26:28 | <@JAA> | And regarding PR, sure, but dropping dozens of millions into the PR department directly would probably do more. |
| 00:27:53 | <atphoenix> | Would insurance cover data recovery? Some policies do. |
| 00:28:29 | <atphoenix> | though I would tend to think barebones hosting services would skip that policy rider |
| 00:29:02 | <atphoenix> | where does ovh sit on the continuum of hosting providers/ |
| 00:29:06 | <atphoenix> | ? |
| 00:30:51 | <atphoenix> | Has anyone seen inside photos from the damaged rooms or of SBG2? |
| 00:31:36 | <@JAA> | I've been looking for them but haven't found any yet. |
| 00:31:40 | <Sylirana> | I meant for DriveSavers. This would be an opportunity for them to make OVH a space one-time offer. In return, DriveSavers will probably have more recovery tasks than they could've ever asked for and will get some good PR out of it for rescuing data in a fire that made the news worldwide. Or they can just not do nothing and get nothing in return. |
| 00:31:59 | <Sylirana> | *-space |
| 00:32:11 | <@JAA> | Ah, right. |
| 00:34:05 | <atphoenix> | attempting to recover, and recovering, both provide positive PR opportunities to OVH and anyone they might partner with (assuming they partner with someone who has dealt with burnt drives). |
| 00:34:43 | <atphoenix> | maybe some tweets are in order |
| 00:48:27 | | brabo quits [Remote host closed the connection] |
| 00:51:18 | <atphoenix> | https://www.ovhcloud.com/en/lp/status-services-backup-strasbourg/ does show that *some* SBG2 servers have recoverable 'internal backups' |
| 00:52:37 | <@JAA> | Well yeah, if you activated that option and ran the backups regularly. |
| 01:21:27 | | onetruth joins |
| 01:42:49 | <Wayward-> | yeah, attempting to recover thousands of harddrives for the few unfortunate souls who don't know the difference between "hosting" (ephemeral) and "storage" (permanent) is a fool's errand. |
| 01:44:46 | <Wayward-> | and I'm not sure "the cheapest dog in town" really needs to worry about PR. |
| 01:48:27 | <Wayward-> | besides, that was an inferno. you don't recover data from drives exposed to an inferno. |
| 01:49:36 | <Wayward-> | magnetic flux disappears at oven baking temps |
| 02:07:31 | <@JAA> | I doubt that. |
| 02:10:16 | <@JAA> | It's difficult to find hard data on this as the HDD manufacturers don't release that kind of stuff, but I'm pretty sure the Curie temperature is more like 500 °C. |
| 02:10:34 | <@JAA> | But yeah, no doubt some hard drives were just completely obliterated in that fire. |
| 02:18:16 | <OrIdow6> | That a company "should" do anything can be justified by "it would be good for PR"; the same argument can be made for why McDonald's should give away food to the poor, or why Microsoft should continue to support Windows XP |
| 02:19:05 | <Wayward-> | Well, even the new permanent neodynium magnets out of China (not very pure) lose their magnetism at 170° C |
| 02:19:21 | <Wayward-> | I can't imagine writable platters holding up to even that |
| 02:20:13 | <@JAA> | Magnetic strength doesn't have much to do with the Curie temperature. |
| 02:21:31 | <@JAA> | And typical 'neodymium magnets' (which aren't pure neodymium at all) have a Curie temperature of 300-400 °C. |
| 02:21:40 | <Wayward-> | and that's not even accounting ridiculous platter expansion |
| 02:21:52 | <@JAA> | That's 600-750 °F for you silly units people. |
| 02:22:19 | <Wayward-> | earlier neodynium magnets were rather pure until around 2015 |
| 02:22:25 | <@JAA> | Uh, no. |
| 02:22:28 | <Wayward-> | quite pure |
| 02:22:36 | <Wayward-> | hense their expense |
| 02:22:41 | <@JAA> | Neodymium is non-magnetic at room temperature. |
| 02:22:44 | <Wayward-> | now the're 1/6th the price |
| 02:22:48 | <@JAA> | It has a Curie temperature of 20 K or something like that. |
| 02:23:11 | <OrIdow6> | OVH has become popular because they have a very low standard of quality, and so can sell things at a low price; were they to do recovery like this, that image could be confused; and it could also look like an act of desparation be seen by investors as showing that they stand to lose customers if they get unlucky |
| 02:24:10 | <@JAA> | 'Neodymium magnets' are almost always NdFeB. |
| 02:25:00 | | britm0b quits [Read error: Connection reset by peer] |
| 02:25:00 | | atphoenix quits [Remote host closed the connection] |
| 02:25:00 | <@JAA> | Usually Nd2Fe14B IIRC. |
| 02:25:00 | | sliccricc_ quits [Read error: Connection reset by peer] |
| 02:25:00 | | HP_Archivist quits [Remote host closed the connection] |
| 02:25:00 | | rsn_ joins |
| 02:25:00 | | qw3rty_ joins |
| 02:25:00 | | britm0b joins |
| 02:25:00 | | Atom-- joins |
| 02:25:00 | | sliccricc_ (sliccricc) joins |
| 02:25:00 | | BlueMaxima quits [Remote host closed the connection] |
| 02:25:08 | | atphoenix (atphoenix) joins |
| 02:25:08 | | HP_Archivist (HP_Archivist) joins |
| 02:25:08 | | BlueMaxima joins |
| 02:25:12 | | Atom quits [Read error: Connection reset by peer] |
| 02:25:12 | <Wayward-> | the cheapest trick in town doesn't lose customers, it rotates customers. that's why you need to stay protected. |
| 02:25:19 | | Wingy quits [Quit: Ping timeout (120 seconds)] |
| 02:25:21 | | britmob25 quits [Quit: Ping timeout (120 seconds)] |
| 02:25:42 | | Wingy (Wingy) joins |
| 02:25:44 | | britmob25 joins |
| 02:28:18 | | qw3rty__ quits [Ping timeout: 264 seconds] |
| 02:28:54 | | rsn quits [Ping timeout: 264 seconds] |
| 02:30:48 | <Wayward-> | JAA: i meant the neodynium *itself* used to be pure, is now no longer pure and just barely even neodynium, but a hodge podge mix of impure rare-earths with some amounts of neodynium. |
| 02:31:35 | <@JAA> | Yeah, that could be. |
| 02:31:45 | <Wayward-> | that purification process is the most expensive part of rare earth magnets |
| 02:31:58 | <Wayward-> | it's also why they've become heat sensitive |
| 02:32:24 | | HP_Archivist quits [Client Quit] |
| 02:32:39 | | HP_Archivist (HP_Archivist) joins |
| 02:40:07 | <atphoenix> | if you read the articles I linked to, you'll see that fire-damaged drives can be recovered, in some cases. Length of exposure, and temperatures involved, both make a difference. However, the conditions that are survivable are fairly broad. |
| 02:59:07 | <Wayward-> | I need to find or write a tool that'll raw-read the same 4096 byte sector over and over, filling the screen with 32768 bits, so I can watch the affects of a strong magnet or blow torch in realtime |
| 03:00:18 | <Wayward-> | alternately reading several sectors and filling the screen with ascii/ansi characters |
| 03:00:52 | <atphoenix> | if you figure out how to raw-read modern disks, let me know. That would be useful for data recovery via the use of statistics and/or brute force of weak bits when the CRC checks fail. |
| 03:02:19 | <Wayward-> | i don't think there have been any attempts to prevent access to physical sector reads on platter drives |
| 03:02:47 | <Wayward-> | any windows hex editor will display absolute sector 000000 for you and its actual contents |
| 03:03:15 | <Wayward-> | I just don't have anything that'll auto refresh 10 times per second |
| 03:04:47 | <atphoenix> | there is drive firmware in between. IIRC there were some drive commands that could be used on 28-bit drives, but not 48-bit addressed drives. |
| 03:05:25 | <atphoenix> | that's based on vague recollections from reading hdparm and/or smartctl docs |
| 03:05:39 | <Wayward-> | sure, firmware sectors aside. unrelated. |
| 03:07:04 | <Wayward-> | but it would be cool to see how close I can get with a monster neodynium magnet before data becomes animated or the drive fails to read. |
| 03:08:07 | <Wayward-> | maybe set the drive on a hot plate with a thermometer and slowly turn the dial up every 5 minutes |
| 03:18:44 | <atphoenix> | A few years ago I saw an interesting video, I think it was by some Sun ZFS engineers. At least is was somewhat related to ZFS, or related to one of the big rack mounted systems they were using that could take something like 50 drives. What they showed is they could *shout* at the drives and see they effects in their measurements. It might have been data transfer rates slowing down. |
| 03:20:55 | <atphoenix> | I think this is it: https://www.youtube.com/watch?v=tDacjrSCeq4 |
| 03:23:28 | <@JAA> | Yeah, that one's a classic. |
| 03:25:17 | | Mineroboter_ joins |
| 03:26:06 | | onetruth quits [Read error: Connection reset by peer] |
| 03:27:02 | | Mineroboter quits [Ping timeout: 250 seconds] |
| 03:27:57 | <atphoenix> | yep. "Shouting in the Datacenter" (effects on drive performance when shouting at hard drives) aka "Brendan Gregg from Sun's Fishworks team makes an interesting discovery about inducing disk latency." with more info at http://web.archive.org/web/20090122051937/http://blogs.sun.com/brendan/entry/unusual_disk_latency |
| 03:37:40 | | etnguyen03 (etnguyen03) joins |
| 04:06:02 | | etnguyen03 quits [Client Quit] |
| 04:24:33 | | qw3rty__ joins |
| 04:27:42 | | qw3rty_ quits [Ping timeout: 250 seconds] |
| 05:00:00 | | treora quits [Client Quit] |
| 05:01:15 | | treora joins |
| 05:04:18 | | minus quits [Ping timeout: 264 seconds] |
| 05:10:01 | | minus joins |
| 05:52:57 | | Sylirana quits [Read error: Connection reset by peer] |
| 05:53:56 | | Sylirana (Sylirana) joins |
| 06:11:32 | | Zopolis4 quits [Remote host closed the connection] |
| 06:48:49 | | Zopolis4 (Zopolis4) joins |
| 06:58:36 | <Wayward-> | videogame speed running has gotten... weird. https://youtu.be/THtbjPQFVZI |
| 07:02:44 | | hooway joins |
| 07:07:58 | | atomicthumbs quits [Quit: No Ping reply in 180 seconds.] |
| 07:09:18 | | atomicthumbs joins |
| 07:20:52 | | Doran (Doranwen) joins |
| 07:21:06 | | Doranwen quits [Ping timeout: 264 seconds] |
| 07:58:37 | | sec^nd quits [Remote host closed the connection] |
| 07:58:49 | | sec^nd (second) joins |
| 08:00:18 | | britmob25 quits [Client Quit] |
| 08:37:31 | | Zopolis4 quits [Remote host closed the connection] |
| 10:16:48 | | Arcorann (Arcorann) joins |
| 10:53:29 | | Arcorann_ joins |
| 10:55:32 | | Arcorann quits [Ping timeout: 250 seconds] |
| 11:00:25 | | Arcorann (Arcorann) joins |
| 11:01:36 | | Arcorann_ quits [Ping timeout: 250 seconds] |
| 11:06:06 | | Arcorann quits [Ping timeout: 264 seconds] |
| 12:09:16 | | BlueMaxima quits [Client Quit] |
| 12:12:14 | | Mateon1 quits [Ping timeout: 250 seconds] |
| 12:14:34 | | Mateon1 joins |
| 12:23:56 | | britmob25 joins |
| 12:37:08 | | etnguyen03 (etnguyen03) joins |
| 14:03:40 | <Jake> | https://twitter.com/olesovhcom/status/1372156948204949507 |
| 14:06:56 | | @kiska quits [Quit: Ping timeout (120 seconds)] |
| 14:08:51 | | kiska (kiska) joins |
| 14:08:51 | | @ChanServ sets mode: +o kiska |
| 14:36:06 | | nertzy quits [Ping timeout: 250 seconds] |
| 14:49:53 | | rsn joins |
| 14:52:34 | | rsn_ quits [Ping timeout: 250 seconds] |
| 14:52:34 | | rsn quits [Read error: Connection reset by peer] |
| 14:52:50 | | rsn joins |
| 15:04:05 | | nertzy (nertzy) joins |
| 15:22:45 | <@kiska> | JAA: table etherpad plugin? |
| 15:23:00 | <@kiska> | Possible addition provided it works |
| 15:24:44 | <@JAA> | If it works and doesn't interfere with other things, why not? |
| 15:27:49 | <@kiska> | Doing some maintenance on pad.notkiska.pw |
| 15:27:54 | <@kiska> | Please do make backups |
| 15:28:20 | <@JAA> | I'm dumping a number of pads every few minutes. |
| 15:28:26 | <@kiska> | Excellent! |
| 15:30:55 | <@kiska> | And... something broke... |
| 15:31:39 | <@kiska> | And by something broke... I mean my ssh session + whatever was running since it seems to have OOM'd |
| 15:31:45 | <@JAA> | lol |
| 15:32:06 | <@kiska> | Lemme go ask Vultr for more memory :D |
| 15:33:05 | <@kiska> | Mem: 985M 830M 96M 1.7M 58M 55M |
| 15:33:06 | <@kiska> | -_- |
| 15:34:30 | <@kiska> | Wot... |
| 15:34:30 | <@kiska> | [2021-03-17 15:29:33.394] [INFO] http - HTTP server closedalled |
| 15:34:30 | <@kiska> | https://server8.kiska.pw/uploads/8b2d35e9c2a2eab6/image.png |
| 15:34:59 | <@kiska> | I am going to reboot this machine and give it some swap |
| 15:35:12 | <@JAA> | Let me guess, Node ate all the memory? |
| 15:36:06 | <@kiska> | Yes node + mysql |
| 15:36:58 | <@kiska> | https://server8.kiska.pw/uploads/f8026d4f9a37c92c/image.png Wonderful! |
| 15:38:41 | <SCSi> | oh thats nice |
| 15:39:13 | <@kiska> | https://server8.kiska.pw/uploads/2ebb44e9cb5617ee/image.png -_- |
| 15:39:40 | <@kiska> | I plan to give this 2G of swap |
| 15:41:05 | <@kiska> | https://server8.kiska.pw/uploads/81d2bc7d624f5134/image.png Done! :D |
| 15:41:58 | <@kiska> | So... this machine now has 1G RAM + 2G swap |
| 15:42:26 | <SCSi> | on fast drives? |
| 15:46:09 | <@kiska> | So... if this thing starts up, I'll go and update it |
| 15:54:05 | <@HCross> | kiska: want to rehome that thing in London? |
| 15:55:21 | <@kiska> | I've broken more things... scrypt doesn't want to install :( |
| 16:16:59 | <@kiska> | Lets see if this works https://www.npmjs.com/package/ep_prompt_for_name |
| 16:21:32 | <@kiska> | JAA: Feature installed please test if it actually works |
| 16:45:16 | <@kiska> | Well maintenance on the pad is over, so hopefully I don't need to look at it for the next like ever |
| 16:57:14 | | xit quits [Quit: Ping timeout (120 seconds)] |
| 16:57:31 | | xit joins |
| 18:36:36 | | sliccricc_ quits [Remote host closed the connection] |
| 18:36:36 | | mutantmonkey quits [Remote host closed the connection] |
| 18:36:36 | | HackMii quits [Write error: Broken pipe] |
| 18:41:09 | | HackMii (hacktheplanet) joins |
| 18:42:10 | | mutantmonkey (mutantmonkey) joins |
| 19:06:20 | <@kiska> | Oh hey some life coming back to SBG https://server8.kiska.pw/uploads/20131311d9ae4f16/image.png |
| 19:07:42 | <@JAA> | Yup: https://twitter.com/olesovhcom/status/1372210400499007491 |
| 19:08:26 | <@kiska> | I don't quite get their visual monitoring symbols |
| 19:08:43 | <@JAA> | I'm glad I don't have to clean up those SBG3 servers that got smoked. |
| 19:08:44 | <@kiska> | Is R61 a room? |
| 19:08:47 | <@JAA> | 12-16 hours per server... Oof |
| 19:10:41 | <@JAA> | I think each line there is a room. I.e. R61A, R61B, etc. are each rooms. |
| 19:11:04 | <@kiska> | I thought they were racks in the room? |
| 19:11:22 | <@JAA> | A00, A01, etc. are the racks I thought? |
| 19:17:56 | <@JAA> | Yeah, I think that's right. |
| 19:21:27 | <@kiska> | *shrug* |
| 19:40:20 | | DogsRNice (Webuser299) joins |
| 19:57:53 | | godane quits [Client Quit] |
| 20:13:40 | | lunik1 quits [Ping timeout: 250 seconds] |
| 20:13:40 | | Trieste quits [Ping timeout: 250 seconds] |
| 20:15:03 | | lunik1 joins |
| 20:15:51 | | Trieste joins |
| 20:36:34 | | Guest69 joins |
| 20:56:04 | | Guest69 quits [Client Quit] |
| 21:11:12 | | ROpdebee is now known as PlsNoJava |
| 21:54:42 | | HP_Archivist quits [Client Quit] |
| 21:54:59 | | HP_Archivist (HP_Archivist) joins |
| 22:11:51 | | BlueMaxima joins |
| 22:47:04 | | PlsNoJava quits [Client Quit] |
| 22:47:27 | | PlsNoJava joins |
| 22:56:56 | | godane (godane) joins |
| 22:56:58 | | PlsNoJava is now authenticated as ROpdebee |
| 22:56:58 | | PlsNoJava quits [Changing host] |
| 22:56:58 | | PlsNoJava (ROpdebee) joins |
| 23:08:53 | | onetruth joins |
| 23:53:02 | | Sylirana quits [Read error: Connection reset by peer] |
| 23:53:11 | | Sylirana (Sylirana) joins |