| 00:50:52 | | Arcorann (Arcorann) joins |
| 04:34:11 | | qw3rty__ joins |
| 04:38:13 | | qw3rty_ quits [Ping timeout: 265 seconds] |
| 04:40:55 | <@JAA> | atphoenix: Continuing from #flashbang, I don't believe there's any sort of dedupe. The 'duplicates' in the WBM are snapshots of the same URL with the same digest as far as I know. They're still stored explicitly (unless the crawler already wrote revisit records). |
| 04:42:24 | <flashfire42> | Items I think there is a deduplication method but WBM I dont think works that way. it would save space for sure but would it really be in the same vein of the mission of the wayback machine |
| 04:42:46 | <flashfire42> | sure nothing may have changed data wise between 2010 and 2012 for a particular page but its still seeing history at that point |
| 04:44:12 | <@JAA> | I'm not aware of any item-level data deduplication. Item data is literally just stored as files on a dumb drive. There's no fancy file system that does block-level dedupe, no symlinks (which wouldn't even work across servers anyway since each data storage node is completely standalone), etc. |
| 04:45:52 | <@JAA> | I think they talked about that in a blog post at one point. Essentially, keep it as simple as possible, which this is. Makes it much, much harder to seriously fuck things up than with a giant multi-PB file system. |
| 04:46:38 | <@JAA> | The work and stress cost of the latter would be much higher than the monetary cost of storing some duplicates, I guess. |
| 04:49:44 | <OrIdow6> | Yeah, I've heard the same; of the stock Linux filesystems, maybe RAID, rsync for transfer between nodes |
| 04:49:50 | <OrIdow6> | s/of the// |
| 04:50:04 | <atphoenix> | I know that running large scale block level dedupe can also require significant resources itself, at least on some filesystems. ZFS is what I'm thinking of. Dedupe only works there if you have a sufficiently large SSD cache to dedicate to ZFS. |
| 04:50:39 | <OrIdow6> | Keeping it simple, with the thought that the more complex it is, even if it's theoretically more durable, the more likely you're to get a human error that erases data |
| 04:51:10 | <atphoenix> | and then there are the potential performance impacts. Some databases explictly duplicate data to improve data locality and performance. |
| 04:52:16 | <Jake> | I do wonder if you could do some research on what percentage of items have duplicates, files.xml should be available on every item, with sha1 and md5? |
| 04:52:31 | <atphoenix> | what about lossless compression? Is that used? That is a lot less complicated in terms of dependencies. |
| 04:52:54 | <OrIdow6> | IA has the advantage of having data that may not change for decades - of course, it's not actually like that, they have to replace hardware; but in any case I don't think petabox (if that's what they're still called) storage is very write-heavy |
| 04:53:18 | <OrIdow6> | So fs-level deduplication could be more efficient there |
| 04:54:03 | <Jake> | I do think that there is some value in having a very easy to understand system, no complex deduplication. |
| 04:54:17 | <OrIdow6> | (Though for all I know, they do fluid simulations and cache the data on the spare Petabox space) |
| 04:55:39 | <flashfire42> | I really want to use the Internet archive data to attempt hash collisions |
| 04:55:50 | <atphoenix> | maybe, but they do have reads, so targeted-deduplication within a certainly level of locality might be reasonable. I'm sure IA could run an internal analysis to see what their potential PB savings would look like. |
| 04:55:51 | <@JAA> | atphoenix: Nope, I don't think there's any compression applied by IA. Most of the large data is anyway essentially incompressible since it's already compressed, potentially even with a specific algorithm for the data type (e.g. video and audio codecs). And WARCs are virtually always using gzip or zstd. |
| 04:55:58 | <OrIdow6> | Did the AT IA Censuses get file hashes? |
| 04:56:19 | <OrIdow6> | Yeah, most big data is already compressed |
| 04:56:21 | <@JAA> | Yeah, I think so, either for https://archiveteam.org/index.php?title=Audit2014 or for IA.BAK. |
| 04:56:45 | <@JAA> | flashfire42: Try git. Just spam commits until you get a collision. I'll wait... :-) |
| 04:57:33 | <@JAA> | Really though, all hash functions used at IA or in git already have broken collision resistance, so meh. |
| 04:57:46 | <OrIdow6> | On that topic, I wonder if those PDFs with the sha1 collision have ever messed up a crawler |
| 04:57:49 | <atphoenix> | I was thinking about WBM data as being compressible. But that is WARCs, so I guess never mind... |
| 04:58:30 | <@JAA> | Yeah, definitely possible. |
| 04:59:52 | <atphoenix> | I am of the opinion that concern related to hash collisions are can be significantly reduced by checking the results of 2 different hashing algorithms |
| 05:00:56 | <flashfire42> | Thats an interesting point do the SHA1 collission pdfs have the same MD5 for example? |
| 05:00:57 | <atphoenix> | even when dealing with cases of those PDFs or docs created to show intentional collisions on a particular hash function |
| 05:01:49 | <@JAA> | Yes, absolutely. WARCs only use SHA-1 though. |
| 05:02:49 | <flashfire42> | https://server8.kiska.pw/uploads/8a790ce7ffb479ee/image.png |
| 05:02:51 | <flashfire42> | interesting |
| 05:03:00 | <atphoenix> | as for file comparisons, I personally consider a hash mismatch to be a guarantee two items are different, but a collision to mean a byte-for-byte comparison is in order to be absolutely sure. |
| 05:04:08 | <atphoenix> | yup, flashfire42 that's why I'm referring to. Even CRC32 can see the difference. |
| 05:04:20 | <atphoenix> | what* I'm |
| 05:04:32 | <@JAA> | Well, to be absolutely sure, you'll need a checksumming file-system, ECC RAM, protect your hardware from cosmic rays, etc. :-) |
| 05:05:36 | <@JAA> | Personally, I just check hashes. Even MD4's preimage resistance is still unbroken, so a hash collision with an existing file just isn't going to happen. |
| 05:08:07 | <atphoenix> | ZFS on mobo with ECC RAM and supporting CPU, in a machine shielded by a thousand feet of rock? in lieu of all that, a file that matches another file via 2 different hashing algorithms is quite resistant to any break in either algorithm. |
| 05:08:46 | <@JAA> | Yes, until an annoying neutrino somehow causes a bit flip inside the CPU registers or something. :-P |
| 05:13:40 | <atphoenix> | 5-way voting algorithm between CPUs. 3 votes wins. |
| 08:49:04 | | kiska quits [Ping timeout: 240 seconds] |
| 08:49:44 | | flashfire42 quits [Ping timeout: 240 seconds] |
| 08:52:02 | | flashfire42 (flashfire42) joins |
| 08:52:24 | | kiska (kiska) joins |
| 12:05:36 | | HP_Archivist quits [Quit: Leaving] |
| 12:05:56 | | HP_Archivist (HP_Archivist) joins |
| 14:44:44 | | Arcorann quits [Ping timeout: 240 seconds] |
| 20:08:32 | | DogsRNice (Webuser299) joins |
| 23:13:19 | | Arcorann (Arcorann) joins |