HomeSearchPrevious dayNext day

00:03:54fuzzy80211 quits [Read error: Connection reset by peer]
00:04:30fuzzy80211 (fuzzy80211) joins
00:45:00<HP_Archivist>JAA: Finally calculated hash values from all files from all items. Each file item has a txt like this: https://transfer.archivete.am/apUFp/a-glorious-accident-vhs-tape-scans_md5_checksums.txt
00:45:00<eggdrop>inline (for browser viewing): https://transfer.archivete.am/inline/apUFp/a-glorious-accident-vhs-tape-scans_md5_checksums.txt
00:45:49<HP_Archivist>I tried using little-things but am confused, would you mind helping?
00:48:25<HP_Archivist>Does iasha1check only compare for sha1 or can it also do md5?
01:04:37<@JAA>HP_Archivist: iasha1check won't help you directly since you've calculated the local hashes independently. It's a script for 'verify that the IA item is identical to this local directory', by generating a list of hashes from IA and then throwing that into `sha1sum -c`.
01:05:26<@JAA>The same technique for building the list of hashes could be used, but you're better off with the jq command I gave you a few days ago.
01:10:10<HP_Archivist>Yup, I did that. This is what I used: https://transfer.archivete.am/Zu47l/bash_script_verification.txt
01:10:11<eggdrop>inline (for browser viewing): https://transfer.archivete.am/inline/Zu47l/bash_script_verification.txt
01:10:59<HP_Archivist>The problem is, it's indicating that while some files match, others are failing the match comparison (this is not my attempt at using iasha1check, btw, I was just asking if that would be better suited)
01:11:29<HP_Archivist>And the problem is - manually inspecting a given file's hash value locally against ia's indicates it is correct. The script is saying the values do not match.
01:11:56<HP_Archivist>And, I'm wondering what I'm doing wrong/overlooking
01:12:43<@JAA>Are you seeing this in the example file above?
01:13:23<@JAA>Your method of comparing will fail for files with spaces and probably a bunch of other things.
01:13:40<HP_Archivist>Ahh
01:15:33<@JAA>diff -u <(curl -s https://archive.org/metadata/a-glorious-accident-vhs-tape-scans | jq -r '.files[] | select(.source == "original") | "\(.md5) \(.name)"' | sort) <(curl -s https://transfer.archivete.am/apUFp/a-glorious-accident-vhs-tape-scans_md5_checksums.txt | sed 's,/mnt/g/iapsource/a-glorious-accident-vhs-tape-scans/,,' | sort)
01:15:33<eggdrop>inline (for browser viewing): https://transfer.archivete.am/inline/apUFp/a-glorious-accident-vhs-tape-scans_md5_checksums.txt
01:16:39<@JAA>That reports only two differences: _files.xml (which is expected) and _archive.torrent (which isn't source original but you downloaded anyway)
01:17:13<HP_Archivist>Yeah, I was hoping I could have the script exclude those when doing the compare
01:17:22<HP_Archivist>But your arg looks like it works much better
01:17:45<HP_Archivist>How do I adjust so it can go through each item's txt?
01:19:49<@JAA>diff -u <(curl -s https://archive.org/metadata/a-glorious-accident-vhs-tape-scans | jq -r --arg id a-glorious-accident-vhs-tape-scans '.files[] | select(.source == "original" and (.name == "\($id)_archive.torrent" | not) and (.name == "\($id)_files.xml" | not)) | "\(.md5) \(.name)"' | sort) <(curl -s https://transfer.archivete.am/apUFp/a-glorious-accident-vhs-tape-scans_md5_checksums.txt | grep -Pv
01:19:50<eggdrop>inline (for browser viewing): https://transfer.archivete.am/inline/apUFp/a-glorious-accident-vhs-tape-scans_md5_checksums.txt
01:19:55<@JAA>'/([^/]+)/\1_(archive\.torrent|files\.xml)$' | sed 's,/mnt/g/iapsource/a-glorious-accident-vhs-tape-scans/,,' | sort)
01:20:04<@JAA>That excludes _files.xml and _archive.torrent on both sides.
01:20:19<@JAA>I don't know your local structure of the files. :-)
01:21:46<@JAA>Do you have a dir full of a-glorious-accident-vhs-tape-scans_md5_checksums.txt files?
01:23:19<HP_Archivist>The directory structure is: /mnt/g/
01:23:19<HP_Archivist>And then g has a .txt for every item downloaded :)
01:24:06<HP_Archivist>So it's /mng/iapsource (where the items are stored in their own folder - folder has the identifier as the name. But to keep things organized the md5 txts are stored one directory up
01:24:10<@JAA>That the full path is in your local hash files is annoying.
01:24:29<@JAA>Else it'd be a simple diff, but that means it needs to be stripped out.
01:25:07<HP_Archivist>Hmm, I could just script out the paths?
01:25:17<@JAA>Right, so you have /mnt/g/iapsource/foo/ with the data for item 'foo' and a /mnt/g/iapsource/foo_md5_checksums.txt file.
01:26:14<@JAA>Maybe this:
01:26:15<@JAA>for f in *_md5_checksums.txt; do identifier="${f%_md5_checksums.txt}"; echo "${identifier}"; diff -u <(curl -s "https://archive.org/metadata/${identifier}" | jq -r --arg id "${identifier}" '.files[] | select(.source == "original" and (.name == "\($id)_archive.torrent" | not) and (.name == "\($id)_files.xml" | not)) | "\(.md5) \(.name)"' | sort) <(grep -Pv '/([^/]+)/\1_(archive\.torrent|files\.xml)$'
01:26:16<HP_Archivist>Second one is actually: /mnt/g/foo_md5_checksums.txt file
01:26:21<@JAA>"${f}" | sed 's,/mnt/g/iapsource/[^/]*/,,' | sort); done
01:26:51<@JAA>Ah, also fine with that command running in /mnt/g/.
01:27:19<@JAA>I went for the lazy way of stripping out the path.
01:27:35<HP_Archivist>I'm for whatever works to get this over and done with asap :P
01:27:45<@JAA>:-)
01:28:10<@JAA>If your dir structure is consistent as described above, the lazy way is perfectly fine.
01:29:55<HP_Archivist>Yup it is. And running now. seems to be working!
01:30:22<@JAA>If you get a diff, something's bad. Otherwise, it should just be a list of identifiers.
01:31:12<HP_Archivist>Thank you so much... Yup. I only see the .xml files failing so far
01:32:12<HP_Archivist>Always a lifesaver JAA :)
01:32:49<@JAA>I only explicitly filtered out _files.xml, maybe you skipped others (reviews?), too.
01:38:47<HP_Archivist>I downloaded, along with source files: _files.xml, _meta.xml, .SQLite, .torrent
01:39:20<HP_Archivist>But actually, your args are noticing that the paths are different but the hashes are the same on source files
01:40:08<@JAA>Hmm?
01:41:03<HP_Archivist>1 sec
01:42:05<HP_Archivist>it looks like source files have identical MD5 hashes, which is great. But the file paths differ between what's on IA and what I have locally. Which is odd
01:43:57<@JAA>Differ how?
01:45:12<HP_Archivist>Ahh, you know what. Idk how this happened. But I think when I did ia download on some of these items, the script flattened the directory to one folder vs what's on IA side
01:45:19<HP_Archivist>That's why it's saying the paths are different
01:45:31ScenarioPlanet quits [Ping timeout: 255 seconds]
01:45:31TheTechRobo quits [Ping timeout: 255 seconds]
01:45:31Pedrosso quits [Ping timeout: 255 seconds]
03:04:20DogsRNice joins
06:08:38DogsRNice quits [Read error: Connection reset by peer]
07:26:37Flashfire42 quits [Ping timeout: 255 seconds]
07:26:37s-crypt quits [Ping timeout: 255 seconds]
07:27:04kiska quits [Ping timeout: 255 seconds]
07:29:05HP_Archivist quits [Read error: Connection reset by peer]
07:42:31s-crypt (s-crypt) joins
07:42:40Flashfire42 joins
07:44:17kiska (kiska) joins
07:54:31kiska quits [Ping timeout: 255 seconds]
07:54:31Flashfire42 quits [Ping timeout: 255 seconds]
07:54:41s-crypt quits [Ping timeout: 256 seconds]
08:10:11s-crypt (s-crypt) joins
08:12:51Flashfire42 joins
08:14:12kiska (kiska) joins
13:54:36sepro8 (sepro) joins
13:54:58sepro quits [Ping timeout: 255 seconds]
13:54:58sepro8 is now known as sepro
14:19:23HP_Archivist (HP_Archivist) joins
14:38:52ScenarioPlanet (ScenarioPlanet) joins
15:23:44ScenarioPlanet quits [Client Quit]
15:24:36ScenarioPlanet (ScenarioPlanet) joins
15:29:35TheTechRobo (TheTechRobo) joins
17:13:07DogsRNice joins
18:20:00Ryz quits [Quit: The Lounge - https://thelounge.chat]
19:33:57DopefishJustin quits [Remote host closed the connection]
19:42:15DopefishJustin (DopefishJustin) joins
20:25:56Ryz (Ryz) joins
21:01:31qwertyasdfuiopghjkl quits [Quit: Client closed]
21:05:04Ryz quits [*.net *.split]
21:05:04DopefishJustin quits [*.net *.split]
21:05:04s-crypt quits [*.net *.split]
21:05:04KoalaBear84 quits [*.net *.split]
21:05:04nicolas17 quits [*.net *.split]
21:05:04nulldata quits [*.net *.split]
21:05:04magmaus3 quits [*.net *.split]
21:05:04Jake quits [*.net *.split]
21:05:04Terbium quits [*.net *.split]
21:05:04DLoader quits [*.net *.split]
21:05:04igloo22225 quits [*.net *.split]
21:05:04yano quits [*.net *.split]
21:05:04c3manu quits [*.net *.split]
21:05:04cm quits [*.net *.split]
21:05:04Barto quits [*.net *.split]
21:05:04Fusl quits [*.net *.split]
21:05:28Exorcism|m quits [*.net *.split]
21:05:29mikolaj|m quits [*.net *.split]
21:05:29phaeton quits [*.net *.split]
21:05:29Nulo|m quits [*.net *.split]
21:05:29yzqzss quits [*.net *.split]
21:05:29s-crypt|m|m quits [*.net *.split]
21:05:29qyxojzh|m quits [*.net *.split]
21:05:29hlgs|m quits [*.net *.split]
21:05:29thermospheric quits [*.net *.split]
21:05:29schwarzkatz|m quits [*.net *.split]
21:05:29Thibaultmol quits [*.net *.split]
21:05:29tomodachi94 quits [*.net *.split]
21:05:29theblazehen|m quits [*.net *.split]
21:05:29audrooku|m quits [*.net *.split]
21:05:29x9fff00 quits [*.net *.split]
21:05:29mattwright324|m quits [*.net *.split]
21:05:29Vokun quits [*.net *.split]
21:05:29DigitalDragon quits [*.net *.split]
21:05:29britmob|m quits [*.net *.split]
21:05:29Sanqui|m quits [*.net *.split]
21:05:29igneousx quits [*.net *.split]
21:05:29SootBector quits [*.net *.split]
21:05:29fireonlive quits [*.net *.split]
21:05:29@ChanServ quits [*.net *.split]
21:08:22qwertyasdfuiopghjkl joins
21:09:47qwertyasdfuiopghjkl quits [Client Quit]
21:10:54qwertyasdfuiopghjkl joins
21:15:54Fusl (Fusl) joins
21:15:54Barto (Barto) joins
21:15:54cm joins
21:15:54c3manu (c3manu) joins
21:15:54yano (yano) joins
21:15:54igloo22225 (igloo22225) joins
21:15:54DLoader (DLoader) joins
21:15:54Terbium joins
21:15:54Jake (Jake) joins
21:15:54magmaus3 (magmaus3) joins
21:15:54nulldata (nulldata) joins
21:15:54nicolas17 (nicolas17) joins
21:15:54KoalaBear84 joins
21:15:54s-crypt (s-crypt) joins
21:15:54DopefishJustin (DopefishJustin) joins
21:15:54Ryz (Ryz) joins
21:15:54SootBector (SootBector) joins
21:15:54Exorcism|m (exorcism) joins
21:15:54fireonlive (fireonlive) joins
21:15:54mattwright324|m joins
21:15:54igneousx (igneousx) joins
21:15:54tomodachi94 (tomodachi94) joins
21:15:54theblazehen|m joins
21:15:54audrooku|m joins
21:15:54schwarzkatz|m joins
21:15:54DigitalDragon (DigitalDragon) joins
21:15:54Sanqui|m (Sanqui) joins
21:15:54britmob|m joins
21:15:54Thibaultmol joins
21:15:54x9fff00 (x9fff00) joins
21:15:54Vokun (Vokun) joins
21:15:54hlgs|m joins
21:15:54thermospheric (Thermospheric) joins
21:15:54qyxojzh|m joins
21:15:54s-crypt|m|m joins
21:15:54Nulo|m joins
21:15:54yzqzss (yzqzss) joins
21:15:54phaeton (phaeton) joins
21:15:54mikolaj|m joins
21:15:54ChanServ joins
21:15:54nuke.hackint.org sets mode: +o ChanServ
21:16:13schwarzkatz|m quits [Max SendQ exceeded]
21:16:13DigitalDragon quits [Max SendQ exceeded]
21:16:13Vokun quits [Max SendQ exceeded]
21:17:25schwarzkatz|m joins
21:17:41DigitalDragon (DigitalDragon) joins
21:17:42Vokun (Vokun) joins
21:20:21qwertyasdfuiopghjkl quits [Client Quit]
21:21:30qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins
21:49:25qwertyasdfuiopghjkl quits [Client Quit]
21:55:54qwertyasdfuiopghjkl (qwertyasdfuiopghjkl) joins
22:38:20Dango360 quits [Quit: Leaving]
22:39:47Dango360 (Dango360) joins

HomeSearchPrevious dayNext day