| 01:46:00 | <joepie91> | btw, SketchCow, I think you may find this useful for keeping track of things: http://www.treesheets.com/ |
| 01:47:00 | <joepie91> | (may also be useful for others, and it runs natively on Linux as well) |
| 02:23:00 | <SketchCow> | Copied FORTUNECITY/com/meltingpot/com-meltingpot-research-20120405-005316.warc.gz to warc |
| 02:23:00 | <SketchCow> | alard: |
| 02:23:00 | <SketchCow> | Checking FORTUNECITY/com/meltingpot/com-meltingpot-gambia-20120401-144041.warc.gz |
| 02:23:00 | <SketchCow> | Could not decompress warc.gz. gunzip returned 2. |
| 02:23:00 | <SketchCow> | Copying FORTUNECITY/com/meltingpot/com-meltingpot-gambia-20120401-144041.warc.gz to tar |
| 02:23:00 | <SketchCow> | So that's good. |
| 02:26:00 | <dashcloud> | did you see my note about the two Coming Soon items? |
| 02:30:00 | <SketchCow> | 17:41 <@dashcloud> so reading the scrollback, I did a brief check of the items, and I came across Coming Soon, which has one item as WARCS, and there's a second item with a WARC file inside a zipfile |
| 02:30:00 | <SketchCow> | That, right? |
| 02:30:00 | <dashcloud> | yes |
| 02:30:00 | <SketchCow> | The thing 6 lines up? |
| 02:30:00 | <dashcloud> | sorry! |
| 02:30:00 | <SketchCow> | Or are you watching joins and parts? |
| 02:30:00 | <SketchCow> | Because I turned THAT shit off MONTHS ago. |
| 02:30:00 | <SketchCow> | I'd have gone insane. |
| 02:32:00 | <SketchCow> | http://archive.org/details/csoon-20111016 this one? |
| 02:32:00 | <SketchCow> | I see. |
| 02:32:00 | <SketchCow> | Yes, it's handled. The csoon-* is a WARC of the same |
| 02:32:00 | <SketchCow> | Good eye, though. |
| 02:33:00 | <dashcloud> | okay |
| 02:37:00 | <joepie91> | ok, seriously, I love scantailor |
| 02:37:00 | <SketchCow> | scantailor fixes everything. |
| 02:37:00 | <joepie91> | yes, pretty much |
| 02:37:00 | <joepie91> | comics, books, it does all of it :o |
| 02:37:00 | <joepie91> | and most of it automated |
| 02:37:00 | <joepie91> | hell, it pretty much successfully cleaned up a book that was copied *on a typewriter* |
| 02:38:00 | <joepie91> | on shitty spotty recycled paper |
| 02:38:00 | <SketchCow> | As my friend Dan Reetz likes to say, sometimes scantailor unwittingly fixes typesetting errors with books |
| 02:38:00 | <joepie91> | heh |
| 02:38:00 | <SketchCow> | Where the plates were off by a millimeter or so |
| 02:38:00 | <joepie91> | SketchCow: http://aarnist.cryto.net:81/vrijheid2.pdf |
| 02:38:00 | <joepie91> | is the result |
| 02:38:00 | <chronomex> | oh yeah I've had books come out less crooked than the original |
| 02:38:00 | <joepie91> | two pages are missing and I should rescan some pages because they were too fuzzy |
| 02:38:00 | <joepie91> | but overall it's VERY nice |
| 02:39:00 | <joepie91> | also, tiff2pdf somehow fucked up the front cover, not sure why :P |
| 02:39:00 | <chronomex> | that is a nice scan. |
| 02:40:00 | <joepie91> | yes, yes it is :) |
| 02:40:00 | <joepie91> | but yeah, a few pages definitely needs fixing |
| 02:40:00 | <joepie91> | need * |
| 02:42:00 | <joepie91> | 109, for example, is a bit meh |
| 02:43:00 | <balrog-> | joepie91: tiff2pdf is picky about input tiff |
| 02:43:00 | <balrog-> | very, very picky |
| 02:47:00 | <joepie91> | yes, so I've noticed |
| 02:47:00 | <joepie91> | I suspect there's some color space fuckup or something |
| 02:48:00 | <joepie91> | what I have noticed that has somewhat surprised me: it's possible to make scans of professional quality on Linux with free software alone |
| 02:48:00 | <joepie91> | from scan to postprocessed PDF |
| 02:48:00 | <joepie91> | and reasonably automate-able |
| 02:48:00 | <balrog-> | joepie91: if you or someone is willing to fix hocr2pdf or write a working alternative, then you can have OCRed too |
| 02:49:00 | <balrog-> | tesseract produces decent output |
| 02:49:00 | <joepie91> | what language is it written in? |
| 02:49:00 | <balrog-> | C |
| 02:49:00 | <joepie91> | ah, not my thing |
| 02:49:00 | <joepie91> | though |
| 02:49:00 | <joepie91> | I may know someone who can do that |
| 02:49:00 | <balrog-> | but there's hOCR-handling stuff in ruby and iirc in python |
| 02:49:00 | <chronomex> | ocropus too |
| 02:49:00 | <joepie91> | will give him a poke :P |
| 02:49:00 | <joepie91> | right |
| 02:49:00 | <balrog-> | does ocropus handle hOCR? |
| 02:49:00 | <chronomex> | idk |
| 02:49:00 | <balrog-> | the OCR step is mostly good |
| 02:49:00 | <balrog-> | the tricky part is putting the hOCR into the PDF |
| 02:49:00 | <joepie91> | speaking of which, a potential nice archiveteam-project: build a fast book scanner with fully automated software 'chain' from scan/photo to OCRed ebook files |
| 02:49:00 | <joepie91> | make it publicly accessible |
| 02:49:00 | <joepie91> | "come turn your book into an ebook here for free" |
| 02:49:00 | <balrog-> | hOCR is the OCRed text in HTML format with tags indicating the location |
| 02:50:00 | <joepie91> | and at the same time, archive/catalogue the scanned books |
| 02:50:00 | <chronomex> | http://en.wikipedia.org/wiki/HOCR says yes, ocropus and tesseract both |
| 02:50:00 | <balrog-> | that's software that OUTPUTS it |
| 02:50:00 | <joepie91> | basically, IRL archiveteam project |
| 02:50:00 | <balrog-> | you need something to input it and stuff it into a PDF |
| 02:50:00 | <chronomex> | yep |
| 02:50:00 | <joepie91> | balrog-: I'll have a look at it some time soon |
| 02:50:00 | <balrog-> | http://www.exactcode.com/site/open_source/exactimage/hocr2pdf/ |
| 02:51:00 | <balrog-> | svn.exactcode.de for the code |
| 02:51:00 | <joepie91> | ok :) |
| 02:51:00 | <joepie91> | but yeah, balrog-, chronomex, thoughts on IRL bookscanning project? |
| 02:51:00 | <balrog-> | well, I'd first need a bookscanner |
| 02:51:00 | <balrog-> | problem is, you don't want to know how many books I have. |
| 02:52:00 | <chronomex> | you have a lot |
| 02:52:00 | <chronomex> | got it |
| 02:52:00 | <joepie91> | well |
| 02:52:00 | <joepie91> | idk if I pasted this, but I ran across a video of a bookscanner |
| 02:52:00 | <joepie91> | that would do the job |
| 02:52:00 | <joepie91> | and I think it should be fairly inexpensive to build |
| 02:52:00 | <chronomex> | the automatic one with the wedge? |
| 02:52:00 | <chronomex> | yeah that's cool |
| 02:52:00 | <joepie91> | yeah |
| 02:52:00 | <chronomex> | dunno about getting the sensors right down at the tip tho |
| 02:52:00 | <joepie91> | all you need is basically a strong servo, a compressor, and two scanner units |
| 02:52:00 | <joepie91> | (I think) |
| 02:52:00 | <chronomex> | s/servo/stepper/ |
| 02:52:00 | <joepie91> | I suck at terms |
| 02:52:00 | <joepie91> | stepper, right |
| 02:53:00 | <joepie91> | terminology* |
| 02:53:00 | <joepie91> | ... wow, that was a self-proving statement lol |
| 02:53:00 | <chronomex> | you need + and - air |
| 02:53:00 | <joepie91> | right, I know some people here that can probably do that |
| 02:53:00 | <joepie91> | and they probably have the parts for it, too |
| 02:54:00 | <joepie91> | but yeah, it would be sort of epic to just have a book scanner somewhere in a public space, where anyone can scan a book and get the resulting ebook emailed to him |
| 02:54:00 | <joepie91> | and at the same time have the source files and postprocessed files archived centrally |
| 02:54:00 | <joepie91> | and judging from the software that is available, that should be fairly easy to automate |
| 02:55:00 | <joepie91> | but then a camera setup would probably be best |
| 02:55:00 | <joepie91> | for starters |
| 02:55:00 | <joepie91> | since the wedge thing is a bit.. large :P |
| 02:56:00 | <chronomex> | yea |
| 02:56:00 | <joepie91> | and while the camera bookscanner can run off some kind of battery, that will be tricky for the wedge model |
| 02:56:00 | <joepie91> | I mean, you could just put the camera bookscanner somewhere outside a mall temporarily |
| 02:57:00 | <joepie91> | and run it off a battery and local storage |
| 03:01:00 | <chronomex> | have it spit out usb sticks or something |
| 03:01:00 | <chronomex> | "insert usb stick or sd card to receive a pdf!" |
| 03:01:00 | <joepie91> | possible as well |
| 03:01:00 | <joepie91> | maybe offer both USB and SD for instant ebook |
| 03:02:00 | <joepie91> | or "give your email and we'll send it at the end of the day" as alternative |
| 03:02:00 | <joepie91> | since USB sticks and SD cards tend to get lost :P |
| 03:03:00 | <joepie91> | combine a custom python script using python-imaging-sane or whatever is needed to take webcam pictures (depending on setup) |
| 03:03:00 | <joepie91> | with postprocessing via scantailor-cli |
| 03:04:00 | <joepie91> | then tiffcp and tiff2pdf |
| 03:04:00 | <joepie91> | and optionally calibre to produce a .mobi and .epub |
| 03:10:00 | <chronomex> | would we trust the user to metadata |
| 03:11:00 | <chronomex> | I don't trust anyone to metadata unless they're 1) a librarian, 2) super picky, or 3) me |
| 03:11:00 | <chronomex> | I suppose 2 is redundant |
| 03:11:00 | <joepie91> | I'd say, let the user give metadata first |
| 03:11:00 | <joepie91> | then review before final archival |
| 03:11:00 | <chronomex> | aye |
| 03:11:00 | <joepie91> | at the end of the day |
| 03:11:00 | <joepie91> | you have to review anyway |
| 03:11:00 | <joepie91> | to get rid of any personal markings |
| 03:11:00 | <joepie91> | owner names, stamps, etc |
| 03:11:00 | <chronomex> | yeah proofing metadata against a title page is pretty straightforward |
| 03:11:00 | <chronomex> | no |
| 03:11:00 | <chronomex> | leave that in |
| 03:12:00 | <joepie91> | that'll cause an issue for people |
| 03:12:00 | <chronomex> | hm? |
| 03:12:00 | <joepie91> | I doubt they'd want their name associated with a scan |
| 03:12:00 | <chronomex> | oh |
| 03:12:00 | <chronomex> | tell them not to scan the bookplate then? |
| 03:12:00 | <joepie91> | that's no use when scanning is automated :P |
| 03:12:00 | <chronomex> | oh |
| 03:12:00 | <joepie91> | most people write their name in the inside |
| 03:12:00 | <chronomex> | ummmmm |
| 03:12:00 | | chronomex shrugs |
| 03:12:00 | <chronomex> | I hadn't considered that |
| 03:12:00 | <joepie91> | you can just blank that out, it's typically not written over any actual book content |
| 03:13:00 | <chronomex> | true |
| 03:13:00 | <joepie91> | same for stamps, they're usually on the inside cover |
| 03:13:00 | <joepie91> | in the blank area |
| 03:13:00 | <chronomex> | you could offer the scanning person an option to do that themselves |
| 03:13:00 | <joepie91> | true |
| 03:13:00 | <joepie91> | but you have to be careful to not introduce too many variables and options |
| 03:14:00 | <joepie91> | or the whole appeal of an ""ebookify your book here" machine will be gone |
| 03:14:00 | <joepie91> | it's a tricky thing to average :P |
| 03:15:00 | <chronomex> | yes |
| 03:18:00 | <joepie91> | good point: if it requires manual pageturning, people won't do it |
| 03:34:00 | <SketchCow> | Tried to get one of you guys a keynote for a conference. |
| 03:34:00 | <SketchCow> | underscor or Chronomex, probably |
| 03:34:00 | <SketchCow> | They wouldn't go for it |
| 03:35:00 | <SketchCow> | Mostly because of the way the place works (they vote on the person, not the organization) |
| 03:35:00 | <SketchCow> | But I tried! |
| 03:35:00 | <SketchCow> | underscor keynoting would be awwwweeessoommmmee |
| 03:35:00 | <SketchCow> | They'd not forget THAT |
| 05:54:00 | <chronomex> | hehe |
| 05:54:00 | <chronomex> | what organization was this? |
| 06:44:00 | <ersi> | ArchiveTeam for president! |
| 07:26:00 | <joepie91> | balrog-, chronomex, good news! |
| 07:26:00 | <chronomex> | oh yeah? |
| 07:26:00 | <joepie91> | I wrote a script to fix the tiff2pdf issue |
| 07:26:00 | <joepie91> | with the discolored PDFs |
| 07:26:00 | <joepie91> | http://pastie.org/5107570 |
| 07:26:00 | <chronomex> | rad |
| 07:26:00 | <joepie91> | does a chunked read of a PDF |
| 07:26:00 | <joepie91> | so it doesn't load all of it in memory at once |
| 07:27:00 | <joepie91> | and replaces a certain string to fix the issue |
| 07:27:00 | <joepie91> | and yes, it handles strings on the border between 2 chunks properly :P |
| 07:27:00 | <joepie91> | if it detects part of the to-be-matched string existing at the end of a chunk |
| 07:27:00 | <joepie91> | it'll read more to get the rest |
| 07:28:00 | <joepie91> | so basically, it always loads at most 512kb of data |
| 07:28:00 | <joepie91> | which means it should be possible to easily process a 1GB PDF if needed |
| 07:28:00 | <joepie91> | without running out of RAM |
| 07:28:00 | <chronomex> | oboy |
| 07:28:00 | <joepie91> | also, I tested it ofc, and it works |
| 07:29:00 | <joepie91> | thanks to these guys: http://www.asmail.be/msg0055295176.html for the fix :P |
| 07:29:00 | <joepie91> | I'll be releasing a few scripts for scanning soon anyway |
| 07:29:00 | <chronomex> | nice |
| 07:30:00 | <chronomex> | I let archive.org's deriver make my pdfs though ;) |
| 07:30:00 | <joepie91> | heh |
| 07:30:00 | <joepie91> | anyway, it also has a simple automation script for scanning |
| 07:30:00 | <joepie91> | interactive CLI script |
| 07:30:00 | <joepie91> | you pick the device from a list, enter DPI, width, height |
| 07:30:00 | <joepie91> | hit enter, and it scans a page |
| 07:31:00 | <joepie91> | hit enter again, and it scans a page |
| 07:31:00 | <joepie91> | saving them as incrementing numbers |
| 07:31:00 | <joepie91> | and a separate script for re-scanning certain pages |
| 07:32:00 | <joepie91> | so, seems I just finished my first comic book scan: http://aarnist.cryto.net:81/straal2.pdf :D |
| 08:07:00 | <SketchCow> | http://sphotos-a.xx.fbcdn.net/hphotos-ash3/46201_4497571931862_1789693667_n.jpg |
| 08:09:00 | <ersi> | SketchCow: gay |
| 08:09:00 | <joepie91> | hahaha |
| 08:10:00 | <joepie91> | also, I *may* have an idea for an ultra-cheap camera-based book scanner... but I'll have to see if the camera I have in mind is suitable. |
| 08:10:00 | <joepie91> | so... searching through boxes it is |
| 08:24:00 | <joepie91> | interesting... I actually get pictures of reasonable quality with this camera |
| 08:26:00 | <joepie91> | after postprocessing: http://i.imgur.com/qtX9w.png |
| 08:29:00 | <joepie91> | I wonder what kind of pictures I could get from this camera with a bit of optimization |
| 08:52:00 | <SmileyG> | tht hurts my eyes to look at ¬_¬ |
| 08:52:00 | <chronomex> | joepie91: I bet alignment would help too |
| 09:04:00 | <joepie91> | chronomex: problem is this is only 640 * 480 |
| 09:04:00 | <joepie91> | and the focus isn't great |
| 09:04:00 | <chronomex> | oh |
| 09:04:00 | <joepie91> | because it obviously doesn't have autofocus |
| 09:04:00 | <joepie91> | this thing *should* have a photo mode that does 1280x1024 photos |
| 09:04:00 | <joepie91> | but it's behaving quite strangely |
| 09:04:00 | <joepie91> | it goes into photo mode, but when I press the button it'll still just make a video |
| 09:04:00 | <joepie91> | instead of taking a photo |
| 09:04:00 | <joepie91> | :| |
| 09:04:00 | <joepie91> | frustrating |
| 09:05:00 | <joepie91> | it's this camera: http://www.chucklohr.com/808/C3/index.html |
| 09:05:00 | <joepie91> | it's an awesome little camera otherwise but it's focused at far objects |
| 09:06:00 | <joepie91> | so doesn't cope with book text too well :P |
| 10:11:00 | <SmileyG> | "hope it can help your life safe and happiness" - wut? :D |
| 11:25:00 | <joepie91> | SmileyG: that's a play on the messages from Chinese eBay sellers |
| 11:25:00 | <joepie91> | lol |
| 11:35:00 | <SmileyG> | :D |
| 15:02:00 | <SketchCow> | So, here we are deep into the WARC transfer of material, either my backhack conversions of previous projects, or the webshots upload. |
| 15:03:00 | <SketchCow> | I'm now waiting to see if anyone yells about the loading of the data, or the system or anything. |
| 15:03:00 | <SketchCow> | But looks like we have quite a lot to give them, and who knows. |
| 15:04:00 | <godane> | i uploaded 2 more linux format dvds this morning |
| 15:04:00 | <godane> | http://archive.org/details/cdrom-linuxformatmagazine-128 |
| 15:05:00 | <godane> | http://archive.org/details/cdrom-linuxformatmagazine-136 |
| 15:05:00 | <SketchCow> | No need to tell me, godane. I'll get to you on my next sweep of you. |
| 15:05:00 | <godane> | ok |
| 15:05:00 | <godane> | i just feel better that my wifi is working again |
| 15:16:00 | <underscor> | hahaha |
| 15:16:00 | <underscor> | SketchCow: that would be awesome |
| 15:16:00 | <underscor> | although |
| 15:16:00 | <underscor> | I have not a lot of experience speaking |
| 15:28:00 | <SketchCow> | I'd have coached you. |
| 15:36:00 | <underscor> | <3 |
| 17:30:00 | <balrog-> | :/ http://www.idigitaltimes.com/articles/12066/20121022/nbc-erases-snl-sketch-digital-archive-copyright.htm |
| 17:43:00 | <SketchCow> | https://twitter.com/shaneb/status/261159783921483776 |
| 17:44:00 | <SketchCow> | balrog-: Non discussion |
| 18:24:00 | <joepie91> | balrog-: http://i.imgur.com/GVajj.png |
| 18:24:00 | <balrog-> | joepie91: yeah I noticed |