00:52:00<godane>balrog_: i just mirrored it
01:26:00<godane>balrog_: uploaded: http://archive.org/details/www.mindsetcomputing.com-20130104-mirror
01:27:00<balrog_>awesome :)
01:38:00<db48x>heh, http://thewholegoddamnedinternet.com has a cvs keyword substitution at the bottom
01:38:00<db48x>that's dedication
01:58:00<godane>uploaded: http://archive.org/details/www.raygunrevival.com-20130103-mirror
02:15:00<godane>you guys may want this: https://www.youtube.com/user/TorontoPETUsersGroup
02:16:00<godane>world of commodore video talks
02:20:00<hiker1>wow there are still talks about commodore.
02:43:00<hiker1>godane: What options do you typicall use on wget?
03:01:00<godane>this is what i used for rayguyrevival.com grab: wget "http://$website/Published/" --mirror --warc-file=$website-$(date +%Y%m%d) --reject-regex='(replytocom)' --warc-cdx -E -o wget.log
03:02:00<godane>website="www.raygunrevival.com"
03:02:00<hiker1>thanks
03:10:00<hiker1>wow that site was big
03:16:00<godane>i had to get the Published folder first cause no of the .pdf links go there
03:16:00<godane>*no=none
03:17:00<godane>anyways i'm uploading my arstechnica image dumps
03:28:00<godane>http://forums.g4tv.com/showthread.php?187820-Save-G4TV-com-%28the-website%29!
03:28:00<godane>i'm doing my best the save the fourms but you guys need to get the videos
03:29:00<godane>not every video has a .mp4 file in it
04:03:00<S[h]O[r]T>someone do the hard work for making a script to save the video :P
04:06:00<godane>i'm trying to figure it out now
04:06:00<godane>best i can get is the url folder
04:07:00<godane>like for http://www.g4tv.com/videos/37668/bible-adventures-gameplay/
04:07:00<godane>the video file will be in http://vids.g4tv.com/videoDB/037/668/video37668/
04:08:00<godane>but nothing in the html gives the name
04:08:00<godane>the filename i mean
04:10:00<S[h]O[r]T>http://vids.g4tv.com/videoDB/037/668/video37668/tr_bible_adventures_FS_flv.flv.
04:10:00<godane>how did you get the filename?
04:11:00<godane>i normally have to download it using a plugin in firefox
04:12:00<S[h]O[r]T>^^ you can prob use chromes inspector too. i should have thought of that..but i used a program called streamtransport that pulls the flash http://www.streamtransport.com/
04:12:00<S[h]O[r]T>question is either can we guess the filename at all or predict it based on their title.
04:13:00<S[h]O[r]T>if you do chrome > inspect element and then network tab you can see if it if you catch it as it loads
04:18:00<S[h]O[r]T>http://www.g4tv.com/xml/BroadbandPlayerService.asmx/GetEmbeddedVideo?videoKey=37668&strPlaylist=&playLargeVideo=true&excludedVideoKeys=&playlistType=normal&maxPlaylistSize=0
04:19:00<S[h]O[r]T>just key in the id # into videoKey, and its in FilePath http://www.g4tv.com/VideoRequest.ashx?v=37668&r=http%3a%2f%2fvids.g4tv.com%2fvideoDB%2f037%2f668%2fvideo37668%2ftr_bible_adventures_FS_flv.flv
04:19:00<godane>fucking a
04:20:00<S[h]O[r]T>i imagine i can run through a list of ids and get all the xml
04:21:00<godane>ok
04:21:00<S[h]O[r]T>and then grep out the filepaths and then wget that list of videos
04:21:00<godane>yes
04:22:00<godane>but i can also imagine that we can create a script that will download file and then up it to archive.org
04:22:00<godane>cause we have desc and date of post
04:23:00<godane>also title
04:23:00<godane>the url for archiving will best to be something like video37668_$filename or something
04:24:00<godane>also grab the thumbnail image too and uploaded with it
04:24:00<godane>there are tons of early techtv stuff on there
04:25:00<godane>some full episodes in 56k
04:27:00<godane>gamespot tv episode in 56k: http://www.g4tv.com/videos/24430/thief-2-pc-review/
04:28:00<S[h]O[r]T>yeah trying to figure out why wget is not keep the rest of the url in my range
04:28:00<S[h]O[r]T>http://www.g4tv.com/xml/BroadbandPlayerService.asmx/GetEmbeddedVideo?videoKey={0..5}&strPlaylist=&playLargeVideo=true&excludedVideoKeys=&playlistType=normal&maxPlaylistSize=0
04:28:00<S[h]O[r]T>cutsoff after the range in braces
04:29:00<S[h]O[r]T>i guess maybe thats not wget but bash related
04:30:00<S[h]O[r]T>well w/e i can just generate a list of urls
04:39:00<godane>i'm making a warc of the xml output
04:39:00<S[h]O[r]T>of everything?
04:39:00<S[h]O[r]T>*every id
04:39:00<godane>i think so
04:39:00<S[h]O[r]T>doing what
04:40:00<godane>the latest id is 61790
04:40:00<godane>thats on the front page
04:43:00<S[h]O[r]T>what did you do to generate the list
04:44:00<godane>i planning on zcat *.warc.gz | grep 'vids.g4tv.com' or somthing
04:44:00<S[h]O[r]T>i mean the list of urls to leech the xmls
04:45:00<S[h]O[r]T>are you going to download all the videos too or dont have the bw?
04:46:00<godane>i don't think i can download all the videos
04:46:00<db48x>S[h]O[r]T: that url has an ampersand in it
04:47:00<db48x>put the url in double quotes
04:47:00<S[h]O[r]T>like ""http://"" or do you just mean "http"
04:47:00<S[h]O[r]T>i cant do "" since the range doesnt execute, it takes it literally
04:48:00<db48x>the alternative is to escape the ampersands with backslashes
04:48:00<db48x>\&strPlaylist=
04:48:00<db48x>actually, I dunno if that will work; lemme test
04:48:00<S[h]O[r]T>yeah, ill try that. ive got a php script going generating 0-99999 but taking awhile :P
04:49:00<db48x>yes, it will
04:49:00<db48x>err, ok
04:49:00<db48x>why not just use a shell loop?
04:49:00<S[h]O[r]T><---half retard
04:49:00<S[h]O[r]T>if it works or works eventually good enough for me :P
04:50:00<db48x>ok :)
04:50:00<godane>i can seed out the list now :-D
04:50:00<S[h]O[r]T>or if godane would type faster :P
04:51:00<db48x>for future reference:
04:51:00<S[h]O[r]T>ill take whatever list you have. ive got the bw/space to leech all the videos
04:51:00<db48x>for (( i=1; i<=99999; i++ ))
04:51:00<db48x>do
04:52:00<db48x>wget ... "http://example.com/$i"
04:52:00<db48x>done
04:52:00<godane>i did for i in $(seq 1 61790); do
04:53:00<godane>echo "http://example.com/$i" >> index.txt
04:53:00<godane>done
04:53:00<S[h]O[r]T>your probably right thats the last id but you never know whats published and isnt
04:53:00<godane>wget -x -i index.txt -o wget.log
04:57:00<S[h]O[r]T>some of the files return videos are no longer active vs doesnt exist
06:07:00<db48x>http://retractionwatch.wordpress.com/2013/01/03/owner-of-science-fraud-site-suspended-for-legal-threats-identifies-himself-talks-about-next-steps/
06:08:00<db48x>might be too late for the original site, but the planned site is something we should make sure to save
06:59:00<joepie91>db48x: shot him an email
06:59:00<joepie91>who knows, maybe I can help out hosting the original stuff
07:26:00<underscor>No idea if this has been linked in here before
07:26:00<underscor>but it's a fun toy
07:26:00<underscor>http://monitor.us.archive.org/weathermap/weathermap.html
08:32:00<chronomex>spiffy
08:42:00<Nemo_bis>grr doesn't load
08:42:00<Nemo_bis>ah
08:43:00<Nemo_bis>oh, so when the traffic with HE is big I know s3 will suck
08:44:00<Nemo_bis>[doesn't work with https]
12:12:00<alard>I have a Yahoo blogs question.
12:12:00<alard>However, at least some of the blogs also have a friendlier name, see
12:12:00<alard>The problem is this: each blog is available via the GUID of its owner, e.g.
12:12:00<alard>http://blog.yahoo.com/_FWXMIGZ6AAZ66QJABH727HC4HM/
12:12:00<alard>http://profile.yahoo.com/FWXMIGZ6AAZ66QJABH727HC4HM/
12:12:00<alard>which links to this profile here
12:12:00<alard>http://blog.yahoo.com/lopyeuthuong/
12:12:00<alard>It's the same blog.
12:12:00<alard>There's a link from the friendly-name version to the profile, so it's possible to go from a name to a GUID. I'd like to go from a GUID to a name. Any ideas?
12:12:00<alard>(Context: tuankiet has been collecting Vietnamese Yahoo blogs to be archived, with a lot of GUIDs. If it's possible I'd also like to archive the blogs via their friendly URLs.)
12:21:00<chronomex>huh.
12:23:00<hiker1>alard: Using warc-proxy, one of the records I have in a warc file it sits trying to serve it to the browser but never sends it. If I remove the record it works. Could you take a look at it?
12:24:00<alard>hiker1: Yes, where is it?
12:25:00<hiker1>Give me one minute to upload it + a version that has the record removed.
12:30:00<hiker1>alard: http://www.fileden.com/files/2012/10/26/3360748/Broken%20Warc.zip
12:32:00<hiker1>alard: A different bug I found is that if the root record does not have a '/' at the end, warc-proxy will not serve it.
12:33:00<hiker1>And it becomes impossible to access that record, at least through chrome. Chrome always seems to request the site with the / at the end
12:34:00<alard>You mean something like: WARC-Target-URI: http://example.com
12:34:00<hiker1>yes.
12:35:00<alard>It's very tempting to say that that's a bug in the tool that made the warc.
12:36:00<hiker1>My program does not require it, so perhaps it is my fault. But I could see other tools not requiring it, so perhaps the behavior should be changed in both places.
12:37:00<alard>Apparently the / is not required: https://tools.ietf.org/html/rfc2616#section-3.2.2
12:39:00<alard>(Should we do this somewhere else?)
12:40:00<alard>Could it be that the record you removed is the only record without a Content-Length header?
12:40:00<hiker1>the WARC record has a content-length header
12:40:00<hiker1>the HTTP header does not
12:44:00<alard>Do you have a modified version that can handle uncompressed warcs?
12:44:00<ersi>I've had the same 'problem' with a Target-URI missing /
12:44:00<ersi>warc-proxy handles uncompressed records just fine for what I've tried
12:44:00<hiker1>alard: I'm not sure I understand you
12:45:00<alard>We're talking bout this warc-proxy, right? https://github.com/alard/warc-proxy At least my version of the Firefox extension doesn't show .warc, only .warc.gz.
12:45:00<hiker1>It's always shown .warc for me.
12:46:00<hiker1>I use Chrome
12:46:00<ersi>I was using it without the firefox extension. Just sat the proxy, then browsed http://warc/
12:46:00<alard>Ah, I see. I'm using the firefox extension (which uses a different filename filter).
12:47:00<ersi>alard: By the way, http://warc/ is beautiful :]
12:47:00<alard>Thanks. (It's the same as in the Firefox extension, except for the native file selection dialog.)
12:48:00<hiker1>Does Hanzo warctools even check Content-Length? I don't think so
12:51:00<hiker1>Hanzo warctools does adds it to a list of headers
12:56:00<hiker1>alard: in warc-proxy, the content-length for the 404 is set to 12. It should be 91. Around line 257.
12:56:00<alard>What may be happening is this: if there's no Content-Length header in the HTTP response, the server should close the connection when everything is sent. The warc-proxy doesn't close the connection.
12:56:00<hiker1>That seems very likely.
13:00:00<hiker1>alard: In Chrome's network view, it shows the connection is never closed.
13:01:00<alard>Yes. I'm currently trying to understand https://tools.ietf.org/html/rfc2616#section-14.10
13:06:00<godane>uploaded: http://archive.org/details/CSG_TV-Game_Collection_Complete_VHS_-_1996
13:52:00<hiker1>alard: I fixed it by having warc-proxy calculate the content-length if it isn't already present in a header.
13:54:00<hiker1>It introduces a little bit of complexity to the __call__ method because now it uses parse_http_response
13:56:00<hiker1>Alternatively, and perhaps a better option, would be for warc-proxy to forcibly close the connection, since this is what the server must do since it does not send a content-length.
14:04:00<godane>so i got the vidoe file url list
14:04:00<godane>for the g4 videos
14:16:00<hiker1>alard: you still there?
14:18:00<alard>Yes. I was concentrating. :) I've now changed the warc-proxy so it sends the correct, reconstructed headers.
14:18:00<hiker1>I found that there is a very easy fix
14:19:00<hiker1>http_server = tornado.httpserver.HTTPServer(my_application, no_keep_alive=True)
14:19:00<hiker1>This tells Tornado to close all connections regardless of what version is used
14:19:00<hiker1>*http version
14:19:00<alard>That's not a fix, that's a hack. :) If you disable keep-alive the request shouldn't have a Connection: keep-alive header.
14:20:00<hiker1>Tornado recommends it
14:20:00<godane>alard: i got g4tv video url list
14:20:00<godane>alard: i need you to make a script for downloading and uploading to archive.org
14:20:00<alard>hiker1: https://github.com/alard/warc-proxy/commit/9d107976ccd47c244669b5e680d67a5caf6e103c
14:20:00<hiker1>`HTTPServer` is a very basic connection handler. Beyond parsing the HTTP request body and headers, the only HTTP semantics implemented in `HTTPServer` is HTTP/1.1 keep-alive connections. We do not, however, implement chunked encoding, so the request callback must provide a ``Content-Length`` header or implement chunked encoding for HTTP/1.1 requests for the server to run correctly for HTTP/1.1 clients. If the request handler is una
14:20:00<hiker1>ble to do this, you can provide the ``no_keep_alive`` argument to the `HTTPServer` constructor, which will ensure the connection is closed on every request no matter what HTTP version the client is using.
14:21:00<alard>hiker1: (Plus the commits before.)
14:22:00<alard>godane: Great. If I may ask, why don't you write the script yourself?
14:22:00<hiker1>Either method should work. I think I prefer the no_keep_alive method because then it sends exactly what was received from the server. It's also simple
14:22:00<alard>It's a good reason to learn it, and people here are probably more than happy to help you.
14:23:00<alard>Yes, but why does it have to send the exact response from the server?
14:23:00<hiker1>alard: That's true. As long as the exact response is saved, it doesn't matter.
14:23:00<hiker1>*saved in the WARC already.
14:24:00<hiker1>thank you for fixing it upstream. I think this has been the root of a few problems I've been having
14:28:00<hiker1>yep. other hacks I added are no longer needed
14:28:00<hiker1>Some of the google ad scripts were causing failures with warc-proxy but are now resolved
14:34:00<hiker1>alard: One other fix: line 193 changed to if mime_type and data["type"] and not mime_type.search(data["type"]):
14:34:00<godane>my www.eff.org warc is still going to live site with warc-proxy
14:36:00<hiker1>Some of them don't have a type set, so that line throws an error
14:38:00<alard>hiker1: Shouldn't that be "type" in data then?
14:38:00<godane>it still looks like https will always go to live web
14:39:00<alard>godane: Ah yes, that was the other bug.
14:43:00<hiker1>alard: sure
14:43:00<hiker1>er...
14:43:00<hiker1>data["type"] is sometimes None.
14:43:00<hiker1>I think "type" in data would return True
14:43:00<alard>So it should be: "type" in data and data["type"]
14:44:00<hiker1>yes
14:45:00<alard>Actually, there's always a "type" field (line 113), so data["type"] will do.
14:47:00<godane>alard: so now typing urls for archives doesn't work with warc-proxy
14:47:00<godane>i using localhost port 8000 for my proxy
14:47:00<godane>in less thats the problem
14:52:00<alard>godane: What do you mean with typing urls for archives?
14:54:00<alard>hiker1: I think the mime-type is None thing works now.
14:54:00<godane>when i run it proxy i use to type urls in that i know the archive will work with
14:58:00<alard>Actually, it seems completely broken at the moment.
14:58:00<alard>(If you remove the cached .idx files, that is.)
15:04:00<SketchCow>Morning
15:08:00<alard>Afternoon.
15:08:00<godane>hey SketchCow
15:09:00<godane>all my image dumps of arstechinca are uploaded now
15:10:00<alard>hiker1: Any idea where these "trailing data in http response" messages come from? I only get them with your warcs.
15:16:00<Nemo_bis>http://www.apachefoorumi.net/index.php?topic=68497.0
15:25:00<SketchCow>Thank you, godane
15:25:00<godane>your welcome
15:26:00<godane>now we good dumps of arstechnica and engadget
15:26:00<godane>also a few of torrentfreak
15:28:00<hiker1>alard: I think my content-length is too short
15:28:00<hiker1>WARC content length
15:29:00<alard>hiker1: There are a lot of \r\n at the end of the response record.
15:29:00<hiker1>alard: What do you mean?
15:29:00<hiker1>Are you saying I have too many?
15:30:00<alard>Perhaps. I thought there should be two, so \r\n\r\n, but you have four, \r\n\r\n\r\n\r\n. You also have four bytes too many, so that matches.
15:31:00<alard>Does the gzipped data always end with \r\n\r\n?
15:31:00<hiker1>I don't know.
15:32:00<hiker1>I fixed the extra \r\n in my program
15:32:00<hiker1>no more trailing data notices
15:33:00<alard>The warc standard asks for \r\n\r\n at the end of the record.
15:34:00<hiker1>Is that included in the block, or after the record?
15:34:00<hiker1>as in, should the WARC length include it or not include it
15:34:00<alard>No, that's not included in the block.
15:35:00<hiker1>hmm
15:35:00<alard>I think the Content-Length for the warc should not include the two \r\n.
15:36:00<alard>Since you have (or had?) \r\n\r\n + \r\n\r\n in your file, the first \r\n\r\n is from the HTTP response.
15:36:00<hiker1>So is having four of them correct?
15:37:00<alard>I think you only need two \r\n, since the gzipped HTTP body just ends with gzipped data.
15:38:00<hiker1>Yes, I think it should only be two.
15:39:00<hiker1>alard: Should I decompress the gzipped content?
15:39:00<hiker1>sorry. Should I decompress the gzipped content body of the HTTP response?
15:39:00<hiker1>or should I have set the crawler to not accept gzip in the first place?
15:42:00<alard>If you decompress, you should also remove the Content-Encoding header. But I wouldn't do that (you're supposed to keep the original server response).
15:42:00<alard>Not sending the Accept-Encoding header might be better.
15:42:00<hiker1>That spends a lot of bandwidth. I think I will leave it compressed.
15:45:00<alard>This is from the Heritrix release notes: "FetchHTTP also now includes the parameter 'acceptCompression' which if true will cause Heritrix requests to include an "Accept-Encoding: gzip,deflate" header offering to receive compressed responses. (The default for this parameter is false for now.)
15:45:00<alard>As always, responses are saved to ARC/WARC files exactly as they are received, and some bulk access/viewing tools may not currently support chunked/compressed responses. (Future updates to the 'Wayback' tool will address this.)"
15:46:00<alard>So it's probably okay to ask for gzip.
15:48:00<S[h]O[r]T>godane how many videos in your list
15:49:00<godane>will check
15:49:00<godane>37616
15:49:00<S[h]O[r]T>ok
15:49:00<S[h]O[r]T>i got the same amount :)
15:49:00<hiker1>that's a lot of videos.
15:50:00<godane>also know that the hd videos may not be in this
15:50:00<S[h]O[r]T>theres a like a button to enable hd?
15:50:00<godane>yes
15:50:00<S[h]O[r]T>link to an hd video if you got one offhand
15:50:00<hiker1>alard: The project I'm working on is at https://github.com/iramari/WarcMiddleware in case you are interested in trying it
15:51:00<godane>not all video is hd
15:51:00<godane>or hd option
15:51:00<godane>but i think most hd videos you just change _flv.flv to _flvhd.flv
15:52:00<balrog_>any chance someone can look at the yahoo group archiver and see what's wrong with it?
15:52:00<balrog_>I'm no good at perl
15:52:00<hiker1>No one is truly good at perl.
15:52:00<balrog_>http://grabyahoogroup.svn.sourceforge.net/viewvc/grabyahoogroup/trunk/yahoo_group/
15:53:00<balrog_>specifically grabyahoogroup.pl
15:53:00<hiker1>Yep. that's definitely Perl.
15:53:00<balrog_>there are TONS of good stuff on yahoo groups, and most groups require registration and approval to see it :/
15:54:00<balrog_>(why? any group that doesn't require approval will get spammed)
15:55:00<alard>godane: With the current warcproxy, entering urls seems to work for me, so I can't find your problem.
15:55:00<alard>Can you click on the urls in the list?
15:55:00<hiker1>Yes, I am able to enter urls as well.
15:56:00<hiker1>be sure that the url has a trailing '/', and also include www. if the url requires it
15:56:00<alard>Also, SSL/https is difficult, it seems. The proxy probably would have to simulate the encryption.
15:56:00<hiker1>the trailing / is only need for the root of the website
15:56:00<alard>Is the / still needed?
15:56:00<hiker1>alard: or rewrite it all to http
15:56:00<hiker1>alard: did you change it?
15:57:00<alard>I tried to, yes, but that may have introduced new problems: https://github.com/alard/warc-proxy/commit/33ca7e5e30722e8ee40e6a0ed9d3828b82973171
15:58:00<alard>(You'll have to rebuild the .idx files.)
15:58:00<hiker1>testing
15:58:00<hiker1>alard: I think your code adds it to every url
15:59:00<alard>Rewriting https to http would work, perhaps, but once you start rewriting you don't really need a proxy.
16:00:00<hiker1>nevermind. Yes, that fix works
16:01:00<godane>its weird
16:01:00<S[h]O[r]T>i guess we can just re-run the list with flvhd after. im trying to see if the xml supports getting the hd videourl then can just have a seperate list and not have to run through 37k urls trying
16:01:00<godane>i'm looking at my romshepherd.com dump
16:02:00<godane>i get robots.txt and images just fine
16:02:00<alard>godane: Is the .warc.gz online somewhere?
16:02:00<godane>no
16:03:00<godane>its a very old one
16:03:00<godane>but you can use my stillflying.net warc.gz
16:03:00<godane>https://archive.org/details/stillflying.net-20120905-mirror
16:03:00<S[h]O[r]T>alard, how did you find out the GUIDs of the blogs
16:04:00<alard>Searching with Google/Yahoo.
16:04:00<alard>For some blogs you find the blog name, for others the GUID.
16:05:00<S[h]O[r]T>do you have another blog by guid you can link me to?
16:08:00<godane>also opening a url folder takes a very long time now
16:09:00<alard>S[h]O[r]T: http://tracker.archiveteam.org:8124/ids.txt
16:09:00<balrog_>if anyone wants to take a shot at fixing that yahoo group downloader, I would greatly appreciate it
16:10:00<SketchCow>Just wanted to drop this in here before I go offline due to stupid wireless.
16:11:00<SketchCow>I'd like to do a project to make a Wiki and translate metadata to and from Internet Archive to it.
16:11:00<SketchCow>So we can finally get decent metadata.
16:12:00<SketchCow>Since I expect the actual site to have collaborated metadata 15 minutes after never, a moderated way for it to happen will be a good second chance.
16:13:00<SketchCow>Anyway, something to think about, I know I have.
16:14:00<alard>hiker1: So you've probably removed this? https://github.com/iramari/WarcMiddleware/blob/master/warcmiddleware.py#L81
16:15:00<hiker1>I did not. That entire file is sort of deprecated.
16:15:00<hiker1>alard: This is the version that is used https://github.com/iramari/WarcMiddleware/blob/master/warcclientfactory.py
16:16:00<hiker1>At the bottom of this recent commit was the change: https://github.com/iramari/WarcMiddleware/commit/bb69bbf6c19bbc7df150f4bc671e7406257eb750
16:16:00<alard>Oh, okay. I was happy I found something, but that commit seems fine. There's no reason to add these \r\n's.
16:18:00<hiker1>What do you think about the project? Using scrapy + WARC?
16:19:00<alard>I didn't know Scrapy, but adding WARC support to web archivers is always a good idea, I think.
16:21:00<alard>Why do you have the DownloaderMiddleware if that's not recommended?
16:21:00<hiker1>because that was what I built first
16:24:00<hiker1>I should probably delete the file.
16:27:00<alard>Yes, if it creates substandard warcs.
16:28:00<alard>Although if I look at the Scrapy architecture, DownloaderMiddleware seems to be the way to do this, and the ClientFactory is a hack.
16:28:00<hiker1>I'm afraid it does. Scrapy does not seem to send enough information through its DownloaderMiddleware interface to properly create WARC files
16:28:00<alard>Then modify Scrapy!
16:29:00<hiker1>hah
16:29:00<hiker1>It's only a two line fix I think
16:29:00<hiker1>Scrapy skips saving what it calls the start_line which contains HTTP 1/1 or whatever with the 200 OK
16:29:00<hiker1>so it throws that out.
16:30:00<hiker1>which is annoying if you are trying to create a WARC file.
16:30:00<hiker1>ClientFactory does seem a bit of a hack, but it lets me capture the raw data without any parsing or reconstruction.
16:31:00<alard>Can't you send a patch to the Scrapy people?
16:31:00<hiker1>probably. I never contribute to projects much though.
16:35:00<alard>You have to start with something.
16:36:00<hiker1>telling someone in IRC is so much easier though
16:36:00<hiker1>:)
16:36:00<hiker1>I just learned how to clone someone else's github repo.
16:38:00<alard>Heh.
16:38:00<hiker1>alard: Isn't it better to be able to grab the raw data w/ the ClientFactory than to reconstruct it from headers?
16:39:00<alard>godane: Like you said, the warcproxy doesn't like your stillflying.net warc. It's really slow.
16:40:00<alard>hiker1: Yes. So I'd think you would need to modify Scrapy so that it keeps the raw data. I don't know how Scrapy works, so I don't know where you'd put something like that, but I assume some data gets passed on.
16:41:00<hiker1>Some information is passed on, yes. I'm not sure the Scrapy devs would want to expose the raw data. Maybe they'd be okay with it, I don't know.
16:42:00<alard>You can always submit the patch and see what happens. What would be wrong about keeping the raw data?
16:42:00<hiker1>They like exposing the parsed data.
16:46:00<hiker1>I'm looking in to how the data could be passed.
16:49:00<alard>As a property of the Request and Response? https://scrapy.readthedocs.org/en/latest/topics/request-response.html
17:16:00<hiker1>alard: The only way to save the raw data would be to basically merge parts of the clientfactory I wrote into the actual one to add a raw_response argument to Request and Response.
17:17:00<hiker1>As it stands, the DownloaderMiddleware version saves gzipped responses uncompressed, because Scrapy uncompresses them before handing them off to the middleware.
17:20:00<alard>hiker1: And if you change the priority of the WarcMiddleware?
17:37:00<hiker1>alard: that worked
17:37:00<hiker1>good idea
17:37:00<alard>You'll probably also want to get to the response before the redirect-handler etc. do anything.
17:38:00<hiker1>It is changing the order of the HTTP headers
17:39:00<hiker1>probably because it's storing them in a list and reconstructing them
18:13:00<alard>godane: Your warc file is faulty. It was made with one of the early Wget-warc versions, so it contains Transfer-Encoding: chunked responses with 'de-chunked' responses.
18:27:00<godane>oh
18:28:00<godane>but i was able to browse it before in warc-proxy
18:35:00<godane>also i was using wget 1.14
18:36:00<godane>so this could mean all warc since august are faulty
18:36:00<godane>please make warc-proxy work with bad warcs
18:37:00<godane>cause if this bug is in wget 1.14 then all warc will have problems
18:38:00<hiker1>we should probably write a clean-warc tool to fix malformed warcs.
18:40:00<godane>i'm doing another mirror of stillfly.net
18:40:00<godane>if it fails this time i blame warc-proxy
18:45:00<godane>does this patch fix it: https://github.com/alard/warc-proxy/commit/8717f33b642f414de896dcafb2e91a3dc27c38ca
18:55:00<godane>so that patch is not working
19:38:00<godane>alard: ok so i'm testing warc-proxy in midori
19:38:00<godane>its working a lot faster here but
19:39:00<godane>but using proxy in midori will block everything
19:39:00<godane>even localhost:8000
19:40:00<godane>ok got it working
19:40:00<godane>can't use localhost:8000 when using proxy
19:43:00<godane>alard: i get connection reset by peer error with warc-proxy
19:49:00<hiker1>midori?
19:55:00<godane>its a webgtk browser
19:56:00<godane>part of my problem with midori is no pages load
19:56:00<godane>i did get view in stillflying.net but nothing in torrentfreak.com
20:25:00<Jizzday_c>www.jizzday.com
20:50:00<alard>godane is no longer here, but here's a correction: the warc file is fine, but the warcproxy removes the chunking.