| 00:40:00 | <asdflol> | hai :3 |
| 01:32:00 | <bsmith095> | joepie91: youtube-dl entire user? wow |
| 01:32:00 | <joepie91> | bsmith095: yush, download is done |
| 01:33:00 | <joepie91> | I'm uploading atm |
| 01:33:00 | <joepie91> | it's 11GB of video in total |
| 01:33:00 | <joepie91> | so will take an hour or two |
| 01:33:00 | <bsmith095> | joepie91: whats the syntac for downloading an entire user yout=ubedl |
| 01:34:00 | <joepie91> | youtube-dl http://url/to/user/page/here |
| 01:34:00 | <joepie91> | :P |
| 01:34:00 | <joepie91> | it detects it as a user page and fetches all videos |
| 01:35:00 | <DFJustin> | playlists work too |
| 01:35:00 | <bsmith095> | anyone knoe how to run thw warrior on ec2, they have a free teir and im taking advantage of it :)) |
| 01:36:00 | <joepie91> | bsmith095: I don't know how to run the warrior on there, but I *do* know how to run the webshots grabber on it :P |
| 01:36:00 | <joepie91> | if that's fine too |
| 01:36:00 | <bsmith095> | gnu tls is missing |
| 01:37:00 | <bsmith095> | ok, how |
| 01:37:00 | <joepie91> | bsmith095: are you on debian? |
| 01:37:00 | <joepie91> | or ubuntu, will probably work on that as well |
| 01:37:00 | <joepie91> | assuming the packages are named the same |
| 01:38:00 | <bsmith095> | ubuntu ... um 12 i think i686 |
| 01:38:00 | <joepie91> | okay, try running this: |
| 01:38:00 | <joepie91> | wget http://cryto.net/projects/webshots/webshots_debian.sh && chmod +x webshots_debian.sh && ./webshots_debian.sh |
| 01:38:00 | <joepie91> | with a bit of luck it'll work on ubuntu |
| 01:39:00 | <bsmith095> | i have the repo git cloned |
| 01:39:00 | <bsmith095> | run these where? in the repo folder |
| 01:40:00 | <joepie91> | nono |
| 01:40:00 | <joepie91> | as root, somewhere |
| 01:40:00 | <joepie91> | in /root for example |
| 01:40:00 | <joepie91> | it'll set up an unprivileged user |
| 01:40:00 | <joepie91> | install all dependencies, clone the right repo in the right place, compile stuff, etc |
| 01:41:00 | <DFJustin> | running the warrior is strongly preferred so we get consistent data |
| 01:41:00 | <bsmith095> | im running it whrer i mounted my ebs dirve |
| 01:41:00 | <bsmith095> | wow, i should spell check |
| 01:43:00 | <bsmith095> | DFJustin: is there a howto for ec2-based warriors? |
| 01:43:00 | <DFJustin> | not that I know of |
| 01:50:00 | <bsmith095> | i just checked, and amazon ec2 vm import export only supports windows vms??!?! weird |
| 01:54:00 | <underscor> | bsmith095: Running it outside of the warrior is easy |
| 01:58:00 | <bsmith095> | joepie91: whats the archiveteam user password |
| 01:58:00 | <joepie91> | none is set, you can just su into it from root |
| 01:58:00 | <joepie91> | you can set a password for it if you want |
| 02:13:00 | <Cameron_D> | "with a bit of luck it'll work on ubuntu" I had no problems installing on Ubuntu 11.10 (That weren't my own fault) |
| 02:18:00 | <bsmith095> | joepie91: put all the installing stuff in the setup script, its much easier |
| 02:18:00 | <joepie91> | bsmith095: what do you mean? |
| 02:19:00 | <asdflol> | joepie91: look at jabber you butt |
| 02:19:00 | <asdflol> | :3<3 |
| 02:19:00 | <joepie91> | lol |
| 02:19:00 | <joepie91> | ok :P |
| 02:20:00 | <bsmith095> | the apt-get installing from the debian_webshots script, put that in setup, so you can run it without making a new user, ists more portable that way |
| 02:24:00 | <bsmith095> | akk the data is inside webshots-grab right? |
| 02:45:00 | <joepie91> | bsmith095: yes |
| 02:45:00 | <joepie91> | also |
| 02:45:00 | <joepie91> | you can't apt-get unless you have root privs |
| 02:45:00 | <joepie91> | that's why it's in the main setup script |
| 02:45:00 | <joepie91> | :P |
| 02:46:00 | <joepie91> | if you want to do it manually from an existing account, you can copypaste the commands :) |
| 02:46:00 | <joepie91> | basically, webshots_debian.sh == commands as root |
| 02:46:00 | <joepie91> | setup.sh == commands as unprivileged user |
| 03:00:00 | <ivan`> | good news. I have all of paste.lisp.org before all the old pastes got nuked. |
| 03:28:00 | <joepie91> | bsmith095: still there? |
| 03:29:00 | <joepie91> | sent 11.58G bytes received 6.43K bytes 2.59M bytes/sec |
| 03:29:00 | <bsmith095> | yes? |
| 03:29:00 | <joepie91> | http://aarnist.cryto.net:81/youtube/ |
| 03:29:00 | <joepie91> | total size is 11.58G speedup is 1.00 |
| 03:29:00 | <joepie91> | :) |
| 03:29:00 | <bsmith095> | thanks? |
| 03:29:00 | <bsmith095> | thanks! |
| 03:29:00 | <bsmith095> | typo |
| 03:30:00 | <joepie91> | wget/etc as you wish, those should be all the videos for that user |
| 03:30:00 | <bsmith095> | umm, i have completely forgotten what these are |
| 03:30:00 | <joepie91> | <bsmith095>rare film footage grab NOW http://www.youtube.com/user/TheThiefArchive/ |
| 03:30:00 | <joepie91> | :P |
| 03:31:00 | <bsmith095> | oh right, well um, ia might want to store them, but i was just putting it out there, that they existed and hes a rare animaotr, so, yeah... |
| 03:31:00 | <joepie91> | :) |
| 03:31:00 | <joepie91> | SketchCow will probably know what to do with them |
| 03:31:00 | <joepie91> | I'll just leave them up there |
| 03:35:00 | <joepie91> | I wonder at what size photo storage site, archiving can be done quicker by generating random images, than by actually archiving the site |
| 03:37:00 | <joepie91> | okay, time to sleep, night all |
| 04:40:00 | <underscor> | ivan`: awesome! |
| 05:16:00 | <ivan`> | underscor: uploading soon |
| 05:17:00 | <ivan`> | NTFS is slow as hell |
| 05:30:00 | <godane> | i got some sky at night episodes from the 1986 and 1987 |
| 05:30:00 | <godane> | :-D |
| 05:30:00 | <godane> | its a poor vhsrip though |
| 05:32:00 | <godane> | the 21-12-1986 episode is a cdrip from sky at night magazine |
| 05:32:00 | <godane> | its very good video compared with vhsrip ones |
| 05:40:00 | <godane> | i think i found sky news when the WTC attacks happened |
| 05:45:00 | <DFJustin> | oh cool, I was watching their stream that day |
| 06:02:00 | <ivan`> | http://ludios.org/tmp/paste.lisp.org.7z |
| 06:03:00 | <ivan`> | hopefully nothing of value was pasted in the last 4 months |
| 06:04:00 | <godane> | looks like my underground gamer dump with 7gb in total |
| 19:18:00 | <bsmith095> | joepie91: hey, umm, i just checked those vids again, umm, you didnt title them, youtube dl -t |
| 19:18:00 | <bsmith095> | also does wgte --spider give an idea of size? |
| 19:54:00 | <bsmith095> | is my data consistent, im running the scripts on ec2? |
| 19:59:00 | <SmileyG> | :o |
| 19:59:00 | <SmileyG> | how much that cfosting you? |
| 20:02:00 | <bsmith095> | free teir |
| 20:02:00 | <SmileyG> | :O |
| 20:02:00 | <SmileyG> | is it utterly free# |
| 20:02:00 | <SmileyG> | ? |
| 20:02:00 | <bsmith095> | t1 micro so, low cpu, but ican have up to 30gb free ebs space |
| 20:03:00 | <bsmith095> | check amazon ec2 free teir google |
| 20:03:00 | <bsmith095> | if youve never used it before its free for a year, there's limits though |
| 20:04:00 | <bsmith095> | and when its not, free anymore, for a micro its 2 cents an hour and the ebs space is 10 cents a gb per month |
| 20:04:00 | <bsmith095> | although if its a gb stored or allocated im not sure |
| 20:04:00 | <bsmith095> | for ex ample, i have a 50gb ebs store, but its only filled 1gb of it |
| 20:04:00 | <bsmith095> | 10gb * |
| 20:04:00 | <SmileyG> | YEAH |
| 20:05:00 | <SmileyG> | yeah ;D |
| 20:05:00 | <SmileyG> | i just urgh, |
| 20:05:00 | <bsmith095> | did you check? |
| 20:05:00 | <SmileyG> | yeah bits, i see the free stuff |
| 20:05:00 | <bsmith095> | and? |
| 20:05:00 | <SmileyG> | just wondering if its worth while trying to put together a script to build out a ec2 instance.... |
| 20:05:00 | <SmileyG> | do run the scripts etc. |
| 20:06:00 | <bsmith095> | the standard script for webshots works fine if you tweak it the run in your user and not make its own |
| 20:06:00 | <bsmith095> | there a gui on amazon ec2 u know |
| 20:06:00 | <SmileyG> | cool, i've never even looked at it etc |
| 20:06:00 | <bsmith095> | ec2 managemment console |
| 20:06:00 | <SmileyG> | due to costs etc... not something I can afforg to go crazy on |
| 20:06:00 | <bsmith095> | i just started one for this and a ftp grab |
| 20:06:00 | <SmileyG> | You can't just set a $ limit and have it "shutdown" can you ? |
| 20:06:00 | <bsmith095> | **free for a year** |
| 20:07:00 | <SmileyG> | As that'd be perfect for me |
| 20:07:00 | <SmileyG> | bsmith095: unless you screw up an dgo over your limits :P |
| 20:07:00 | <bsmith095> | you can, atually set billing alarms |
| 20:07:00 | <bsmith095> | its the storage you have to worry about, and the iops |
| 20:07:00 | <SmileyG> | yeah |
| 20:07:00 | <SmileyG> | but obviously amazon don't want to make it too easy not to get charged. |
| 20:08:00 | <bsmith095> | 10 million disk io per month, sound like alot cause i have no idea |
| 20:10:00 | <SmileyG> | me neither. |
| 21:46:00 | <chronomex> | disk io is a sector read/write |
| 21:46:00 | <chronomex> | 4k I think |
| 21:47:00 | <chronomex> | I have a database server that's sustaining about 300 ios/second under light-medium load |
| 21:48:00 | <chronomex> | I dropped a couple indexes and it shot up, unfortunately, I'm going to have to go sql spelunking to undo that |
| 21:48:00 | <SmileyG> | lol |
| 21:48:00 | <SmileyG> | i need to sit down and grab all the info I have on tuning the warrior :< |
| 21:48:00 | <SmileyG> | except my brain has shutdown. |
| 21:49:00 | <chronomex> | it used to be at 25-50 ios/sec, and an idle machine is a lot less |
| 22:07:00 | <S[h]O[r]T> | is there a channel for webshots? |
| 22:07:00 | <S[h]O[r]T> | been idling havent paid attention in here since coh, must have missed some tweets :P |
| 22:22:00 | <SmileyG> | #webshots :D |
| 23:16:00 | <bsmith095> | SmileyG: so, i checked, and ec2 counts storage you are actually using, rather that the total you have allocated for yourself |
| 23:17:00 | <SmileyG> | thats cool. |
| 23:18:00 | <bsmith095> | $0.11 per 1 million I/O requests |
| 23:19:00 | <bsmith095> | is that a lot, i have no idea, could somebody clear that up? |
| 23:19:00 | <bsmith095> | thats off free teir, cause apaprently i wnet over so they dinged me a nickel |