HomeSearchPrevious dayNext day

00:05:08<Lord_Nightmare>should i run snscrape on his account again?
00:05:16<Lord_Nightmare>there's a lot of new material since it was last run
00:06:39<@JAA>Since yesterday?
00:10:22OrIdow6 (OrIdow6) joins
04:01:49qw3rty__ joins
04:05:16qw3rty_ quits [Ping timeout: 252 seconds]
04:11:58<nico_32>not now
04:12:03<nico_32>*much
04:12:12<nico_32>less than 10 tweets
04:37:46<SketchTheCow>Something is breaking on FOS.
04:37:47<SketchTheCow>Not sure what.
04:38:36<jodizzle>nico_32: You can use `twitter-search` and `since:` instead of `twitter-user` to only get the recent tweets
04:41:36Arcorann (Arcorann) joins
04:42:24Arcorann_ quits [Ping timeout: 240 seconds]
05:00:11qw3rty_ joins
05:03:44qw3rty__ quits [Ping timeout: 240 seconds]
05:18:21lukash quits [Client Quit]
05:28:34lukash (lukash) joins
06:12:53lukash quits [Ping timeout: 252 seconds]
06:46:23DopefishJustin quits [Read error: Connection reset by peer]
06:47:43DopefishJustin joins
06:51:57<SketchTheCow>OK, so
06:52:05<SketchTheCow>I moved things around, but the chances are FOS will fill.
06:52:20<SketchTheCow>Something is making the disk absolutely chug. If I have to, I'll turn off rsync to see if it helps tomorrow.
07:07:51lukash (lukash) joins
08:08:07<SketchTheCow>I had to stop rsync
08:08:17<SketchTheCow>I am trying to figure out why one of the two drives keeps filling
08:35:10HP_Archiv joins
08:37:02<Ryz>That might've explained why pipeline series Igloo-EU-SYS-01 is having uploading problems right now~
08:37:44HP_Archivist quits [Ping timeout: 240 seconds]
08:39:24ftl quits [Ping timeout: 600 seconds]
08:41:04@jrwr quits [Ping timeout: 600 seconds]
08:42:04Dragnog quits [Ping timeout: 600 seconds]
08:43:04@Fusl_ quits [Ping timeout: 600 seconds]
08:43:24@hook54321 quits [Ping timeout: 600 seconds]
08:44:44@Kaz__ quits [Ping timeout: 600 seconds]
08:51:19tech234a quits [Read error: Connection timed out]
08:52:46Dragnog joins
08:52:56Fusl_ (Fusl) joins
08:52:57@ChanServ sets mode: +o Fusl_
08:53:03tech234a (tech234a) joins
08:53:13Kaz__ joins
08:54:12ftl joins
08:55:35jrwr (jrwr) joins
08:55:35@ChanServ sets mode: +o jrwr
08:59:28hook54321 (hook54321) joins
08:59:28@ChanServ sets mode: +o hook54321
09:26:37ftl quits [Read error: Connection timed out]
09:30:27ftl joins
10:27:34godane2 joins
10:30:16godane1 quits [Ping timeout: 252 seconds]
11:05:44godane1 joins
11:05:44godane2 quits [Read error: Connection reset by peer]
11:20:39godane2 joins
11:20:50godane1 quits [Read error: Connection reset by peer]
12:36:12Arcorann quits [Read error: Connection reset by peer]
12:37:05Arcorann (Arcorann) joins
12:40:18<purplebot>Discourse edited by Themadprogramer (+56, /* Notable Discourses */) just now -- https://www.archiveteam.org/?diff=45751&oldid=45665
13:15:53cadence quits [Quit: Leaving]
13:16:12cadence (cadence) joins
13:19:19<purplebot>News+C/hu edited by Bzc6p (+252, updates) just now -- https://www.archiveteam.org/?diff=45753&oldid=45447
13:29:12benjins quits [Ping timeout: 252 seconds]
13:35:18<purplebot>TVN.hu edited by Bzc6p (+16, finished for now) just now -- https://www.archiveteam.org/?diff=45754&oldid=45529
13:38:40<@JAA>Oof
13:38:52benjins joins
13:48:24luckcolors quits [Ping timeout: 240 seconds]
13:49:10<SketchTheCow>Something is very wrong.
13:49:13<SketchTheCow>I am not sure what it is.
14:08:24<SketchTheCow>I found the problem.
14:08:32<SketchTheCow>It will take a little longer to clean up and then it'll be fine.
14:09:39<@JAA>Yay
14:11:59<SketchTheCow>I have a script that makes a table.txt and an html output for a pipeline report.
14:12:13<SketchTheCow>You folks have been such full bore, it surpassed 1.3tb. A textfile.
14:12:23<@JAA>lol
14:12:31<SketchTheCow>And over time, it was fine, grinding away but it must have crossed a threshold and everything started blowing up.
14:20:53<SketchTheCow>Yeah, there it goes
14:21:05<SketchTheCow>As /1 goes from hellscape to 35% util.
14:27:13<SketchTheCow>I turned on rsync so other people don't die.
14:27:28<SketchTheCow>In 2 hours or less, it will finish uploading my crappy reocities thing that doesn't work.
14:30:44Arcorann quits [Ping timeout: 240 seconds]
14:30:50<SketchTheCow>I've set off a second one. Additional fun is s3 is overloaded.
15:18:07HP_Archiv quits [Client Quit]
15:18:25HP_Archivist (HP_Archivist) joins
16:05:02kiwec joins
16:07:41<kiwec>hey, there was a recent discovery that you could download deleted twitch clips and vods by accessing the direct cdn link, which reddit often cached. I ran a script on pushift and now have 676407 of those. I think it would be nice to get back DMCA'd clips, or clips of banned streamers (like drdisrespect, iceposeidon...). anybody interested?
16:13:17<SketchTheCow>I mean, I'm interested.
16:13:30<SketchTheCow>How big is your stuff
16:13:39<SketchTheCow>Did you download it or did you just get the list
16:14:18<kiwec>I just got the list. at approx 30M per clip that would be 18Tb, and the list doesn't include any vods, and I should double check if it's actually complete
16:15:46<kiwec>also, the list includes non-deleted clips, I don't know if there is an easy way to check if it's deleted or not (without spamming the twitch api)
16:23:37<benjins>For Twitch clips, it looks like you'd make a request to gql.twitch.tv/gql (which is what the browser does for most lazily-loaded things), and if you get null back the clip is gone
16:23:51<benjins>This is going off what's in Network Inspector
16:25:36<SketchTheCow><whispers> spam the twitch api </whispers>
16:45:12<kiwec>I'll dump what I currently have here for reference. script: https://hastebin.com/iwukagigaz.py clip list: https://litter.catbox.moe/o9jzz3.gz clips by sub: https://hastebin.com/axoruqovuq.apache
16:54:30<benjins>I can try to work through the clips, depending on how fast it can go. Here's a PoC shell script I made to check if a script is up or not: https://gist.github.com/Benjins/a5f79585771d577df505731c1a09fa2d for anyone else's reference
17:07:03<benjins>kiwec: Are the clips-media-assets2.twitch.tv links supposed to still be active? I see thumbnails, but I can't hit any mp4 files
17:07:29<kiwec>if you replace "-preview.jpg" by ".mp4" it works on most of the links
17:07:35odemgi_ quits [Remote host closed the connection]
17:07:51odemgi_ joins
17:08:49<benjins>I'm getting access denied on the ones I'm trying
17:09:09<benjins>Even clips that are still up
17:09:44<benjins>Oh wait, are you supposed to also remove the -social?
17:10:04<kiwec>oh right, I think the clips need to be old enough for this to work
17:10:31<benjins>"-social.mp4" doesn't seem to work, but just ".mp4" does, on the ones I tried
17:12:11Laverne quits [Quit: ZNC 1.7.4+deb0+bionic0 - https://znc.in]
17:12:22Laverne joins
17:17:53<kiwec>thanks, that gets rid of some of the 403s I had. On a /r/Ice_Poseidon test run this gives me 6237 clips (out of 7113 urls), totalling 61Gb. some duplicates, some 403s, hard to tell if the link are wrong (like -social) or if twitch just purged them
17:31:18<benjins>Running the check on which clips are still up, by my estimates it'll take ~30 hours
18:56:12yanome7 (yano) joins
18:56:50Aoede quits [Ping timeout: 252 seconds]
18:57:20ivan_ joins
18:57:56ivan quits [Ping timeout: 252 seconds]
18:58:16ivan_ is now known as ivan
18:58:29yanome quits [Ping timeout: 252 seconds]
18:58:30yanome7 is now known as yanome
18:58:49EggplantN quits [Read error: Connection reset by peer]
18:59:03Aoede (Aoede) joins
18:59:09EggplantN joins
19:01:14Jonimus quits [Ping timeout: 252 seconds]
19:09:04Jonimus joins
19:31:50Ajay quits [Ping timeout: 252 seconds]
19:33:54Ajay joins
21:09:38<@arkiver>18 TB is not too bad
21:20:32VADemon joins
21:47:21kiwec quits [Remote host closed the connection]
21:57:51<odemgi_>Datechnom, no nerve other than the one that loved the good old days where the playing field was level and people otherwise contributing didn't also pour cash all over getting to the top (arkiver has it right, he writes the code and doesn't run a worker pool). I think the structure of the board should change if this is how we're going to do things today, I contribute under various names to keep it _looking_ interesting and fair, nobody want's to
21:57:55<odemgi_>fight for 4th,5th,6th place and so on. So the original spirit of the lb is dead and now it's just who wants to waste most cash because again it doesn't change the outcome now a few people could just do this on their own and everyone else go home because that's what we're left with right now. Should AT really be the 'you can't sit with us' group? (posting this waste of time in bs because it is)
22:04:26<@Fusl_>odemgi_: 1. it's a leaderboard for a reason, 2. if people dont have money to throw at it they can also use their free resources like soon the new warrior will allow them, 3. i personally dont throw any money at workers, i use leftover credits from hetzner to run mine and i usualky only run my workers short term to help kicking a project off or making sure we dont overshoot the deadline
22:07:23<odemgi_>I have nothing good to say.
22:11:17<@Fusl_>what i can do is allow users to hide themselves from leaderboards
22:12:27<@Fusl_>but that would be opt-in and project and site admins would still see them
22:14:30<odemgi_>read most of what I say with the idea that I fucking love complaining in mind, my main point here is that the leaderboard was more fun when we had no idea who would be at the top of it, that's all.
22:15:29<nico_32>when?
22:15:56<nico_32>because when we were doing the xanga grab
22:16:15<nico_32>I throw some serious money on Amazon EC2 & Azure
22:16:54<nico_32>and it was in 2013
22:21:23<nico_32>correction it was Hyves
22:21:26<nico_32>same period
22:24:00<@JAA>Even 2-3 years ago... I was at the top of the leaderboard of a couple smaller projects with just a couple servers.
22:24:41<@JAA>But ultimately, I don't really care who archives something or how much resources they throw at it, just *that* it gets archived.
22:26:29<@Fusl_>^
22:27:11Arcorann (Arcorann) joins
22:35:14<Datechnom>^ Agreed. AT is about volunteering your time or money (or both) and working towards the same goal of archiving something so that it is kept and preserved
22:35:44<Datechnom>Its great to see people on reddit and jumping on IRC to thank AT for their efforts and being able to download data from a project that was worked on as they would have lost all of it
22:36:17<Datechnom>At that point no one is caring what the leaderboard was/is for that project just that its on Archive.org's systems
22:43:33lukash quits [Client Quit]
22:43:48lukash (lukash) joins
22:45:40<@Fusl>but as said, the user system later on could allow you to hide your name from the leaderboard
22:53:24<nico_32>just name yourself YOURNICKHERE
22:54:09<@JAA>Been there, done that. lol
22:54:12<@Fusl>yeah
22:54:41<@Fusl>https://tracker.archiveteam.org/tumblr/#show-all
22:54:46<@Fusl>30 items, 11.87gb
22:55:02<@Fusl>https://tracker.archiveteam.org/googleplus/#show-all
22:55:08<@Fusl>1.61k items, 65.1gb
23:08:55<SketchTheCow>The good (?) news is that we are in the same stream as internet archive television grabs.
23:09:13<SketchTheCow>That means that we will have a second group who notice when s3 barfs, like it did
23:14:33<@arkiver>SketchTheCow: seems better now
23:14:43<@arkiver>it slows down every now and then
23:15:00<@arkiver>often resolved pretty fast
23:16:07<@JAA>Well, it was having issues for ~10 hours today, but yeah.
23:16:32<@arkiver>that's not unusual
23:16:40<@arkiver>well
23:16:45<@arkiver>it's not very rare for that to happen
23:16:46<@JAA>Been a while since I've seen it, apart from power cuts etc.
23:17:00<nico_32>let metadata cache that was unavailable?
23:17:10<nico_32>s/let/the/
23:17:12<@arkiver>I noticed it quite often
23:17:25<@arkiver>affects my a lot, but it's usually better in a day or two or so
23:17:28<@arkiver>rare cases take longer
23:24:03DLoader_ joins
23:26:30DLoader quits [Ping timeout: 252 seconds]
23:26:33DLoader_ is now known as DLoader
23:27:16<SketchTheCow>s3 absolutely barfed
23:27:20<SketchTheCow>Fixed 1pm EST

HomeSearchPrevious dayNext day