00:01:29<EggplantN_>hey urm
00:01:31<EggplantN_>Orldow6
00:01:34<EggplantN_>did we annoy someone
00:01:35<EggplantN_>http://www.smackjeeves.com/comicprofile.php?id=177533http://www.smackjeeves.com/comicprofile.php?id=177533http://www.smackjeeves.com/comicprofile.php?id=177533http://www.smackjeeves.com/comicprofile.php?id=177533%5Cr%5CnFUCKING%20TY%20THAT%20DICKHEAD%20%E2%80%98OH%20HEY%20DONT%20GO%20TO%20THAT%20SITE%20BECAUSE%20IM%20A%20FUCKING%20IGNORANT%20BRAT%20WHO%20DOESNT%20CARE%20ABOUT%20LAG!%E2%80%99%5Cr%5CnFUCK%20YOU!%20WHY%20DO%20YOU%20THINK%20EV
00:01:35<EggplantN_>ERYBODY%20LEFT!%20THEY%20DIDNT%20WANT%20TO%20RP%20IN%20THESE%20CONDITIONS!
00:02:12<EggplantN_>confused how this came
00:02:57<OrIdow6>What
00:02:59<OrIdow6>Strange
00:03:07<EggplantN_>Yeah I'm confused where thats come from
00:03:16<OrIdow6>What item is this on?
00:03:22<EggplantN_>177533
00:03:37<EggplantN_>I linked the main error above
00:03:45<EggplantN_>That covers most remaining items
00:03:48<OrIdow6>Yeah, think I have a fix for that first one
00:04:05<OrIdow6>Decoded text of this strange problem, by for those at home, is "FUCKING TY THAT DICKHEAD ‘OH HEY DONT GO TO THAT SITE BECAUSE IM A FUCKING IGNORANT BRAT WHO DOESNT CARE ABOUT LAG!’\r\nFUCK YOU! WHY DO YOU THINK EVERYBODY LEFT! THEY DIDNT WANT TO RP IN THESE CONDITIONS!"
00:04:35<EggplantN_>love to know where that come from lol
00:06:09<@arkiver>probably wrongly extracted by Lua
00:06:22<OrIdow6>That'd be my guess
00:06:25<@arkiver>OrIdow6: you could just put an ignore in allowed for \r and \n
00:06:39<@arkiver>if one of those is in the URL
00:06:46<EggplantN_>its only a single item afaik
00:06:51<@arkiver>these things sometimes happen with aggressive URL extraction
00:06:59<EggplantN_>heck we only have 65 left
00:07:11<EggplantN_>then this one can be closed up
00:07:25hooway quits [Client Quit]
00:08:09<OrIdow6>Ok, version 20201230.01 (https://github.com/ArchiveTeam/smackjeeves-grab/commit/b1be942da6b74784216db7c4af7cc19fa315e948) is up
00:09:19<EggplantN_>lemme set min version and requeue
00:11:10<EggplantN_>ffs
00:11:18<EggplantN_>kiska added targets to LB without SJ setup
00:11:24<EggplantN_>kk
00:13:58<EggplantN_>okay Orldow6 that takes us to 2
00:14:09<EggplantN_>lemme checkout those 2 and close this shit up
00:14:11<EggplantN_>cheers!
00:16:45<OrIdow6>Thank you
00:19:12<OrIdow6>Stdill need to do the users for this one, almost forgot about that
00:20:36<OrIdow6>That strange string is certainly topical for 177533, but I can't find it anywhere, nor is that URL extracted when I try to run it locally
00:21:19<OrIdow6>Hopefully it won't be among the 2
00:22:05<EggplantN_>the final 2 are running now
00:22:20<EggplantN_>(ignore the last item returned by tracker, it was a forced return as I already have done it)
00:22:28<EggplantN_>Any idea how many users there will be?
00:24:53<EggplantN_>anyway that items gone through now.
00:25:17<EggplantN_>Last 2 are 176093 & 79290 which are running
00:26:12<OrIdow6>Maybe about the same, up to 350k or so
00:26:24<OrIdow6>Max I have in a "scan" is 319905
00:26:30<OrIdow6>But these should be much smaller
00:26:47<OrIdow6>& are not really as important to save
00:27:33<OrIdow6>(So no huge loss there if the site goes down before we get all the user profile pages)
00:28:01<@arkiver>JAA: I'm looking at the project sonar data, looks like only the most recent datasets are downloadable. what do you think of archiving datasets into the Wayback Machine?
00:28:05<@arkiver>also getting every new dataset of course
00:28:38<OrIdow6>Anyhow I'll start working on user pages, shouldn't be too complicated (famous last words)
00:29:11benjinsmith joins
00:30:46<EggplantN_>Actually no sorry Orldow6 I mistook comic:176093 is that item that gives the dickhead shit
00:31:24benjins quits [Ping timeout: 240 seconds]
00:31:35<EggplantN_>any quick ignore you could give for me if you dont mind
00:31:37<@JAA>arkiver: Yeah, only one of the most recent datasets is publicly available. You have to request an account and give them some reasonable explanation to get free access to the entire collection.
00:31:45benjinsmith is now known as benjins
00:34:50<OrIdow6>EggplantN_: Try 20201230.02
00:34:53<@arkiver>JAA: you think this is something for archivebot? or rather something separate?
00:36:38<OrIdow6>Did not test this, because there are >200 pages in that comic
00:36:51<EggplantN_>i'll test dw
00:36:57<EggplantN_>https://www.irccloud.com/pastebin/xdN1Vqfn/error
00:37:01<EggplantN_>also got this issue?
00:38:42tzt joins
00:42:34<OrIdow6>Of course they return a non-200 status code... for this one comic
00:43:23<EggplantN_>the other ones done
00:43:23<EggplantN_>lol
00:43:46benjinsmith joins
00:43:53<EggplantN_>well uploading
00:44:01<EggplantN_>lets see if tracker marks it as done done
00:44:04<OrIdow6>So it's just this one item with the 500 left?
00:44:13<EggplantN_>yep
00:44:15<EggplantN_>lol
00:44:31<EggplantN_>if you wanna do it yourself locally or give me the script and i'll finish her off
00:45:54benjinss joins
00:46:20benjins quits [Ping timeout: 265 seconds]
00:47:19benjinss is now known as benjins
00:48:24benjinsmith quits [Ping timeout: 240 seconds]
00:49:05<OrIdow6>20201230.03
00:49:22<OrIdow6>EggplantN_
00:50:18<EggplantN_>running ty Orldow6 just before I debate my evening will you be adding user support tonight or tomorrow
00:50:26<EggplantN_>(is like 1am here )
00:50:37<OrIdow6>I'll try to work on it now
00:50:46<OrIdow6>Hopefully should be under an hour
00:50:58<EggplantN_>got me an item list to feed the tracker?
00:51:12<OrIdow6>No
00:51:22<OrIdow6>It will just be user:[integers 1-350k]
00:51:32<EggplantN_>i'll generate one now and get it loaded
00:51:35<OrIdow6>Ok
00:52:39<OrIdow6>Thanks
00:54:54<EggplantN_>no problemo
01:16:21<OrIdow6>OK, version 20201230.04 should support user: items now
01:16:51<OrIdow6>Vast majority will be under 5 requests each
01:17:11<EggplantN_>okay lemme update tracker
01:18:55<EggplantN_>okay kiska has fucked their target
01:18:57<EggplantN_>lemme go get one
01:25:24<EggplantN_>Matthww update and lets continue
01:26:43<EggplantN_>All looks good Orldow6 thank you!
01:29:16<OrIdow6>Ok, thank you also
01:29:21<OrIdow6>:)
01:29:52<EggplantN_>Anytime, short time want to get this done and moved onto Endomondo :P
01:30:57<OrIdow6>This one is only making 2 non-S3 requests per item (or it should be, if nothing goes wrong), so you should be able to be a bit more liberal with the rate limit
01:31:17<EggplantN_>i've set it at 400
01:31:24<EggplantN_>my box is getting a bit toasty
01:31:24<OrIdow6>Yeah, will be nice to get it done
01:31:26<OrIdow6>Ok
01:32:14<EggplantN_>5.7 load avg
01:32:16<EggplantN_>we can go harder
01:34:10<EggplantN_>no other items except users left Orldow6 no? If so I'm gonna scale and go to bed
01:34:47<OrIdow6>Nope - once these are finished, SmackJeeves wil lbe done
01:34:50<OrIdow6>*will be
01:36:36<OrIdow6>Goodnight
01:51:53britmob joins
01:54:09<fuzzy8021>getting this for smackjeeves - Tracker returned status code 403. The tracker has probably malfunctioned. Retrying after 160 seconds...
01:54:24<fuzzy8021>that due to only allowing egg or tracker issues
01:54:43<fuzzy8021>allowing egg to pull items*
02:00:01Dallas quits [Client Quit]
02:04:00Dallas (Dallas) joins
02:10:10<OrIdow6>Maybe you're below the min version? I can only guess
02:21:59<atphoenix>is there a new docker image for smackjeeves? How do I force my image to update?
02:26:53<fuzzy8021>if its a version issue it should tell you
02:27:31<fuzzy8021>docker pull atdr.meo.ws/archiveteam/smackjeeves-grab
02:30:54<@JAA>fuzzy8021: Yeah, you're blocked on the smackjeeves tracker. I don't know why and can't find anything in my logs, so I won't touch it.
02:31:32<fuzzy8021>hmm k thanks
02:36:41<fuzzy8021>suggestion for tracker update, could it sned a message that you are banned and tell the user to contact staff?
02:44:58<atphoenix>thanks, I was able to update my docker image. Created a new container using that image with concurrent 5. Now I get a lovely 'tracker rate limiting is active'. Is EggplantN_ saturating it?
02:58:23<OrIdow6>So it looks like
03:07:40<fuzzy8021>docker container run -d --restart=unless-stopped --network=host --name=watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower -i 30 --cleanup
03:07:54<fuzzy8021>atphoenix run that and docker will autoupdate
03:11:29<atphoenix>I had used a similar command, per the AT wiki: docker run -d --name watchtower --restart=unless-stopped -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --label-enable
03:12:12<fuzzy8021>then you shouldnt need to pull new image nor restart your container
03:12:19<fuzzy8021>it will take care of it
03:15:16<atphoenix>idk, maybe I need to start over as it didn't autoupdate the SJ image
03:29:07Fusl_ (Fusl) joins
03:29:08@ChanServ sets mode: +o Fusl_
03:30:22<purplebot>YouTube edited by Nintendofan885 (+305, +inactive accounts policy) just now -- https://www.archiveteam.org/?diff=46008&oldid=44293
03:59:51lukash7 quits [Client Quit]
04:03:12lukash7 joins
04:04:53lukash7 quits [Client Quit]
04:06:34lukash7 joins
04:07:43lukash7 quits [Client Quit]
04:08:39lukash7 joins
04:16:04HackMii quits [Ping timeout: 240 seconds]
04:23:52HackMii (hacktheplanet) joins
04:27:22Quirk joins
04:27:44Quirk is now known as Ruthalas
04:38:20qw3rty__ joins
04:41:44qw3rty_ quits [Ping timeout: 240 seconds]
04:48:47pew joins
05:28:22Ruthalas quits [Remote host closed the connection]
05:39:09VADemon quits [Read error: Connection reset by peer]
05:39:26Ruthalas joins
05:54:41@dxrt quits [Quit: ZNC - http://znc.sourceforge.net]
05:54:59dxrt joins
05:55:04dxrt quits [Changing host]
05:55:04dxrt (dxrt) joins
05:55:04@ChanServ sets mode: +o dxrt
06:20:22<purplebot>Running Archive Team Projects with Docker edited by Atphoenix (+596, /* Instructions for using Docker …) just now -- https://www.archiveteam.org/?diff=46009&oldid=45974
06:22:22<purplebot>Archiveteam:IRC edited by Atphoenix (+4, added internal link) just now -- https://www.archiveteam.org/?diff=46010&oldid=45738
06:47:39<mgrandi>I can't tell if this was added as irccloud doesn't have search on mobile , but https://twitter.com/LukeLetlow passed away today, house rep elect
06:53:22<purplebot>ArchiveTeam Warrior edited by Atphoenix (+392, updated Warrior message to include …) just now -- https://www.archiveteam.org/?diff=46011&oldid=45976
06:53:58<OrIdow6>Looks like it was run last month, might want to do it again
07:12:24@dxrt quits [Client Quit]
07:14:58dxrt joins
07:15:01dxrt quits [Changing host]
07:15:01dxrt (dxrt) joins
07:15:01@ChanServ sets mode: +o dxrt
07:18:13Stiletto joins
07:43:14rsn joins
07:45:23rsn_ quits [Ping timeout: 265 seconds]
07:49:51@dxrt quits [Client Quit]
07:50:10dxrt joins
07:50:12dxrt quits [Changing host]
07:50:12dxrt (dxrt) joins
07:50:12@ChanServ sets mode: +o dxrt
09:00:20britmob2 quits [Quit: britmob2]
09:02:05<@kiska>OrIdow6: SmackJeeves is complete?
09:02:17<@kiska>Also where did you find another 30ish k?
09:02:48<@kiska>btw I am withdrawing lb1 from SJ
09:19:30ragu__ joins
09:20:46<OrIdow6>kiska: Comics are complete; currently running are users
09:21:10<OrIdow6>There were 350k user: items, now down to 22k because they are so small
09:21:51<OrIdow6>After this (and maybe I'll get some redirects into the WBM through #//, undecided on that) it will all be complete
09:23:30ragu_ quits [Ping timeout: 265 seconds]
09:25:13<@kiska>ok
09:33:16<EggplantN_>kiska I removed your LB
09:33:43<EggplantN_>As there was targets in it that didn't have SJ setup
09:39:11<@kiska>I know :D
09:39:38<@kiska>I am rescaling them so don't turn it on
09:40:34<EggplantN_>I've got my own target on it
09:40:39<EggplantN_>Clean down SJ
09:40:46ragu_ joins
09:44:17ragu__ quits [Ping timeout: 265 seconds]
09:44:55hooway joins
09:45:09ragu__ joins
09:46:44ragu_ quits [Ping timeout: 240 seconds]
09:47:37ragu_ joins
09:50:34ragu__ quits [Ping timeout: 265 seconds]
09:51:58<OrIdow6>Well, looks like SJ is down to 0 to do and 63 out (AKA broken)
09:52:53ragu_ quits [Remote host closed the connection]
09:52:53<OrIdow6>Oh, that's 0 and 1, respectively, now
09:53:26ragu_ joins
09:53:30<EggplantN_>Yeah quick requeue
09:53:35<EggplantN_>It's that comic from earlier
09:53:43<EggplantN_>I'll check it in a hour or 2
09:55:03<OrIdow6>Ugh
09:55:06<OrIdow6>Ok
09:55:18<OrIdow6>Oh, it's still running, I see
09:55:52ragu_ quits [Remote host closed the connection]
09:56:30ragu_ joins
09:57:53ragu_ quits [Remote host closed the connection]
09:58:32ragu_ joins
10:05:36ragu_ quits [Remote host closed the connection]
10:06:01ragu_ joins
10:07:44ragu__ joins
10:08:06ragu_ quits [Remote host closed the connection]
10:29:44xit quits [Client Quit]
10:30:02xit joins
10:54:56<EggplantN_>Orldow6 done
10:55:13<EggplantN_>that comic had finished but because I requeued it went through again
10:55:18<EggplantN_>so i've manually marked it as done
10:57:27<EggplantN_>Final pack is in queue
10:57:42rsn_ joins
10:59:12rsn quits [Ping timeout: 265 seconds]
11:06:27<EggplantN_>SJ removed from tracker
11:09:13spirit joins
11:28:20<OrIdow6>Woo
11:28:35<OrIdow6>Thank you for running everything
11:28:44<OrIdow6>Well
11:28:50<OrIdow6>A portion of everything
12:06:11<EggplantN_>Hey no, thank you!
13:32:54Barto quits [Ping timeout: 265 seconds]
13:33:48Barto (Barto) joins
13:34:47LeGoupil joins
13:50:32britmob2 joins
14:12:08vicarage joins
14:12:34vicarage quits [Client Quit]
14:34:04Arcorann quits [Ping timeout: 240 seconds]
14:54:16etnguyen03 (etnguyen03) joins
16:07:51spirit quits [Client Quit]
16:12:28spirit joins
16:43:29seatsea quits [Read error: Connection reset by peer]
16:46:19seatsea joins
17:26:04<themadpro>So uh.... seeing as Miniclip and several gamesites are taking down Flash games (within the next ~48 hours) I'm surprised we haven't yet initiated a #saveflash project?
17:26:12<themadpro>Granted Flashpoint has come a long way but still
17:32:05seatsea quits [Read error: Connection reset by peer]
17:32:53seatsea joins
17:35:44Matthww quits [Ping timeout: 240 seconds]
17:38:20<@arkiver>maybe JAA knows about archiving activities regarding ^
17:40:40<@JAA>Some sites have been grabbed with AB, but lots haven't. No idea regarding Miniclip specifically.
17:41:23<@JAA>Looks like someone mentioned it here about 2 weeks ago, but it doesn't seem like anything happened after that.
17:41:50Matthww joins
18:25:34spriteclad joins
18:44:33spriteclad quits [Client Quit]
18:59:12<mgrandi>@themadpro: are there links to all of the sites that are doing that?
19:00:26<themadpro>https://cdn.discordapp.com/attachments/675397790204952636/793819660797214770/image.png https://usercontent.irccloud-cdn.com/file/lt6yHo7y/situation.png
19:00:35<themadpro>Not readily available, not yet anyway
19:00:57<themadpro>But, hypothetically, let's say if I was able to gather a list of ~50 websites which have announced that they have such plans
19:01:03<themadpro>could we #archivebot them?
19:01:32<mgrandi>Possibly, I was just gonna see if I can get the SWFs themselves as an emergency thing
19:02:01<mgrandi>And then archivebot can try and get them if it has time?
19:02:44<mgrandi>Also unrelated, luke letlow's twitter account is now private, hopefully the AB job finished
19:06:12<mgrandi>@themadpro: see if you can get a list together and we can start a last minute thing
19:06:22<themadpro>Will do
19:06:39<mgrandi>If it's a big enough site I would assume other things like flashpoint would have gotten them but let's be safe
19:07:56<@JAA>We should probably start a wiki page on this as well. There are a few sites on Deathwatch, but that might blow up if we add them all.
19:08:25<@JAA>Or well, start a list on https://archiveteam.org/index.php?title=Flash I suppose.
19:08:39<@JAA>Or a subpage for that.
19:19:11<mgrandi>Yeah, place them there
19:19:56<wessel1512>is ther a cannel for the Domains project
19:21:52<jodizzle>wessel1512: #noteurdomain
19:23:52<jodizzle>We did have someone here a little while back who claimed to have a big list of sites that Flashpoint hadn't grabbed much, but it looks like they've disappeared
19:35:31<mgrandi>What was their name?
19:57:17<atphoenix>12/22 spirit, HP_Archivist conversation? 12/17 grawity conversation? 11/23 Zopolis4 conversation about http://spikedmath.com/math-games/ which said flashpoint may not cover. and said '<Zopolis4> i posted a big list a while back which is slowly being worked through'
19:59:09<atphoenix>that's only from looking through -bs . There might be more in the AB logs.
20:08:27<themadpro>This is by no means complete but https://docs.google.com/spreadsheets/d/1bHr_7VO6Qr90NTTc0sednnGEOYfyE2qN2s_fzYnI08I/edit#gid=0
20:08:51<themadpro>Flashpoint seems to have a watchlist for the "BIG" platforms, including those which have announced plans for removal
20:08:59<themadpro>Probably the best place to start from
20:13:17<mgrandi>Should we make a channel for the coordination
20:25:04etnguyen03 quits [Ping timeout: 240 seconds]
20:25:52etnguyen03 (etnguyen03) joins
20:29:38<jodizzle>mgrandi: I was thinking of Zopolis4
20:36:48<wessel1512>themadpro i added some Dutch game portal to the list
20:36:56<themadpro>Awesome!
20:44:15<mgrandi>Should we make a channel to take this off of -bs? Regardless, I'll work on these tonight
21:04:36<wessel1512>what is the plan for saving the flash games site
21:06:40<wessel1512>just throw them in archive bot ore something different
21:10:28<atphoenix>channel ideas: flashgoesoff, theflashgoesoff, flashbang
21:11:12<@JAA>flush
21:11:20<EggplantN_>flashbank
21:11:21<wessel1512>i like flashbang as its the end of the year
21:11:25<EggplantN_>*flashbang
21:11:26<EggplantN_>deffo
21:16:24<atphoenix>flushed, flashphotofinish, unflashy, flashgoestitsup
21:25:32pew leaves [WeeChat 2.3]
21:36:09LeGoupil quits [Client Quit]
22:00:04mutantmonkey quits [Ping timeout: 240 seconds]
22:02:06mutantmonkey (mutantmonkey) joins
23:36:30Arcorann (Arcorann) joins
23:44:54<OrIdow6>Here was Zoopolis4's list: https://transfer.notkiska.pw/hHpTQ/lenks%20-%20Copy.txt
23:45:18<OrIdow6>Obviously needs some filtering (e.g. the site with its WBM copy given)
23:56:58hooway quits [Client Quit]