| 00:01:39 | | bithippo quits [Client Quit] |
| 00:10:52 | | lunik1 quits [Read error: Connection reset by peer] |
| 00:11:08 | | lunik1 joins |
| 00:28:25 | <@arkiver> | the URLs project is online on github at https://github.com/ArchiveTeam/urls-grab |
| 00:28:36 | <@arkiver> | with this we'll archive random discovered outlinks from various sources |
| 00:28:46 | <@arkiver> | it also uses the newest Wget-AT |
| 00:29:11 | <@arkiver> | tracker is at https://tracker.archiveteam.org/urls/ |
| 00:38:06 | <@arkiver> | kiska: we could use a target if you have something available :) |
| 00:38:16 | <@kiska> | Just woke up |
| 00:38:19 | <@kiska> | Hrm? |
| 00:38:33 | <@arkiver> | for the URLs project |
| 00:38:46 | <@arkiver> | it needs the newest megaWARC factory though |
| 00:39:31 | <@kiska> | rsync://rsync.hel1.kiska.pw/urls/:downloader/ |
| 00:39:49 | <@arkiver> | wooh |
| 00:40:07 | <@kiska> | rsync://rsync.hel1.kiska.pw/earls/:downloader/ |
| 00:40:11 | <@kiska> | :D |
| 00:40:18 | <@arkiver> | :P |
| 00:40:20 | <@arkiver> | which do you want |
| 00:40:21 | <@arkiver> | haha |
| 00:40:35 | <@arkiver> | also we can push to |
| 00:40:38 | <@arkiver> | archiveteam_urls_ |
| 00:40:44 | <@arkiver> | Archive Team URLs: |
| 00:41:08 | <@kiska> | Either I am making fun of how you pronounce URLs :D |
| 00:42:05 | <@JAA> | Make sure it's the latest factory. |
| 00:42:34 | <@kiska> | archiveteam@rsync ~/archiveteam-megawarc-factory $ git pull |
| 00:42:34 | <@kiska> | Already up to date. |
| 00:42:35 | <@kiska> | archiveteam@rsync ~/archiveteam-megawarc-factory $ cd megawarc/ |
| 00:42:35 | <@kiska> | archiveteam@rsync ~/archiveteam-megawarc-factory/megawarc $ git pull |
| 00:42:35 | <@kiska> | Already up to date. |
| 00:42:35 | <@kiska> | :D |
| 00:42:50 | <@kiska> | I did that 12h ago |
| 00:42:58 | <@JAA> | :-) |
| 00:42:59 | <@kiska> | And then went to sleep without creating the rsync endpoint |
| 00:43:05 | <@kiska> | Ooops? |
| 00:43:19 | <@kiska> | IA_ITEM_TITLE="Archive Team URLs:" |
| 00:43:19 | <@kiska> | IA_ITEM_PREFIX="archiveteam_urls_" |
| 00:43:19 | <@kiska> | FILE_PREFIX="urls_" |
| 00:43:20 | <@kiska> | Yes? |
| 00:45:40 | <@arkiver> | kiska: yes! |
| 00:45:52 | <jodizzle> | Is there a separate channel? |
| 00:46:31 | <@kiska> | Now I am going to catch up on some messages |
| 00:49:19 | <@arkiver> | project started! |
| 00:49:24 | <@JAA> | Wheeeeee |
| 00:49:26 | <@arkiver> | 200 million URLs/items being queued |
| 00:49:30 | <@arkiver> | 20 items per job |
| 00:49:39 | <@arkiver> | (first use of 'multi-items') |
| 00:52:21 | <@arkiver> | scripts updated already |
| 00:52:25 | <@arkiver> | 20201031.04 |
| 00:52:43 | <@JAA> | 4 versions in less than an hour? Impressive. |
| 00:52:56 | <@kiska> | :D |
| 00:52:58 | <@arkiver> | yeah forgot some stuff, and fat fingered something |
| 00:58:13 | <@arkiver> | feel free to go wild :P this is a ton of different domains |
| 01:20:30 | <fuzzy8021> | is there a site to see new docker images like https://hub.docker.com/u/warcforceone now that everything is bult as atdr.meo.ws/archiveteam? |
| 01:21:43 | <@JAA> | Fusl: ^ |
| 01:22:13 | <@Fusl> | fuzzy8021: atdr.meo.ws/archiveteam/<project>-grab |
| 01:22:19 | <@Fusl> | *everything* that contains a Dockerfile is there |
| 01:22:45 | <@JAA> | I think fuzzy8021 was asking about a web page listing the images. |
| 01:22:57 | <fuzzy8021> | ya |
| 01:23:59 | <@Fusl> | there is none |
| 01:24:10 | <@Fusl> | github is the source of truth for this |
| 01:24:11 | <fuzzy8021> | k thnaks |
| 01:24:22 | <fuzzy8021> | that works |
| 02:34:49 | <@arkiver> | this is working well |
| 02:50:51 | | mgrandi quits [Read error: Connection reset by peer] |
| 02:51:55 | | mgrandi (mgrandi) joins |
| 03:46:24 | <nico_32> | hum |
| 03:46:31 | <nico_32> | my warrior is not able to find wget+at |
| 03:47:03 | <nico_32> | let's if i can update the container |
| 03:48:22 | <@JAA> | nico_32: This project probably won't work in the warrior. |
| 03:50:41 | <mgrandi> | i have time this weekend, can we just...update that docker image |
| 03:50:49 | <mgrandi> | literally all it needs is a newer version of ubuntu |
| 03:51:03 | <mgrandi> | and ubuntu-20.10 just came out |
| 03:51:03 | <Arcorann> | Agreed, it seems to me like most projects don't work in the warrior nowadays |
| 03:51:17 | <Arcorann> | What's the latest LTS again? |
| 03:51:31 | <mgrandi> | well, in this case, wget-at needs `libzstd` and the one in the version the docker image is using (ubuntu 18.X? ) istoo old |
| 03:51:37 | <mgrandi> | well, in this case, wget-at needs `libzstd` and the one in the version the docker image is using (ubuntu 18.X? ) is too old |
| 03:51:46 | <mgrandi> | ffff i wish that wasn't a feature in irccloud |
| 03:52:33 | <mgrandi> | Ubuntu 20.04.1 LTS |
| 03:54:27 | <nico_32> | ho |
| 03:54:42 | | thuban joins |
| 03:57:05 | <@JAA> | mgrandi: Feel free to try, but test it against a test system, not the live tracker. (Note that the Lua script also talks to the prod system on many current projects.) |
| 03:57:34 | <nico_32> | # Use phusion/baseimage as base image. |
| 03:57:35 | <nico_32> | FROM phusion/baseimage:0.11 |
| 03:57:46 | <mgrandi> | i have to learn how to docker first |
| 03:58:29 | <nico_32> | FROM phusion/baseimage:18.04-1.0.0 |
| 03:58:40 | <nico_32> | & retry |
| 03:58:48 | <mgrandi> | why are we basing it off of that |
| 03:58:49 | <nico_32> | JAA: is there any test tracker? |
| 03:58:50 | <mgrandi> | is it smaller? |
| 03:58:59 | <nico_32> | that's the current config |
| 03:59:05 | <nico_32> | https://github.com/ArchiveTeam/warrior-dockerfile/blob/master/Dockerfile |
| 04:00:09 | <@JAA> | nico_32: Not sure. I never worked much with the tracker. |
| 04:00:26 | <nico_32> | (or we could build wget-at static and just add it to ArchiveTeam/warrior-code2) |
| 04:00:50 | <mgrandi> | that works too |
| 04:01:00 | <@JAA> | We already have a static binary (I think) on the project images. |
| 04:01:05 | <mgrandi> | .oh lol |
| 04:01:22 | <mgrandi> | we should just github it and then have the script `wget` it then |
| 04:01:37 | <nico_32> | ], |
| 04:01:37 | <nico_32> | ['./wget-at'] |
| 04:01:40 | <@JAA> | https://github.com/ArchiveTeam/grab-base-df/blob/474a18cb229913cca0bea8309fd791a1c8975512/Dockerfile |
| 04:01:40 | <nico_32> | )https://github.com/ArchiveTeam/google-sites-grab/blob/e3ccc63b4230baeebb55f368d37a523666aa42fc/pipeline.py#L47 |
| 04:01:51 | <nico_32> | it will always use the local copy |
| 04:01:52 | <mgrandi> | "what is your purpose", "to download wget-at" "oh my god" |
| 04:02:13 | <mgrandi> | /s/your/my |
| 04:03:19 | <mgrandi> | yes, there is a lot of duplication of logic, we could make a separate git repo that has a script that tries to download static / compile / throw an error if nothing else works and just submodule that or something |
| 04:03:29 | <mgrandi> | (rather than including get-wgetat.sh in every project) |
| 04:03:48 | <@Fusl> | there's the next person wasting brain cycles on implementing something that will be made obsolete |
| 04:03:52 | <@Fusl> | -> #warrior |
| 04:04:01 | <@Fusl> | it will be entirely rewritten from scratch |
| 04:04:03 | <@Fusl> | no ubuntu |
| 04:04:04 | <@Fusl> | no debian |
| 04:05:03 | | Dallas quits [Client Quit] |
| 04:08:06 | | Dallas (Dallas) joins |
| 04:08:09 | <mgrandi> | ok |
| 04:08:44 | <mgrandi> | Also: i have been working on the store.playstation.com stuff, and i was wondering if its worth to set up a full blown warrior project for this |
| 04:09:14 | <mgrandi> | Turns out the site did not get shut down on the 28th, but the new site was merely deployed , and you can still access the old site |
| 04:10:00 | <mgrandi> | Some erratta: we don't have all the URLs yet, that is in progress. Sony is also...inconsistent with banning and i cannot for the life of me figure out what their strategy is |
| 04:10:15 | <@Fusl> | the last time someone wasted braincycles around here was when they were trying to rewrite the tracker in python and getting rid of redis, we have since made progress on rewriting it in lua and improving performance by a factor of literally 300000% |
| 04:10:31 | <mgrandi> | i ran 20 boxes to do HTML scrapes with wget-at and 10 of them were banned but 10 weren't |
| 04:10:55 | <mgrandi> | (what made the perf go up that much? o.o) |
| 04:10:59 | <@Fusl> | lua |
| 04:11:12 | <mgrandi> | what was it written in before? |
| 04:11:14 | <@Fusl> | ruby |
| 04:11:24 | <mgrandi> | hmm, i guess lua is neat then |
| 04:11:35 | <@Fusl> | and many redis single command calls |
| 04:11:52 | <@JAA> | It's more about how the code is written than what language. |
| 04:12:07 | <mgrandi> | Anyway, store.PSN project, the urls are just simple URLs, potential banning, the URL list is not complete yet but the VGPC discord is working on it |
| 04:12:11 | <@Fusl> | it's now pretty much doing a lot of the magic in redis for locking items and scanning through all the queues |
| 04:12:51 | <mgrandi> | not sure if everyone is too busy to set up another project |
| 04:14:24 | <mgrandi> | but thought i should bring it up here before i try to make something myself to do it |
| 04:27:04 | | qw3rty_ joins |
| 04:30:38 | | qw3rty__ quits [Ping timeout: 252 seconds] |
| 04:31:00 | <nico_32> | hum |
| 04:31:12 | <nico_32> | there is a new version of the warrior ova |
| 04:31:59 | | deni quits [Read error: Connection reset by peer] |
| 04:32:11 | | deni (deni) joins |
| 04:34:02 | <nico_32> | wget-at built and installed sucessfully! |
| 04:37:33 | <nico_32> | i recommand a note to tell users to upgrade their warrior to v3.1 |
| 04:45:08 | | godane1 quits [Ping timeout: 252 seconds] |
| 05:17:58 | | godane joins |
| 05:17:58 | | godane is now authenticated as godane |
| 05:41:39 | <nico_32> | success! i can run the docker image for google-sites-grab on my synology nas |
| 07:29:12 | | odemgi_ joins |
| 07:31:27 | | odemgi quits [Ping timeout: 240 seconds] |
| 07:39:16 | | Hackerpcs quits [Quit: Hackerpcs] |
| 07:41:32 | | Hackerpcs (Hackerpcs) joins |
| 12:27:12 | <@Kaz> | Sean Connery has died |
| 12:27:19 | <@Kaz> | pls archivebot all the things |
| 12:52:52 | <Arcorann> | Is there a channel for the URL project? |
| 12:55:21 | <wessel1512> | and is ther a docker image for URL project? |
| 12:55:44 | | qw3rty_ quits [Ping timeout: 240 seconds] |
| 12:55:54 | <wessel1512> | cant find it on https://hub.docker.com/u/warcforceone |
| 12:58:48 | <@Kaz> | we no longer use dockerhub |
| 12:59:12 | <@Kaz> | docker image is available at atdr.meo.ws/archiveteam/urls-grab |
| 12:59:36 | <@Kaz> | apart from the fact it appears to be broken |
| 12:59:58 | <Arcorann> | For that matter is there a page which describes the goals/etc. of the project in more detail? |
| 13:00:48 | <@Kaz> | should probably make a wiki page I guess |
| 13:01:04 | <@Kaz> | tl;dr I think is basically 'a project we can throw a collection of random urls at and have them grabbed' |
| 13:01:17 | <@Kaz> | distributed archivebot, in a very limited form |
| 13:01:41 | <wessel1512> | archivebot lite |
| 13:03:28 | <wessel1512> | it works atdr.meo.ws/archiveteam/urls-grab |
| 13:05:37 | <@Kaz> | ah perfect |
| 13:05:51 | <@Kaz> | yeah, we're moving pretty much everything over to drone |
| 13:05:53 | <@Kaz> | so avoid dockerhub limits |
| 13:06:02 | <Arcorann> | I've been wondering if it was feasible to archive every page linked from https://www.novelupdates.com/ (novel translation link aggregation site) |
| 13:06:10 | <@Kaz> | sure it is |
| 13:06:32 | <@Kaz> | spam arkiver enough to see if he'll do it |
| 13:15:37 | <wessel1512> | also maybe add a web gui to the docker registry |
| 13:16:16 | <@Kaz> | there is one - I think you have to log in to it though |
| 13:16:35 | <@Kaz> | the simple assumption is that every new project should be auto-built there |
| 13:20:02 | <wessel1512> | maybe also deploy something like this https://hub.docker.com/r/joxit/docker-registry-ui/ |
| 13:21:00 | <wessel1512> | so that normal pleps like me can see what docker images are available |
| 13:22:35 | <@kiska> | https://server8.kiska.pw/uploads/fd9061eb92b61b77/image.png :D |
| 13:23:00 | <@Kaz> | https://atdrone.meo.ws/ArchiveTeam/urls-grab |
| 13:23:16 | <@Kaz> | repos are public, you just can't list them |
| 13:23:23 | <@Kaz> | if you log in via github it may let you |
| 13:25:07 | <wessel1512> | let you what ? |
| 13:25:12 | <@Kaz> | list repos |
| 13:32:35 | | Hackerpcs quits [Ping timeout: 252 seconds] |
| 13:33:30 | | Hackerpcs (Hackerpcs) joins |
| 13:53:38 | <@arkiver> | mgrandi: how many URLs/apps? |
| 14:19:52 | | Arcorann quits [Read error: Connection reset by peer] |
| 14:20:52 | <@JAA> | Aw, RIP. https://seanconnery.com/ is dead, couldn't find anything else. I set up a monitor for the site. |
| 14:22:44 | | Hackerpcs quits [Ping timeout: 240 seconds] |
| 14:26:56 | | Hackerpcs (Hackerpcs) joins |
| 16:12:42 | | tar-xvf joins |
| 16:14:57 | | odemgi_ quits [Ping timeout: 240 seconds] |
| 16:26:07 | | @arkiver quits [Excess Flood] |
| 16:26:38 | | arkiver (arkiver) joins |
| 16:26:38 | | @ChanServ sets mode: +o arkiver |
| 16:32:18 | | Smiley quits [Remote host closed the connection] |
| 16:48:50 | <@JAA> | https://seanconnery.com/ is back up since a few minutes and has been replaced by a message about his death. |
| 16:48:56 | <@arkiver> | ah |
| 16:50:24 | <@arkiver> | there's quite some utm_* params in urls-grab |
| 16:51:06 | <@arkiver> | we'll queue those again to backfeed |
| 16:52:16 | <@Kaz> | it's also just an image, which is an interesting way to go about making a static site |
| 16:58:51 | <@arkiver> | yeah so upcoming is: |
| 16:59:03 | <@arkiver> | - don't abort all items if one of the multi items is bad |
| 16:59:14 | <@arkiver> | - queue back without utm_* params |
| 16:59:55 | <@arkiver> | later on we'll also queue some page requisites, but need to add some good checks for that |
| 17:10:25 | | @arkiver quits [Excess Flood] |
| 17:10:54 | | arkiver (arkiver) joins |
| 17:10:54 | | @ChanServ sets mode: +o arkiver |
| 17:19:35 | | qw3rty joins |
| 17:31:43 | | godane quits [Read error: Connection reset by peer] |
| 18:00:12 | | Ryz quits [Remote host closed the connection] |
| 18:01:10 | | Ryz (Ryz) joins |
| 18:40:29 | <jodizzle> | JAA: Do you have a method for getting Disqus comments? I think I remember you mentioning something about that a while back. I was thinking about getting the Disqus comments on https://afreshcup.com/ |
| 18:40:46 | <jodizzle> | Hm, but it seems like the Disqus comments there aren't loading at all, at least for me. |
| 18:40:56 | <jodizzle> | Would be good to know anyway, though. |
| 18:41:35 | <@JAA> | jodizzle: Er, I never actually finished that. But yes, I have something that kind of works, mostly. |
| 18:45:42 | <jodizzle> | Oh? |
| 18:48:22 | <@JAA> | Reminds me that I wanted to grab the Picosong and Kotaku/Gizmodo/Lifehacker UK discussions from there though. |
| 18:48:43 | <@JAA> | As for grabbing the comments on the site, that's a disgusting mess. |
| 18:50:04 | | Dallas quits [Client Quit] |
| 18:51:08 | <jodizzle> | Darn, that sounds really annoying |
| 18:51:42 | <jodizzle> | Can someone confirm, by the way, that the Disqus comments aren't loading for them on https://afreshcup.com/? I want to know if it's just my browser/connection. |
| 18:53:07 | | Dallas (Dallas) joins |
| 18:53:30 | <@JAA> | The 'disgusting mess', by the way, is that the Disqus URL includes the <title> of the page embedding it. |
| 18:53:53 | <@JAA> | And it includes it twice, and the two might differ IIRC. |
| 18:53:55 | <@JAA> | So yeah... |
| 18:54:32 | <@JAA> | It basically has to be done with specific code for each site on which you want to archive those to emulate what the site's JS does with the <title>. |
| 18:54:40 | <@JAA> | I did some of this on Picosong, I believe. |
| 18:55:44 | <@JAA> | And yes, I can confirm that it doesn't work. Looks like the forum no longer exists: https://disqus.com/home/forum/a-fresh-cup/ |
| 19:00:41 | <nico_32> | Kaz: https://pix.milkywan.fr/B0RCcmXh.png <= it look like that on Synlogy' docker app |
| 19:01:17 | <@Kaz> | hmm, neat |
| 19:06:00 | <nico_32> | yes |
| 19:06:10 | <nico_32> | but the app is a little buggy |
| 19:08:22 | <jodizzle> | JAA: Thanks |
| 19:11:44 | | RKenshin quits [Ping timeout: 240 seconds] |
| 19:14:13 | | Kenshin joins |
| 19:15:02 | | godane (godane) joins |
| 20:34:29 | | VADemon joins |
| 21:17:10 | | lennier2 joins |
| 21:21:04 | | lennier1 quits [Ping timeout: 240 seconds] |
| 21:21:11 | | lennier2 is now known as lennier1 |
| 21:27:48 | <Wingy> | JAA apparently I don't need an AT wiki password reset lol |
| 21:27:56 | <Wingy> | It was in my password manager the whole time |
| 21:27:58 | <Wingy> | just no URL |
| 21:28:03 | <Wingy> | so not in the "this tab" category |
| 21:28:26 | <Wingy> | and I couldn't remember it because it was random |
| 21:30:53 | <@JAA> | lol |
| 21:30:58 | <@JAA> | Well, great. :-) |
| 21:37:12 | | Dallas quits [Client Quit] |
| 21:37:25 | | Dallas (Dallas) joins |
| 21:54:44 | | Dallas quits [Ping timeout: 240 seconds] |
| 22:04:04 | | Dallas (Dallas) joins |
| 22:11:45 | <OrIdow6> | Sounds like this URL project is basically warriorbot |
| 22:13:21 | <@arkiver> | sort of |
| 22:13:25 | <@arkiver> | updates coming yp |
| 22:13:26 | <@arkiver> | up |
| 22:15:38 | | Dallas quits [Ping timeout: 252 seconds] |
| 22:16:52 | | Dallas (Dallas) joins |
| 22:19:12 | <@Fusl> | Out of memory: Kill process 7067 (redis-server) score 447 or sacrifice child |
| 22:19:14 | <@Fusl> | RIP redis |
| 22:25:06 | <Zerote> | is there a wiki page and/or dedicated channel for the URL project? |
| 22:26:02 | <@JAA> | Nope |
| 22:31:35 | <@arkiver> | let's make a channel! |
| 22:31:49 | <@arkiver> | code is updated, this will speed things up a lot (not aborting 20 URLs when one is bad) |
| 22:33:12 | <@arkiver> | update asap please, will set as minimum version soon |
| 22:33:15 | <@arkiver> | 20201031.06 |
| 22:33:23 | <@arkiver> | any ideas for a channel? |
| 22:34:39 | | godane1 joins |
| 22:35:34 | | godane quits [Read error: Connection reset by peer] |
| 22:37:31 | | Arcorann (Arcorann) joins |
| 22:38:24 | | sirvy quits [Ping timeout: 240 seconds] |
| 22:39:46 | | Doranwen quits [Remote host closed the connection] |
| 22:48:09 | <OrIdow6> | Pretty hard to "mock" URLs as a general concept |
| 22:48:11 | <OrIdow6> | #emailaddress |
| 22:50:12 | <@arkiver> | https://bits.blogs.nytimes.com/2009/10/12/the-webs-inventor-regrets-one-small-thing/ |
| 22:50:27 | <@arkiver> | #noslashes ? |
| 22:50:34 | <@arkiver> | not sure if I like it :P |
| 22:50:49 | <OrIdow6> | #// |
| 22:51:08 | <@arkiver> | fuck yes |
| 22:51:10 | <@arkiver> | #// works |
| 22:51:12 | <@arkiver> | nice |
| 22:51:38 | <@arkiver> | any strong opinions about that channel anyone? |
| 22:52:19 | <@Fusl> | nope |
| 22:52:23 | | sirvy joins |
| 22:52:25 | <@Fusl> | i like it |
| 22:53:56 | <@JAA> | +1 |
| 22:55:15 | <@arkiver> | HCross: I see you just went faster, but with the old version |
| 22:56:04 | <@HCross> | … Maybe because nobody told me there was an update |
| 22:56:41 | <@arkiver> | well I did write here, but didn't ping you |
| 22:56:48 | <@arkiver> | pinged you now though :) |
| 23:03:04 | | Dallas quits [Ping timeout: 240 seconds] |
| 23:04:52 | | lennier1 quits [Client Quit] |
| 23:08:10 | | Dallas (Dallas) joins |
| 23:13:40 | | lennier1 (lennier1) joins |
| 23:16:23 | <@arkiver> | #// for the URLs project! |
| 23:33:43 | | DLoader_ joins |
| 23:36:50 | | DLoader quits [Ping timeout: 252 seconds] |
| 23:36:53 | | DLoader_ is now known as DLoader |