| 00:14:34 | | lennier2 joins |
| 00:17:04 | | lennier1 quits [Ping timeout: 240 seconds] |
| 00:17:05 | | lennier2 is now known as lennier1 |
| 00:26:34 | | hooway quits [Client Quit] |
| 00:50:52 | | Arcorann (Arcorann) joins |
| 00:54:51 | | XXLuigiMario (XXLuigiMario) joins |
| 01:33:58 | | ivan (ivan) joins |
| 01:34:54 | <ivan> | brad: what I would really like is for someone to maintain the docker support and deal with users using it, since I do not use docker |
| 01:39:09 | <Arcorann> | https://twitter.com/SteadyStudios <-- owner of this account appears to have been arrested on child pornography charges |
| 01:42:17 | <Arcorann> | https://teddit.net/r/manga/comments/kp22ke/news_stefan_koza_official_translator_for_jujutsu/ <-- source |
| 01:54:28 | <XXLuigiMario> | ~450 GB worth of OSU beatmaps are going down as the website http://bloodcat.com is shutting down soon |
| 01:55:03 | <XXLuigiMario> | I've compiled a list with all the files: https://transfer.notkiska.pw/4uLOA/bloodcat.txt |
| 01:56:19 | <XXLuigiMario> | Site a bit slow at the minute. Source: https://www.reddit.com/r/osugame/comments/kos8ei/you_had_a_long_run_we_will_miss_you_bloodcat/ |
| 02:00:01 | | Dallas quits [Client Quit] |
| 02:03:41 | | Dallas (Dallas) joins |
| 02:04:09 | <brad> | @ivan: I’m certainly not an expert with Docker, but if you’re starting with “From Ubuntu”, there shouldn’t be a whole lot of steps there and it should play out just like the instructions for installing on a normal Ubuntu host. You just record them in a Dockerfile instead. And if that layer is registered with Dockerhub, then I think others shouldn’t have to worry about the steps where things have to be compiled, they just install from |
| 02:04:10 | <brad> | the layer that has already been compiled and installed. |
| 02:04:34 | <brad> | But I will let you know once I’ve gotten it working. |
| 02:05:10 | <@JAA> | Fusl has done that before. |
| 02:05:25 | <@JAA> | Also, Docker Hub sucks. |
| 02:05:31 | <brad> | We might even start from minimal Ubuntu. |
| 02:07:11 | <brad> | I haven’t played around much with dockerhub. I know we’ve used it in the past for some of the container stuff we’ve done at work, but we’ve also made extensive use of AWS Elastic Container System and their Elastic Container Registry. |
| 02:07:51 | <brad> | Dockerhub (and tools like that) are more for the use case where you’ve done something and you want to share it with the public. |
| 02:08:39 | <brad> | But there might be a better public container registry than Dockerhub. That’s just the only one I know of, and I think it’s probably the first one most people think of. |
| 02:09:01 | <@JAA> | Mhm. We're running our own registry now. Docker Hub sucks because you can't get old builds (even though they're still stored on their servers), they have quite strict rate limits as of late, and they're about to delete 'inactive' images. Turns out hosting petabytes of content for free isn't profitable. Shocking, I know. |
| 02:09:42 | <brad> | Cool, I’ll be happy to use whatever registry you guys like. But I do have to get something working first. |
| 02:12:00 | <@JAA> | Since you mentioned minimal Ubuntu: if the goal is to get the image as small as reasonable, I think Alpine is the way to do that normally. |
| 02:27:52 | <jodizzle> | XXLuigiMario: I threw the bloodcat link into AB, we'll see how it goes. |
| 02:28:33 | <@JAA> | Urban Dictionary seems to have removed their 'previous/next words' thingy in the sidebar at some point since I last looked into it. Not sure if it's still possible to enumerate all entries. |
| 02:28:39 | | lennier2 joins |
| 02:30:10 | <XXLuigiMario> | Thank you jodizzle |
| 02:31:41 | | tzt joins |
| 02:32:04 | | lennier1 quits [Ping timeout: 252 seconds] |
| 02:32:10 | | lennier2 is now known as lennier1 |
| 02:34:22 | <purplebot> | Coronavirus/Notable deaths edited by Ka (+1020) just now -- https://www.archiveteam.org/?diff=46065&oldid=46058 |
| 02:43:03 | <ivan> | I fixed the install instructions for macOS so hopefully I will stop receiving emails about halfbaked docker things that people don't even link to |
| 02:43:53 | <ivan> | but there is so much demand for The Docker that I'll allow it if someone wants to do all the work instead of throwing a Dockerfile over the wall |
| 02:46:20 | <ivan> | my understanding of Docker suggests that there should be one grab-site process per container, but no one's done the work of writing out a whole page or two about how to use grab-site that way through Docker and have a working gs-server |
| 02:49:12 | <ivan> | maybe someday someone can explain to me why users must interact with a 'docker' binary instead of getting docker-platform-wrapped shell scripts installed for them |
| 02:50:30 | <brad> | It would depend on how much resources are made available to the Docker process. You could run multiple grabs from the same container, if you had enough resources. |
| 02:50:57 | <brad> | But there’s definitely a lot that could be done with container orchestration. |
| 02:53:23 | <ivan> | let's say we could only document one approach to using grab-site through docker because people can barely as much as I've already written |
| 02:53:34 | <ivan> | would that be one-container-per-grab-site or 'stuff em all in there' |
| 02:54:42 | <ivan> | can barely read* |
| 03:06:35 | | XXLuigiMario quits [Remote host closed the connection] |
| 03:06:36 | <@JAA> | This is why I mentioned Fusl earlier. She had a setup on mips where I believe gs-server was in one container and each grab-site was in its own container. Not entirely sure though, never looked into the guts of it. |
| 03:06:59 | | Ka quits [Remote host closed the connection] |
| 03:19:35 | | HackMii_ (hacktheplanet) joins |
| 03:20:24 | | HackMii quits [Ping timeout: 240 seconds] |
| 03:20:35 | | HackMii_ is now known as HackMii |
| 03:31:18 | <@Kaz> | in theory I don't see any issue with having them all in their own containers - they can all still communicate with eachother anyhow |
| 03:31:37 | <@Kaz> | docker will give you """dns""" and other lark like that for discovery |
| 04:13:45 | <brad> | Speaking only for myself, I’m with @ivan — document the one official method that is the simplest, and then if people want to try to go beyond that point, then that’s up to them. |
| 04:15:29 | <brad> | I mean, at some point, it seems to me like you would be trying to put together a replacement for the ArchiveTeam Warrior, or at least the entire ArchiveBot project, if you’re trying to get all the containers talking to each other and coordinating with each other. |
| 04:16:15 | <@JAA> | Bad things happened last time someone tried to force an AB pipeline into Docker. |
| 04:16:29 | <OrIdow6> | EggplantN_: I'm here now |
| 04:16:45 | <@JAA> | And a replacement for the warrior is indeed planned. |
| 04:18:41 | <brad> | At the moment, I’m just trying to get a simple Dockerized solution together for myself, which I can pick up and run most anywhere — without having to worry about the OS underneath, or what software and what versions may or may not be installed, etc.... And I’m more than happy to use or re-use any previous work that has been done in this space. |
| 04:19:18 | <brad> | And once I’ve got something working for myself, I’m happy to document that and share that with the rest of the community. |
| 04:20:05 | <brad> | I can say that I do not like trying to re-invent the wheel, at least not if I have a choice. |
| 04:34:11 | | qw3rty__ joins |
| 04:38:13 | | qw3rty_ quits [Ping timeout: 265 seconds] |
| 04:56:43 | | hexa- quits [Quit: WeeChat 2.9] |
| 05:00:03 | | hexa- (hexa-) joins |
| 06:04:10 | | godane quits [Read error: Connection reset by peer] |
| 06:04:37 | | godane (godane) joins |
| 08:49:04 | | @kiska quits [Ping timeout: 240 seconds] |
| 08:49:44 | | flashfire42 quits [Ping timeout: 240 seconds] |
| 08:49:44 | | s-crypt quits [Ping timeout: 240 seconds] |
| 08:51:46 | | s-crypt (s-crypt) joins |
| 08:51:50 | | flashfire42 (flashfire42) joins |
| 08:52:07 | | kiska (kiska) joins |
| 08:52:07 | | @ChanServ sets mode: +o kiska |
| 09:54:32 | | hooway joins |
| 09:55:04 | | hooway_ joins |
| 09:59:04 | | hooway quits [Ping timeout: 240 seconds] |
| 10:44:20 | | asdd joins |
| 10:44:53 | | asdd quits [Remote host closed the connection] |
| 12:05:36 | | HP_Archivist quits [Quit: Leaving] |
| 12:05:56 | | HP_Archivist (HP_Archivist) joins |
| 12:12:22 | <purplebot> | GTF Képhost edited by Bzc6p (+74, /* Archives */ some archives are …) 1 minute ago -- https://www.archiveteam.org/?diff=46066&oldid=46030 |
| 12:30:21 | | HackMii quits [Remote host closed the connection] |
| 12:31:17 | | HackMii (hacktheplanet) joins |
| 13:11:51 | | ragu joins |
| 13:12:08 | | ragu is now authenticated as ragu |
| 13:39:48 | | XXLuigiMario joins |
| 13:39:48 | | XXLuigiMario is now authenticated as XXLuigiMario |
| 14:44:44 | | Arcorann quits [Ping timeout: 240 seconds] |
| 16:06:04 | | nerdguy1138 quits [Ping timeout: 240 seconds] |
| 16:22:14 | | nerdguy1138 (nerdguy1138) joins |
| 16:47:10 | | LeGoupil joins |
| 18:40:55 | <atphoenix> | at risk: at least some fandom.com wikis have messages they are merging with gamepedia.com. e.g. https://wowwiki.fandom.com/wiki/Portal:Main has a very prominent message about merging, with a link to https://wow.gamepedia.com/Wowpedia |
| 18:47:15 | <atphoenix> | also noted in #wikiteam . I'm not sure if this is a new thing, or a continuation of the stuff that was mentioned at https://archiveteam.org/index.php?title=Wikia#Moved_wikis |
| 20:08:32 | | DogsRNice (Webuser299) joins |
| 21:06:50 | <brad> | Hmm. So, while translating the installation instructions for Ubuntu at https://github.com/ArchiveTeam/grab-site#install-on-ubuntu-1604-1804-debian-9-stretch-debian-10-buster I have discovered that the `wget https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer` fails with a certificate problem. |
| 21:06:54 | <brad> | Yes, I can work around this with `--no-check-certificate`, but if I do that then the `pyenv-installer`also fails with a certificate problem. |
| 21:07:38 | <ivan> | do you not have ca-cetificates installed? |
| 21:07:45 | <brad> | I'd rather fix the underlying reason why the `wget` fails, but I'm not sure what the right solution is there. Does anyone have any advice? |
| 21:07:56 | <brad> | Hmm. Good question. |
| 21:09:13 | | spirit quits [Quit: Leaving] |
| 21:10:10 | <ivan> | also why not use a base nix container since grab-site is maintained by me in nixpkgs |
| 21:11:14 | <brad> | I was starting from ubuntu:20.10, but I'm happy to try nix instead. I don't know anything about nix, but I'm happy to give it a try. |
| 21:11:56 | | XXLuigiMario quits [Client Quit] |
| 21:13:30 | | LeGoupil quits [Client Quit] |
| 21:14:45 | <ivan> | maybe https://github.com/LnL7/nix-docker |
| 21:15:08 | <ivan> | nix-env -iA nixpkgs.grab-site |
| 21:17:01 | <hexa-> | or just use nixery.dev/shell/grab-site |
| 21:17:29 | <hexa-> | https://nixery.dev/ |
| 21:18:45 | <ivan> | Oh yeah thanks |
| 21:18:59 | <brad> | https://nixery.dev/shell/grab-site comes up as 404 for me. |
| 21:20:58 | <brad> | Hmm. Is there an example somewhere of how I would build a Dockerfile with https://nixery.dev/ ? |
| 21:23:29 | <brad> | I was thinking that pulling grab-site directly from the repo at https://github.com/ArchiveTeam/grab-site would be a better choice, but I'm happy to use pre-built supported packages that are available from somewhere else. |
| 21:24:00 | <ivan> | nixery is a registry, it doesn't take Dockerfiles. I don't know if it has grab-site available or not, try the docker command instead of the URL in your browser |
| 21:24:11 | <ivan> | yeah it would be reasonable to build from the git repo |
| 21:24:19 | <hexa-> | podman pull wfm |
| 21:24:30 | <hexa-> | also nixery states they're pulling from 20.03 /o\ |
| 21:24:59 | <hexa-> | but if you're maintaing the too yourself and require recent versions, best not rely on the an out of date channel |
| 21:29:23 | <brad> | The goal I was working towards was to be able to provide a simple Dockerfile that could be contributed back to the grab-site repo, so that others could easily get their own container up and running with a minimum amount of fuss. |
| 21:30:17 | <brad> | The repo at https://github.com/LnL7/nix-docker does give a minimal example that it looks like I can start from towards that goal. |
| 21:31:25 | <brad> | Would I just add `nixpkgs.grab-site` to the end of that list to provide to `nix-env -iA`? |
| 21:35:10 | <ivan> | sure |
| 21:37:18 | <brad> | https://www.irccloud.com/pastebin/uZwYIiiK/ |
| 21:39:44 | <brad> | Does that make sense? |
| 21:41:37 | <ivan> | maybe? I don't use docker :-) |
| 21:41:37 | <brad> | And this? |
| 21:41:48 | <brad> | https://www.irccloud.com/pastebin/xbxoacHq/ |
| 21:42:00 | <ivan> | no need for jq or curl |
| 21:42:45 | <brad> | And if we pull out my local images I've built so far, we get: |
| 21:42:53 | <brad> | https://www.irccloud.com/pastebin/bvIDZBJp/ |
| 21:43:27 | <ivan> | I see. maybe debian-based is the way to go then |
| 21:43:59 | <ivan> | though I would recommend trying a stable debian base instead of latest ubuntu releases which are kind of just arbitrary snapshots of debian unstable |
| 21:44:16 | <brad> | For the nix-based version, should bash be the package listed before grab-site, or after? Does it make a difference? |
| 21:44:26 | <ivan> | doesn't matter |
| 21:45:13 | <brad> | I was working from Ubuntu 20.10 because I understood there were some problems that grab-site had with the most recent LTS, 20.04. |
| 21:45:56 | <brad> | I've built grab-site on a plain-jane Ubuntu 20.10 instance, and fired it up and it seemed to work fine, so I took that as the baseline for what I would be trying to do with docker. |
| 21:46:36 | <ivan> | Ubuntu is a system that takes Debian unstable packages at 6 month intervals and makes it much harder to report bugs because the Debian packagers do most of the work but Ubuntu issues can't be reported directly to Debian packagers |
| 21:48:07 | <ivan> | I suppose the base can be changed in the future anyway |
| 21:48:31 | <ivan> | the hard part is writing the documentation for Docker-based use |
| 21:48:51 | <brad> | Okay, I can re-base on Debian. I was doing Ubuntu under the mistaken impression that was the platform preferred by the project. I'm fairly agnostic with regards to the server OS. |
| 21:49:03 | <brad> | Which version of Debian would you suggest as a good starting point? |
| 21:49:16 | <ivan> | whatever the latest stable is, I don't use debian either :-0 |
| 21:51:07 | <@JAA> | Yeah, just debian:stable unless you want to manually update it on new stable releases. |
| 21:51:25 | <brad> | Latest stable appears to be Buster, see https://www.debian.org/releases/ |
| 21:51:55 | <@JAA> | Or even just debian (= debian:latest), which is equivalent I believe. |
| 21:51:58 | <brad> | But I'll try to make a Dockerfile config work for Nix, too. That way people can have a choice. |
| 21:52:20 | <ivan> | the more choices there are, the more configurations the maintainer has to test |
| 21:53:02 | <brad> | Okay, so in that case I will abandon the Ubuntu and Debian versions, and work only on Nix. |
| 21:53:04 | <@JAA> | Actually, there are also the slim images, e.g. debian:stable-slim. Might be worth trying that. |
| 21:53:18 | <@JAA> | Or that :-P |
| 21:53:23 | <ivan> | I'm ok with that |
| 21:54:24 | <brad> | I want to do what is considered best by the maintainers of the repo, and it looks to me like Ivan is the most recent contributor. If there are any other contributors to the repo here that would like to voice their opinion, I'm all ears. |
| 21:54:52 | <@JAA> | ivan built the thing, so yes. :-) |
| 21:54:54 | <brad> | Would you like me to go ahead and open an issue on the grab-site repo for this work? |
| 21:55:19 | <ivan> | sure |
| 21:55:29 | <brad> | I've already forked it and working in a local branch, but I haven't pushed anything up to github yet. |
| 21:55:37 | <brad> | Okay, will do. |
| 21:57:50 | <ivan> | the grab-site README is largely responsible for anyone being able to do anything useful with it and it does not assume that users have much knowledge of the systems used to install grab-site |
| 21:58:12 | <ivan> | if it starts supporting Docker it should tell users exactly how to use grab-site and gs-* tools through docker |
| 21:58:45 | | AeonG joins |
| 21:59:08 | <brad> | Okay, issue #175 is now open at https://github.com/ArchiveTeam/grab-site/issues/175 |
| 22:00:52 | <brad> | Yeah, my thought is that I would document how to run the container under docker to grab a particular site using the defaults, and then provide at least one other example that changes some of the defaults. |
| 22:01:16 | <ivan> | oh you're bknowles. did you get a chance to test the new homebrew on macOS install? |
| 22:02:18 | <brad> | My homebrew on my MacBook Pro is hosed. I fear that I may have to completely wipe it and re-install. So, I had given up on trying that. |
| 22:02:35 | <ivan> | ah ok |
| 22:02:42 | <brad> | My MacMini with 64GB RAM that I use as the home server may also be in the same state. |
| 22:03:15 | <brad> | I figured I'd get the Docker container version going, and then switch over my machines to using that instead of running the code locally. |
| 22:03:47 | <brad> | I also have an Intel NUC that I haven't set up yet, but I hope to soon. |
| 22:04:24 | <ivan> | honestly if you just want it working, Nix works on macOS |
| 22:05:25 | <brad> | The Nix package manager runs locally on macOS? |
| 22:05:27 | <ivan> | yes |
| 22:05:49 | <ivan> | check out the README below the Homebrew section :-) |
| 22:06:29 | <brad> | Hmm. I'll give that a thought. In the meanwhile, I'd kinda like to keep playing with setting up Docker, and then I can run Nix inside of that. |
| 22:07:18 | <brad> | I'm coming around to the idea that I should run less and less stuff directly on the machine(s), and make more use of containers for isolation purposes. |
| 23:01:54 | | XXLuigiMario (XXLuigiMario) joins |
| 23:12:29 | | e joins |
| 23:13:19 | | Arcorann (Arcorann) joins |
| 23:20:21 | | e quits [Remote host closed the connection] |
| 23:46:28 | <brad> | Okay, so I now have a minimal Dockerfile that allows me to start bash interactively under Nix, where I can then log in and start `gs-server` and `grab-site` interactively. Now I need to figure out how to put those two commands into the Dockerfile. |
| 23:49:32 | <brad> | You still want some way to extract the downloaded WARC files to some location where they can be shared publicly, and maybe do that by default. |