HomeSearchPrevious dayNext day

00:03:18hooway_ quits [Client Quit]
01:54:20Joel_ joins
01:55:24Joel_ quits [Remote host closed the connection]
01:57:24XXLuigiMario quits [Ping timeout: 240 seconds]
02:00:01Dallas quits [Client Quit]
02:02:10godane1 joins
02:03:38Dallas (Dallas) joins
02:04:24godane quits [Ping timeout: 240 seconds]
03:06:22<purplebot>Deathwatch edited by Kyan (+154, Webs.com shutting down March 31st …) just now -- https://www.archiveteam.org/?diff=46068&oldid=46060
03:09:29<OrIdow6>Webs.com: paid users are being migrated instead of dropped; there will be no read-only period
03:10:03<@JAA>Migration is incomplete though: 'Unfortunately we are not able to migrate the Members or Forums apps. These pages will be omitted from any sites being migrated to Vistaprint.'
03:14:06<OrIdow6>All Webs sites I've checked contain some JS near the end with a field called "siteId", the values of which suggest to me that there are something like 100m-200m of these sites (if it's sequential_
03:14:09<OrIdow6>*)
03:19:47<OrIdow6>Going to do some checking on this, getting IA CDX right now
03:20:17<@kiska>Name for chan webbed
03:20:27<@kiska>Idea*
03:22:57<@arkiver>webs channel ideas?
03:23:32<@kiska>I propose webbed
03:23:41<@arkiver>fine with that one
03:23:51<@arkiver>#webbed
03:23:55<@arkiver>(from kiska)
03:23:59<@kiska>Not sure what people think
03:24:44<@arkiver>unless someone comes with something *really* great, we'll do #webbed
03:32:22<purplebot>Deathwatch edited by Olicool10 (+157), JustAnotherArchivist (-15, /* 2020 */ Fix Voat announcement …) 24 minutes ago -- https://www.archiveteam.org/?diff=46070&oldid=46068
03:36:45<atphoenix>something spider related? spiderwebs/brokenwebs/webless
03:40:05<atphoenix>Shelobsweb with references to https://en.wikipedia.org/wiki/Shelob (LOTR) and https://en.wikipedia.org/wiki/Charlotte%27s_Web
03:40:56<@JAA>I'm cleaning up Deathwatch at the moment, by the way.
03:41:49<atphoenix>webbedfeet (walking away from webs.com)
03:44:06<atphoenix>if the subdomains aren't too large, maybe they can be run like #noteurdomain and #flashbang as those also had some subdomains in them. e.g. an item could be https://themasterscarpenter.webs.com
03:46:54<atphoenix>^that example wasn't even in WBM just now. Who knows how many more are not there. http://web.archive.org/web/*/https://themasterscarpenter.webs.com/*
03:47:07<atphoenix>*until just now
03:47:28AeonG quits [Read error: Connection reset by peer]
04:27:15honno joins
04:30:24DogsRNice quits [Ping timeout: 240 seconds]
04:30:26<honno>I used the web archives you great folk made for sandbox.yoyogames.com before it closed to curate a collection of all the games (+metadata). currently uploading stuff to https://archive.org/details/@honno :D
04:30:48<honno>hope you dont mind a ping chfoo, thanks for leading that project, means a lot to me
04:33:08qw3rty_ joins
04:36:49qw3rty__ quits [Ping timeout: 252 seconds]
04:37:22<purplebot>Deathwatch edited by JustAnotherArchivist (+263, Move countless deaths and update …) 14 minutes ago -- https://www.archiveteam.org/?diff=46071&oldid=46070
05:07:17<brad>@ivan: Okay, I've got a PR at https://github.com/ArchiveTeam/grab-site/pull/176 . This is not very useful yet. It needs a way to capture WARC files and then move or upload them somewhere. But at least it builds and runs on my machine.
05:08:20<brad>Now may be a good time for me to talk to @Fusl about their containerized version, and see if there is anything I can add to it that may be useful.
05:09:07<@Fusl>brad: https://github.com/ArchiveTeam/grab-site/pull/149
05:09:24<@Fusl>CMD gs-server & \
05:09:25<@Fusl> grab-site 'https://community.fantasyflightgames.com/'
05:09:26<@Fusl>what the
05:09:56<@Fusl>just use the dockerfile we have already based on alpine
05:14:16<brad>I was trying to build something based on Nix, per the previous conversation with @ivan.
05:14:22<purplebot>Webs.com created by JustAnotherArchivist (+18, Redirected page to [[Webs]]) just now -- https://www.archiveteam.org/?diff=46073&oldid=0
05:17:16<@Fusl>ah welp
05:17:22<purplebot>Webs created by JustAnotherArchivist (+521, Created page with "{{Infobox project …) just now -- https://www.archiveteam.org/?diff=46074&oldid=0
05:17:30<@Fusl>you got my dockerfile as example how to properly do it
05:17:33<@Fusl>well
05:17:34<@Fusl>almost properly
05:17:51<@Fusl>it does pip3 install from the pip repo instead of the git repo
05:18:06<@Fusl>err, git repo instead of local repo
05:18:54<@Fusl>you also have examples how to run the gs-server and grab-site in a separate docker container which is what you should be doing
05:19:40<brad>I tried to build that Dockerfile, and got the error I showed at https://github.com/ArchiveTeam/grab-site/pull/149#issuecomment-753761330
05:20:02<@JAA>And volume mounts to solve the 'WARCs inside the container' issue.
05:21:20<brad>That's the "-v" option?
05:21:26<@JAA>Yeah
05:22:10<brad>I only really started playing with Docker today, so it's taking me a while to learn various tricks on how to get it to do certain things.
05:23:22<brad>I've used containers before with AWS ECS and AWS ECR, but I've never spent much time building them myself. It was always someone else building them and uploading them to the registry.
05:26:43<brad>It is interesting to me that the Ubuntu:20.10 based version of the Dockerfile I was building came out with containers that were about 80MB in size, while Nix gives us containers that are 384MB in size, and so far the Python container that I have partially built is at least 236MB in size.
05:30:22<purplebot>Deathwatch edited by JustAnotherArchivist (+4, /* 2021 */ Link Webs.com page) 17 minutes ago -- https://www.archiveteam.org/?diff=46072&oldid=46071
05:31:30<brad>I'm not seeing an obvious way for me to solve this libffi problem when trying to build the Python version of the container. Maybe this is a Python 3.9 versus 3.7 problem?
05:32:57<@JAA>Likely a missing dev package.
05:33:37<@JAA>libffi-dev exists, so try that maybe?
05:35:48<brad>Hmm. I tried python:3.7.9-alpine and that gave me the same error. ;(
05:36:01<brad>I'll try adding libffi-dev
05:38:58<brad>And it failed to find `patch` as well. Added that, too.
05:41:01<brad>Hmm. Now it fails to build the wheel for cryptography, following some issue with openssl. I'm going to try adding that package as well.
05:41:25<brad>Not that I've ever really done anything with Alpine before, but I have heard of it.
05:44:50<brad>Based on the error messages:
05:44:59<brad>https://www.irccloud.com/pastebin/fLd4ZWK5/
05:45:33<@JAA>Hmm, perhaps OpenSSL 1.0 vs 1.1?
05:45:44<brad>I'm thinking we also need to upgrade pip just as soon as we can, before we actually try to install grab-site.
05:48:50<@Fusl>brad: fixed
05:48:58<brad>openssl-dev is getting installed as a package.
05:51:54<brad>Okay, here's my current version of the Python Dockerfile:
05:52:06<brad>https://www.irccloud.com/pastebin/Tum88EEq/
05:52:19<brad>And this seems to build fine.
05:53:22<brad>I'll try running it, along the lines of the PR at https://github.com/ArchiveTeam/grab-site/pull/149#issuecomment-472267153
06:02:29<@Fusl>brad: fixed #2
06:02:32<@Fusl>just tried a test build
06:04:28<brad>Is there a reason you're pulling from the repo https://github.com/ludios/grab-site.git instead of https://github.com/ArchiveTeam/grab-site ? Did you not need to upgrade your version of pip before installing grab-site?
06:04:45<brad>Am I pulling from the wrong repo?
06:05:01<brad>Should I be sending a PR to https://github.com/ludios/grab-site.git instead?
06:13:32<brad>I'm now trying to build the version of the Dockerfile as shown at https://raw.githubusercontent.com/ArchiveTeam/grab-site/927a68bc5ea5d43e9b3958e98a9db0b05b4bee10/Dockerfile
06:17:16<@chfoo>honno: i don't mind. glad to see it being curated too
06:19:30<brad>I can get the gs-server up and running. But the second command to get grab-site itself running, well that's not working.
06:19:49<brad>It shows:
06:19:54<brad>https://www.irccloud.com/pastebin/myzOKqVc/
06:20:27<brad>But then never shows up in `docker ps`, not even `docker ps -a`.
06:21:45<brad>And the gs-server webUI doesn't show anything in progress.
06:22:48<@Fusl>remove --rm and then do docker logs on the container id
06:24:41<brad>Hmm. Looks like I need to set up the /data/ignores file:
06:24:45<brad>https://www.irccloud.com/pastebin/RTFPVFA6/
06:24:59<ivan>brad: ludios/grab-site was moved to ArchiveTeam/grab-site
06:25:03<brad>That doesn't automatically get created in the container.
06:25:06<ivan>github keeps redirects
06:25:15<@Fusl>yeah grab-site expects ignores to exist
06:25:19<@Fusl>or you can just not add the argument
06:27:36<brad>Pulling out the ignores, I now get:
06:27:42<brad>https://www.irccloud.com/pastebin/wocTc9q7/
06:27:59<ivan>erorr is above
06:28:14<ivan>also yeah, you're on python3.9
06:28:27<brad>That's what the container installed.
06:28:56<brad>We will have to change the Dockerfile to specify a different version of Python.
06:29:13<brad>Like `FROM python:3.7.9-alpine`
06:30:48<brad>It looks like gs-server appears to be fine with python 3.9, but maybe grab-site isn't?
06:33:02<ivan>every Python 3.x release now has backwards-incompatible language changes. this is how the Python developers generate new paid maintenance work for developers worldwide
06:33:41<ivan>also yeah grab-site is tested only on Python 3.7
06:33:49<ivan>namedlist was broken on 3.8 I believe
06:34:54<brad>Okay, so now gs-server is up and running on the rebuilt container, now running under python 3.7.9.
06:36:58<brad>Okay, now I'm guessing that my IP has been banned by that site.
06:37:16<brad>Here's the tail end of the logs:
06:37:20<brad>https://www.irccloud.com/pastebin/xpW5bBrp/
06:37:54<ivan>try another user agent, I dunno?
06:38:36<brad>Going down into what looks like the most recent subdirectory under /data, in the `wpull.log` file, I see:
06:38:40<brad>https://www.irccloud.com/pastebin/htZTLKmt/
06:38:46<brad>So, yeah. I got banned.
06:39:03<ivan>it happens sometimes
06:39:42<ivan>some sites insist that you go really slow
06:39:43<brad>But I can also confirm that the grab-site docker container did successfully communicate with the gs-server docker container.
06:41:07<brad>It would be nice to know for how long this ban will last. ;(
06:41:15<brad>But I can test with other urls.
06:44:48<brad>And re-starting the correct container with www.fantasyflightgames.com is now working. It's a throw-away test, but it's working.
06:47:12<brad>@Fusl -- I added a comment at https://github.com/ArchiveTeam/grab-site/pull/149/commits/927a68bc5ea5d43e9b3958e98a9db0b05b4bee10#r551140796 . If you want to update your PR to suit, then I'll be happy to close my issue and withdraw my PR. We can then move forward with your version of the Dockerfile.
06:48:59<brad>I'll be happy to open another issue and related PR for some documentation on how to use the new Dockerfile, based on my experiments.
06:59:16tzt quits [Ping timeout: 252 seconds]
07:23:08<@Fusl>brad: this should do, lmk if it works for you
07:24:51<brad>That should work, but I’ll need to test that when I get back to my laptop upstairs. Gimme a couple of minutes.
07:29:41<brad>Yeah, that results in the same SHA-256 hash for the python:3.7-alpine base image as I got for python:3.7.9-alpine, so everything else on top of that remains the same.
07:30:07<brad>I'll delete my issue and PR.
07:33:06<brad>@ivan -- once you approve the PR at https://github.com/ArchiveTeam/grab-site/pull/149 then I can work on another Issue and related PR to create some documentation around that.
07:34:31<ivan>brad: you can just cherry-pick Fusl's commit
07:34:41<ivan>or apply https://github.com/ArchiveTeam/grab-site/pull/149.patch
07:35:04<ivan>URL needs fixing there too
07:42:29<brad>I can cherry-pick, sure. But I wanted to make sure that their work got committed to the main branch on the repo before I started doing work on top of it.
07:43:43<brad>I want to make sure they get full credit in the repo history before I start making my documentation changes around that file.
07:44:26<ivan>brad: is this how you start a grab-site container?
07:44:28<ivan>docker container run --rm -d -e GRAB_SITE_HOST=172.17.0.1 --name "grab-site_$(cat /proc/sys/kernel/random/uuid)" -v /data:/data:rw IMAGENAME --igon --import-ignores /data/ignores URL
07:44:53<brad>It is almost 2:00AM here. I need to feed the cats and go to bed. I'll try to pick back up on this tomorrow.
07:45:02<ivan>brad: anyone can claim authorship of anything on github, just `wget` and `git am` :-)
07:45:12<brad>You need to make a couple of changes.
07:45:12<ivan>alright
07:45:37<brad>I actually ran the command `sudo docker container run -d -e GRAB_SITE_HOST=172.17.0.1 --name "grab-site_$(cat /proc/sys/kernel/random/uuid)" -v /data:/data:rw efa041e43124 --igon https://www.fantasyflightgames.com/`
07:46:45<brad>I left off the stuff about /data/ignores, and I updated the URL to suit. You also have to fill in the appropriate image name -- in this case, since I built it locally, it was just the short image id.
07:47:26<brad>And this does assume that you've already got a matching `gs-server` running and listening on port 29000 in a different container on that host.
07:48:00<brad>Anyway, bed time.
07:54:02LeGoupil joins
07:54:24HackMii quits [Ping timeout: 240 seconds]
07:54:44HackMii (hacktheplanet) joins
08:04:48ivan leaves
09:13:43hooway joins
09:46:46honno quits [Client Quit]
11:22:22spirit joins
11:54:05burnmjd joins
11:54:13hooway quits [Client Quit]
12:11:22tzt joins
14:26:44Arcorann quits [Ping timeout: 240 seconds]
14:35:44cadence quits [Ping timeout: 240 seconds]
14:55:59godane1 quits [Client Quit]
15:09:38mutantmonkey quits [Remote host closed the connection]
15:10:42mutantmonkey (mutantmonkey) joins
15:52:38lennier2 joins
15:55:04lennier1 quits [Ping timeout: 240 seconds]
15:55:08lennier2 is now known as lennier1
18:08:22<purplebot>Gopher edited by Squidboy (+25, marking link as broken) just now -- https://www.archiveteam.org/?diff=46075&oldid=43077
19:28:38XXLuigiMario (XXLuigiMario) joins
19:46:24DogsRNice (Webuser299) joins
19:49:35godane (godane) joins
20:35:53Pdrwiz joins
20:38:03<Pdrwiz>hello?
20:38:16XXLuigiMario quits [Client Quit]
20:42:13Jon (jon) joins
20:42:59<thuban>hello
20:43:24Jon waves
20:44:01<Pdrwiz>I'm looking for some help viewing the smackjeeves archive that was made. I downloaded everything here: https://archive.org/details/archiveteam_smackjeeves_20201227112535_dd0d5a82 and opened the .warc with Webrecorder. It opened but is all messed up. Is there a place somewhere that describes how to access the archive properly?
21:18:50godane1 joins
21:21:40godane quits [Ping timeout: 252 seconds]
21:40:19godane2 joins
21:41:35LeGoupil quits [Client Quit]
21:43:07godane1 quits [Ping timeout: 252 seconds]
21:52:36cadence (cadence) joins
21:58:58<OrIdow6>Pdrwiz: So first of all, if you are not doing anything special with it, you should be able to use web.archive.org to view it
22:01:35<OrIdow6>If you want to use a local copy, I am going to guess that the problem is that the warcs like the one you have downloaded don't contain various static resources (because they would have been redundant)
22:05:12<OrIdow6>If you want to do this, tell me and I can send you a file that should contain the resources you need
22:07:19<Pdrwiz>Well, it appears that I am a fool and didn't check the web archiving site for an archive. Thanks OrIdow6. I am curious about what the other files that are not warc are, is there a place to read up on what they do? And no thank you, but I do not need a local copy now that I know it's up to browse.
22:10:19<OrIdow6>Pdrwiz: They're just various resources that the website uses - CSS, Javascript, images used in the UI (e.g. the "next" arrow)
22:10:50<Pdrwiz>how do you "add" them back in?
22:11:54<OrIdow6>I have a copy of a warc that contains these resources (from when this project was being tested)
22:12:28<OrIdow6>Which can be merged with the preexisting one
22:12:47<OrIdow6>*the one you downloaded
22:13:20<OrIdow6>web.archive.org mostly seems to gets its copies from people using "save page now"
22:15:22<Pdrwiz>If there is no available tool to merge them then I would love to have it. Internet archive can be slow at times. I'm technically adept so you can just throw me a github link or the like.
22:28:22<purplebot>Freewebs created by Wickedplayer494 (+18, Redirected page to [[Webs]]) just now -- https://www.archiveteam.org/?diff=46076&oldid=0
22:34:22<purplebot>Current Projects edited by Wickedplayer494 (+163, Webs/Freewebs to proposed) just now -- https://www.archiveteam.org/?diff=46077&oldid=46036
22:39:53<OrIdow6>Pdrwiz: https://transfer.notkiska.pw/hLo1s/smackjeeves_some_test_warc.warc.gz
22:40:22<purplebot>File:WebsLogo.jpg uploaded by Wickedplayer494 (+0) just now -- https://www.archiveteam.org/?diff=46078&oldid=0
22:40:25<Pdrwiz>thanks!
22:40:31<OrIdow6>You should just be able to append this to the end of a preexisting warc; though I think webrecorder has an "import" feature that should work anyway
22:40:58<Pdrwiz>wonderful!
22:41:22<purplebot>File:Webs - 1-4-21.png uploaded by Wickedplayer494 (+0) just now -- https://www.archiveteam.org/?diff=46079&oldid=0
22:42:22<purplebot>File:Freewebs February 2008.png uploaded by Wickedplayer494 (+0) just now -- https://www.archiveteam.org/?diff=46080&oldid=0
22:42:22<purplebot>Webs edited by Wickedplayer494 (+191, Assorted imagery and a navbox) just now -- https://www.archiveteam.org/?diff=46081&oldid=46074
22:43:11<OrIdow6>Yep
22:43:17<OrIdow6>No problem
22:44:22<purplebot>Endomondo edited by Wickedplayer494 (+0, It's dead...sort of.) just now -- https://www.archiveteam.org/?diff=46082&oldid=46022
23:06:53spirit quits [Read error: Connection reset by peer]
23:07:22spirit joins
23:23:22<purplebot>Current Projects edited by Wickedplayer494 (-115, Remove Freeml from MtM) 18 minutes ago -- https://www.archiveteam.org/?diff=46083&oldid=46077
23:25:18Pdrwiz quits [Client Quit]
23:34:25<atphoenix>Python compatibility issues have long threads. I like this comment that includes "Thou Shalt Not Break User Space." https://old.reddit.com/r/programming/comments/ev7nld/python_39_and_beyond_backwards_compatibility/ffweud5/
23:41:28jon_ joins
23:43:47Jon quits [Read error: Connection reset by peer]
23:49:22<purplebot>File:Mediafire.com-frontpage.png uploaded by Arkiver (+0) just now -- https://www.archiveteam.org/?diff=46084&oldid=0
23:49:55XXLuigiMario joins
23:51:22<purplebot>File:Mediafire-logo-blue.png uploaded by Arkiver (+0) just now -- https://www.archiveteam.org/?diff=46085&oldid=0
23:55:22<purplebot>File:Mediafire-icon.png uploaded by Arkiver (+0) just now -- https://www.archiveteam.org/?diff=46086&oldid=0

HomeSearchPrevious dayNext day