| 01:11:11 | | jbest (jbest) joins |
| 01:20:26 | | mutantmonkey quits [Remote host closed the connection] |
| 01:20:44 | | mutantmonkey (mutantmonkey) joins |
| 03:59:54 | | DogsRNice quits [Read error: Connection reset by peer] |
| 04:02:52 | | mutantmonkey quits [Remote host closed the connection] |
| 04:03:11 | | mutantmonkey (mutantmonkey) joins |
| 07:16:23 | | Sylirana quits [Ping timeout: 244 seconds] |
| 07:16:50 | | Sylirana (Sylirana) joins |
| 08:00:50 | | LeGoupil joins |
| 11:46:10 | | @Fusl quits [Excess Flood] |
| 11:46:27 | | Fusl (Fusl) joins |
| 11:46:27 | | @ChanServ sets mode: +o Fusl |
| 12:16:03 | | Warrior4h6654y quits [Ping timeout: 244 seconds] |
| 12:56:52 | | housies joins |
| 12:57:20 | <housies> | Hi. |
| 12:57:36 | <Jake> | Hey. |
| 12:57:45 | <housies> | I've got a question about archiving Yahoo! Answers with Warrior. |
| 12:58:03 | <Jake> | Sure! What's up? |
| 12:58:35 | <housies> | The instructions say you just leave it running, and it archives automatically. But is there a way to specify what it archives? I want to make sure my own account is archived, then archive everything else. |
| 13:00:08 | <Jake> | That's not really possible with the current code. All tasks are assigned by the tracker, making it hard to specifically request one account be archived. |
| 13:00:33 | <housies> | That makes sense. |
| 13:00:40 | <housies> | Thanks for answering anyway. |
| 13:01:58 | | masterX244 (masterX244) joins |
| 13:06:09 | <Jake> | No problem, sorry we couldn't help out more. Something like Save Page Now (https://web.archive.org/save/) or something else locally could assist in saving your Y!A account. |
| 13:06:36 | <housies> | Thanks. |
| 13:06:46 | <housies> | I'll see if I can run Warrior anyway. |
| 13:07:35 | | hackerman joins |
| 13:09:54 | | sliccricc (sliccricc) joins |
| 13:10:25 | | zizox joins |
| 13:10:56 | | Nebogipfel2020 joins |
| 13:14:34 | | housies quits [Remote host closed the connection] |
| 13:35:49 | | hackerman14 joins |
| 13:37:41 | | hackerman quits [Ping timeout: 244 seconds] |
| 13:41:13 | | billy549 quits [Quit: ZNC - https://znc.in] |
| 14:07:20 | | Josey9 joins |
| 14:10:37 | | etnguyen03 (etnguyen03) joins |
| 14:12:53 | | fofofofofofofofo (fofofofofofofofo) joins |
| 14:13:30 | | LeighR (LeighR) joins |
| 14:15:16 | <LeighR> | As mentioned in #noanswers, fofofofofofofofo figured out that Yahoo treats each IPv6 /128 as an individual IP address - not really a block, or are using a really small block size. Upshot of that is, running a ton of Docker containers with concurrency=1 using IPv6 networking appears to be a great use of resources. |
| 14:15:59 | <LeighR> | I figured out that you can add new Docker IPv6 networks, including to a host where the main bridge network is IPv4 and has running containers. |
| 14:16:18 | | G4te_Keep3r joins |
| 14:16:50 | <LeighR> | 1) Find your eth0's IPv6 block - "ip addr" on modern Ubuntu. On Hetzner VM's, this will be a /64. |
| 14:19:06 | <LeighR> | 2) Use the four populated address segments (example: 201f:4544:ab33:9283), then add a colon and a new random one, and end with two colons and the subnet (201f:4544:ab33:9283:3922::/80) |
| 14:19:46 | <LeighR> | 3) docker network create --ipv6 --subnet 201f:4544:ab33:9283:3922::/80 myipv6subnet |
| 14:20:57 | <LeighR> | 4) run your grabbers in that new network: docker run -d --name reddit-grab-01 --label=com.centurylinklabs.watchtower.enable=true --restart=unless-stopped atdr.meo.ws/archiveteam/reddit-grab:latest --concurrent 1 YourNicknameHere |
| 14:21:44 | <LeighR> | 5) use "df" and "htop" occasionally as you scale up to make sure you're not going to crash your VM ;) |
| 14:22:36 | | Jeroen joins |
| 14:24:43 | | matmen joins |
| 14:27:06 | | m0nika (m0nika) joins |
| 14:36:21 | <Josey9> | I'm following the wiki, but the first command (which starts "docker run -d --name watchtower"...) just gives me errors. I get "docker run requires at least 1 argument" and "-bash: /var/run/docker.sock: Permission denied"" |
| 14:37:35 | <LeighR> | do "docker ps" and see what you get |
| 14:37:53 | <LeighR> | it might be a permissions issue |
| 14:38:19 | <LeighR> | ignore what I wrote about IPv6 - it's for people who have got everything else working with docker |
| 14:39:03 | <LeighR> | (need to run to the post office before it closes, be back later) |
| 14:39:25 | <Josey9> | I get "CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES" |
| 14:39:35 | <Josey9> | Thanks. No rush. |
| 14:46:31 | <hilda> | the user you're logged in as needs to be a member of the docker group to run docker commands |
| 14:46:39 | <hilda> | that's what the 'permission denied' means |
| 14:47:57 | <hilda> | you can add yourself to the group by running 'sudo usermod -aG docker [your username]' |
| 14:50:10 | <Josey9> | I've just done that, but the error is the same |
| 14:50:16 | | Minkafighter joins |
| 14:51:12 | | Jack_Thompson joins |
| 14:54:43 | <hilda> | did you copy and paste the entire docker run command? the one for watchtower should end with '--interval 3600' |
| 14:54:46 | <hilda> | all one line |
| 14:57:47 | <Josey9> | Yes I did, but I've just seen the mistake. For some reason if I paste into the VPS any : become ; - I've changed it back to : and it's doing something now! |
| 14:59:01 | <Josey9> | For the next command, do I replace [username] with my PC's username? |
| 15:01:34 | <Minkafighter> | username should be the name you will be using on the scoreboard |
| 15:02:02 | <Josey9> | Cool :) |
| 15:10:10 | <Josey9> | I think I have all the basics set up now. I've got to go out for a bit, so I'll finish up later. |
| 15:11:15 | | Warrior4h6654y joins |
| 15:13:47 | <masterX244> | thanks for the info on the warrior/Docker IPV6 @leighR |
| 15:16:53 | | Josey9 quits [Ping timeout: 244 seconds] |
| 15:54:03 | | billy549 (Billy549) joins |
| 16:33:09 | <LeighR> | I missed a bit in the run command... the network! |
| 16:33:55 | <LeighR> | 4) run your grabbers in that new network: docker run -d --name reddit-grab-01 --network=myipv6subnet --label=com.centurylinklabs.watchtower.enable=true --restart=unless-stopped atdr.meo.ws/archiveteam/reddit-grab:latest --concurrent 1 YourNicknameHere |
| 16:42:22 | | LeGoupil quits [Client Quit] |
| 17:39:21 | | Josey9 joins |
| 17:58:01 | | hackerman joins |
| 17:58:23 | | hackerman14 leaves |
| 18:39:20 | | Josey947 joins |
| 18:39:25 | | Josey9 quits [Ping timeout: 244 seconds] |
| 18:41:35 | <Josey947> | Are you still here LeighR? I'm trying to do the docker IPv6 thing. With ip addr I get "inet6 2a01:4f9:c011:20cd::1/64 scope global". Is that the bit I want? |
| 18:41:55 | <LeighR> | yes. |
| 18:42:20 | <LeighR> | have you used Docker much before? |
| 18:43:04 | <Josey947> | Never! But I'm good with Linux, so I'm hoping to get it right. |
| 18:43:45 | <LeighR> | hmmm... you might want to get used to running a few containers in "normal" IPv4 first |
| 18:44:13 | <LeighR> | before throwing Docker networking into this |
| 18:45:54 | <Josey947> | Ok. Is the networking likely to mess things up if I don't know what I'm doing? |
| 18:46:09 | <LeighR> | yep, and cause yourself all sorts of weird problems |
| 18:46:40 | <LeighR> | right now, when you do a docker run command, your containers are all landing in the default network, "bridge" |
| 18:48:25 | <LeighR> | do "docker network ls" |
| 18:49:53 | <Josey947> | There three, bridge, host and none. |
| 18:50:33 | <Josey947> | *There's three: |
| 18:51:38 | <Josey947> | If I'm sticking to IPv4, should I still leave the yahoo answers docker with "--concurrent 1" or should I up that to 5 (or run 5 containers?)? |
| 18:55:42 | <LeighR> | I recommend starting out with --concurrent 1 because if we need to back off even lower than 5 per IP, it will be easier |
| 18:56:17 | <Josey947> | Cool. Thank you for helping. |
| 18:56:28 | <LeighR> | get used to checking your disk usage (not so much with this project, but with some, it can fly up quickly!) |
| 18:57:35 | <LeighR> | there is very little resource consumption difference between one container with concurrent=5, and five containers with concurrent=1 |
| 18:58:05 | <LeighR> | but in my opinion, it's a lot easier to see what's going on with the latter |
| 18:59:09 | <LeighR> | oh, and you might want to run watchtower at 600 or 900 seconds instead of 3600 |
| 19:00:03 | <Josey947> | Can I edit it, or do I delete it, and put it back? |
| 19:01:43 | <LeighR> | you'd stop it, then remove it, then run it again |
| 19:02:18 | <Josey947> | Cool. Thank you. |
| 19:02:23 | <LeighR> | generally speaking, you stop, remove then relaunch docker containers if you need to change something about their configuration |
| 19:02:43 | <LeighR> | be careful when they contain data you might care about - that's what docker volumes are for |
| 19:09:49 | <Josey947> | Excellent. So I have the watchtower running with a 600 interval, and I have one Yahoo Answers container running. |
| 19:09:53 | <Josey947> | How can I 'see messages' from the container, to know if it's doing anything? |
| 19:10:21 | <Josey947> | (which I know it isn't at the moment because it's pasued) |
| 19:11:01 | <LeighR> | docker logs containername |
| 19:11:19 | <LeighR> | and if you only want the last 100 lines, docker logs -n 100 containername |
| 19:11:59 | <Josey947> | Awesome! Thank you :) |
| 19:12:03 | <LeighR> | if you want some practice, the ongoing reddit job is the one in my sample |
| 19:18:06 | <Josey947> | I've started that, and the log says "There aren't any items available for this project at the moment". Is that normal? |
| 19:19:14 | <masterX244> | yes, reddit is not a steady stream since it only has items when the users create the data there |
| 19:19:30 | <masterX244> | reddit grab is acting like a "tail -f" on new stuff |
| 19:20:19 | <Josey947> | Good to know. Thanks. |
| 19:51:45 | | Josey947 quits [Ping timeout: 244 seconds] |
| 20:03:46 | | matmen quits [Remote host closed the connection] |
| 20:40:51 | | Nebogipfel2020 quits [Remote host closed the connection] |
| 21:09:10 | | Warrior4h6654y quits [Remote host closed the connection] |
| 21:23:10 | | LeighR quits [Remote host closed the connection] |
| 22:00:03 | | mutantmonkey quits [Remote host closed the connection] |
| 22:00:19 | | mutantmonkey (mutantmonkey) joins |