| 00:13:19 | | Arcorann (Arcorann) joins |
| 00:20:04 | | Arcorann_ joins |
| 00:22:24 | | Arcorann quits [Ping timeout: 240 seconds] |
| 00:48:12 | | Arcorann (Arcorann) joins |
| 00:51:29 | | Arcorann_ quits [Ping timeout: 265 seconds] |
| 00:56:13 | | Yg84WJchYH87d3A5tF quits [Client Quit] |
| 01:06:21 | | Arcorann quits [Ping timeout: 244 seconds] |
| 01:30:19 | | Arcorann (Arcorann) joins |
| 01:38:44 | | blue507 quits [Ping timeout: 240 seconds] |
| 01:40:55 | | nepeat quits [Remote host closed the connection] |
| 01:41:01 | | nepeat joins |
| 01:41:03 | | nepeat is now authenticated as nepeat |
| 01:43:56 | | blue507 joins |
| 02:25:22 | | systwi quits [Read error: Connection reset by peer] |
| 02:26:06 | | systwi (systwi) joins |
| 02:38:44 | | TheFluffyCat joins |
| 02:54:13 | <TheFluffyCat> | Are we looking for a faster URL archiver than TTT2? I'm working on one written in C#, and so far it shreds right through bitly. |
| 03:17:41 | | hex joins |
| 03:29:39 | | @Fusl quits [Changing host] |
| 03:29:39 | | Fusl (Fusl) joins |
| 03:29:39 | | guybrush.hackint.org sets mode: +o Fusl |
| 03:29:50 | | @Fusl quits [Changing host] |
| 03:29:50 | | Fusl (Fusl) joins |
| 03:29:50 | | guybrush.hackint.org sets mode: +o Fusl |
| 04:21:34 | | atphoenix quits [Changing host] |
| 04:21:34 | | atphoenix (atphoenix) joins |
| 04:23:25 | | qw3rty__ joins |
| 04:27:32 | | qw3rty_ quits [Ping timeout: 265 seconds] |
| 05:05:27 | | TheFluffyCat quits [Remote host closed the connection] |
| 05:23:42 | | systwi quits [Read error: Connection reset by peer] |
| 05:24:20 | | systwi (systwi) joins |
| 05:24:46 | | nepeat quits [Client Quit] |
| 05:24:54 | | nepeat joins |
| 05:27:55 | | blue507 is now authenticated as blue507 |
| 05:34:10 | | manyworlds joins |
| 05:34:37 | | manyworlds quits [Remote host closed the connection] |
| 07:28:48 | | HackMii quits [Remote host closed the connection] |
| 07:30:39 | | HackMii (hacktheplanet) joins |
| 09:00:18 | | britmob2 quits [Quit: britmob2] |
| 09:34:27 | | hex quits [Ping timeout: 265 seconds] |
| 09:54:16 | | hex joins |
| 10:31:04 | | hex quits [Client Quit] |
| 12:13:29 | | Arcorann_ joins |
| 12:17:20 | | Arcorann quits [Ping timeout: 265 seconds] |
| 12:34:38 | | Matthww quits [Quit: The Lounge - https://thelounge.chat] |
| 12:57:08 | | Matthww joins |
| 13:24:31 | | Datechnom quits [Ping timeout: 265 seconds] |
| 13:34:23 | | Datechnom (Datechnom) joins |
| 13:44:55 | | Yg84WJchYH87d3A5tF joins |
| 13:50:06 | | britmob2 joins |
| 14:04:58 | | Arcorann (Arcorann) joins |
| 14:07:32 | | Arcorann_ quits [Ping timeout: 265 seconds] |
| 14:14:15 | | OpSecNoob joins |
| 14:14:35 | <OpSecNoob> | Hey, whats the docker link for this project? |
| 14:18:16 | <AK> | atdr.meo.ws/archiveteam/urls-grab:latest |
| 14:18:24 | <@JAA> | No, that's another project. |
| 14:19:36 | <AK> | Oh poo good point, The warrior docker image is here: https://hub.docker.com/r/archiveteam/warrior-dockerfile/ |
| 14:20:00 | <@JAA> | That's not the Docker image for this project though. :-P |
| 14:20:27 | <OpSecNoob> | JAA do you have it? |
| 14:20:40 | <@JAA> | atdr.meo.ws/archiveteam/terroroftinytown-client-grab:latest |
| 14:20:57 | <OpSecNoob> | thanks! |
| 14:22:07 | <AK> | Ahh I didn't know there was a separate image for this project, I'd thought this was the only one baked into warrior. Won't make that mistake again. |
| 14:39:26 | | Arcorann quits [Ping timeout: 265 seconds] |
| 14:42:57 | <OpSecNoob> | So this appears to be the linux image, is there one for windows (I'm getting: docker: image operating system "linux" cannot be used on this platform.) |
| 14:53:34 | <OpSecNoob> | nvm, got it :) |
| 14:59:01 | | AK quits [Client Quit] |
| 15:00:19 | | AK joins |
| 15:08:55 | | Matthww quits [Ping timeout: 265 seconds] |
| 16:01:37 | | Matthww joins |
| 16:09:04 | | Matthww quits [Ping timeout: 240 seconds] |
| 16:12:50 | | Matthww joins |
| 16:13:10 | | nepeat is now authenticated as nepeat |
| 16:44:44 | | OrIdow6 quits [Ping timeout: 240 seconds] |
| 16:45:23 | | OrIdow6 (OrIdow6) joins |
| 16:47:21 | | OrIdow6 quits [Remote host closed the connection] |
| 16:48:55 | | OrIdow6 (OrIdow6) joins |
| 17:42:48 | | TheFluffyCat joins |
| 18:08:16 | | sierra_wright joins |
| 18:10:50 | <sierra_wright> | how long has the urlteam project been going on? and what % of the target spaces have been checked? out of curiosity |
| 18:12:41 | <TheFluffyCat> | At the current rate of scanning (650/sec), it has been running for about 3 and a half years. |
| 18:12:50 | | AK is now authenticated as AK |
| 18:13:54 | <TheFluffyCat> | It's about time we upgrade the project's code, too... My machines are running at about 5% capacity at the moment. |
| 18:15:16 | <sierra_wright> | i get a LOT of no free items messages but i can clearly see that for example ow.ly isn't even out of the numbers in the most significant space yet |
| 18:15:42 | | TheFluffyCat68 joins |
| 18:16:17 | <TheFluffyCat68> | arkiver do you know the database schema/where we can get the crawl results? I'd like to make a compatible crawler that could replace the current one. |
| 18:16:53 | <AK> | sierra_wright: I think that's the tracker limiting how many items it gives out to avoid ddosing the target websites |
| 18:17:34 | <@JAA> | Correct |
| 18:18:03 | | dsi joins |
| 18:18:27 | <sierra_wright> | i figured that was part of it |
| 18:18:39 | | TheFluffyCat quits [Ping timeout: 244 seconds] |
| 18:18:49 | <TheFluffyCat68> | Ha. Bye former me. |
| 18:18:56 | <@JAA> | TheFluffyCat68: There is no DB. The output are simple text files (compressed), and you can find them here: https://archive.org/search.php?query=terroroftinytown |
| 18:19:01 | <TheFluffyCat68> | Thanks. |
| 18:19:01 | <@JAA> | The format should be quite self-explanatory. |
| 18:19:21 | <TheFluffyCat68> | I assumed it was Anti-DDoS- but even so, 600/second is not a lot, especially for the 20-or-so projects that are currently running. |
| 18:19:55 | <TheFluffyCat68> | At that rate, you wouldn't even net a network of people helping you- you can afford that hosting easily between 2-3 people. |
| 18:20:46 | | madmax (madmax) joins |
| 18:20:49 | <@JAA> | Only 2 or so projects are running currently I think. |
| 18:20:55 | <AK> | Some of the other projects have different numbers of people working on it, so it varies |
| 18:21:30 | <@JAA> | Ok, 4 it seems. |
| 18:22:06 | <TheFluffyCat68> | Also, seems that the site digbig.com is having issues with their shortener; all requests to the domain go on a 301 loop. |
| 18:22:20 | <TheFluffyCat68> | I noticed a few of my machines crawling it earlier today on TTT. |
| 18:22:36 | <sierra_wright> | i just got assigned urlteam by hitting the "archiveteam choice" project, so if the other ones have even less scanning... |
| 18:22:58 | <TheFluffyCat68> | Urlteam is the only one available on the Warrior. |
| 18:22:58 | <madmax> | i got a question: is there any reason to have maximum 6 concurrent items running per warrior? |
| 18:23:11 | <TheFluffyCat68> | I think it's just arbitrary. Frustrates me too, madmax |
| 18:23:21 | <TheFluffyCat68> | The rest are independent docker containers, sierra_wright |
| 18:24:03 | <sierra_wright> | i see |
| 18:24:05 | <@JAA> | That's mostly historic at this point. We used to run all projects through the warrior. Nowadays, this is the only one that still works. |
| 18:24:07 | <TheFluffyCat68> | You can find them on the website, alongside the project descriptions |
| 18:24:34 | <@JAA> | But back then, when we switched the default project, people would start running whatever we were working on. And many times, doing that at high concurrency was a Bad Idea. |
| 18:24:41 | <madmax> | ahh okay, JAA. it ended up that i run 17 docker containers now, each 6 items… |
| 18:24:46 | <@JAA> | Rate limits, disk space, etc. |
| 18:25:12 | <@JAA> | You can run the terroroftinytown-client-grab container or the scripts at up to 20 concurrency though if you want. |
| 18:25:12 | <TheFluffyCat68> | Think of it this way: If bit.ly did not want to be crawled, they would have released their database. |
| 18:25:33 | <@JAA> | Note that you'll anyway get limited to one item per shortener per IP because rate limits. |
| 18:28:05 | <TheFluffyCat68> | As a test I just downloaded 300 bit.ly URLs in quick succession; no rate limiting on there. |
| 18:28:51 | <TheFluffyCat68> | Especially with how lightweight serving a URL shortener is- and how many people likely use it per second- |
| 18:28:59 | <TheFluffyCat68> | I think it's worth it to speed up the process a little. |
| 18:29:32 | <sierra_wright> | yeah crank up the volume! |
| 18:29:50 | <TheFluffyCat68> | Wait- correction- 400 URLs. |
| 18:29:56 | <madmax> | TheFluffyCat68: same for me. i wrote a script to just fest the headers but there was definitely no rate limit, at least for bit.ly |
| 18:30:10 | <TheFluffyCat68> | I'll try tinyurl now, too. |
| 18:30:23 | <AK> | The risk is they do spot it, implement rate limiting or just start banning ips |
| 18:31:26 | <@JAA> | You can get banned from bit.ly quite easily. |
| 18:31:54 | <@JAA> | Also, I don't think we can control the items/IP limit on a per-shortener basis. |
| 18:32:31 | <@JAA> | The better approach is to just spread things out by adding more shorteners. |
| 18:32:52 | <TheFluffyCat68> | Heh... are you suggesting I make a URL shortener? |
| 18:33:02 | <@JAA> | But well, someone has to do it, monitor it, adapt the rules as needed, etc. |
| 18:33:26 | | @JAA slaps TheFluffyCat68 around a bit with a large trout |
| 18:33:40 | <madmax> | JAA: i totaly agree. would be too hard to have separate rates for each shortener. bit.ly has published the rates for api access |
| 18:33:51 | <TheFluffyCat68> | You use the API? |
| 18:33:55 | <AK> | Doh just realised the log messages for ttt are different to the other projects, can't collect them in the same way |
| 18:33:58 | <TheFluffyCat68> | Wait, no. |
| 18:34:01 | <madmax> | 1000 calls per ip per hour |
| 18:34:03 | <@JAA> | I'm pretty sure we don't. |
| 18:34:16 | <madmax> | no in a custom script |
| 18:34:21 | <@JAA> | AK: Yes, this shares nearly no code with the other projects. |
| 18:35:03 | <TheFluffyCat68> | Have we ever tried... asking them... for a list of their URLs? |
| 18:35:12 | <@JAA> | Another thing that's been on the todo list for a loooooong time is writing WARCs. Now we're just throwing away much of the information, and the redirects don't go into the Wayback Machine either. |
| 18:35:37 | <@JAA> | Bitly is a member of 301Works. They know about efforts like this. |
| 18:35:59 | <@JAA> | They're just arseholes and don't share their DB. |
| 18:36:26 | <TheFluffyCat68> | Why don't we ask nicely |
| 18:36:35 | <@JAA> | I'm certain IA has done just that years ago. |
| 18:37:04 | <@JAA> | The whole concept behind 301works is that shorteners give their mappings to IA for safekeeping, and once the shortener ceases to operate, they're made public. |
| 18:37:24 | <TheFluffyCat68> | "Dear Sir or Madam, we are a group of rogue internet archivists. For a while, we've been crawling your site, and seeing which shortened links go where. This is very slow, however, and I think it'd be better for the both of us if we could just have a list of your shortened URLs. Sincerely, Terror of Tiny Town." |
| 18:37:56 | <TheFluffyCat68> | *No one*'s tried that? |
| 18:38:11 | <@JAA> | <@JAA> I'm certain IA has done just that years ago. |
| 18:39:58 | <TheFluffyCat68> | Hmm, okay. |
| 18:40:53 | <TheFluffyCat68> | (Off topic) While we're here, has anyone considered re-making Warrior with Lua as the primary environment? Would be nice to centralize projects again. |
| 18:45:26 | <@JAA> | A new warrior is planned, yes. |
| 18:45:51 | <TheFluffyCat68> | Where can I find development? #warrior? |
| 18:48:07 | <@JAA> | Nothing's public yet, but yes, discussion will likely be there once things start moving. |
| 19:23:40 | | TheFluffyCat68 quits [Remote host closed the connection] |
| 19:23:49 | <dsi> | Hey everyone, has anyone gotten the standalone to work through Azure Containers? I can get warrior to work (az container create -g ArchiveTeamWarrior --name warrior --image archiveteam/warrior-dockerfile --ip-address public --port 8001) but not so much luck with the standalones. |
| 19:47:02 | <madmax> | dsi: i don't know what does not work for you. but a container in azure would be way to expensive. i have tried it some time ago for another project, one instance costs us 30€/month |
| 19:47:37 | <madmax> | a vps which can handle several docker containers is cheaper (approx. 10€/month or less) |
| 19:56:52 | <dsi> | madmax i get a lot of free Azure credit that i'm not currently using. |
| 19:57:21 | <dsi> | but that does sounds about right; each warrior container is ~1 a day. |
| 20:05:53 | | blue507 quits [Remote host closed the connection] |
| 20:10:59 | | madmax quits [Client Quit] |
| 20:11:57 | | sliccricc quits [Changing host] |
| 20:11:57 | | sliccricc (sliccricc) joins |
| 20:12:30 | | sliccricc quits [Client Quit] |
| 20:12:45 | | sliccricc joins |
| 20:13:04 | | sliccricc is now authenticated as sliccricc |
| 20:13:05 | | sliccricc quits [Changing host] |
| 20:13:05 | | sliccricc (sliccricc) joins |
| 20:13:26 | | sliccricc quits [Remote host closed the connection] |
| 20:13:40 | | sliccricc (sliccricc) joins |
| 20:26:37 | | atphoenix quits [Quit: Leaving] |
| 20:27:02 | | atphoenix (atphoenix) joins |
| 21:00:31 | | madmax joins |
| 21:00:31 | | madmax is now authenticated as madmax |
| 21:15:05 | | madmax quits [Client Quit] |
| 21:32:35 | | inf33ct joins |
| 22:06:01 | | ragu quits [Read error: Connection reset by peer] |
| 22:06:27 | | ragu joins |
| 22:55:41 | | OpSecNoob quits [Remote host closed the connection] |
| 23:21:52 | | Arcorann (Arcorann) joins |