| 00:05:48 | <OrIdow6> | Looks like it could be pretty big? |
| 00:06:11 | <OrIdow6> | So that would be a good reason for early discovery |
| 00:09:29 | | Cook_Me_Plox joins |
| 00:15:43 | | hooway quits [Client Quit] |
| 00:16:55 | <Cook_Me_Plox> | Hi ArchiveTeam! I was wondering if someone could give me some advice on trying to get a website un-excluded from Internet Archive. runescape.com has been blocked for over a decade, but I work closely with the website owner and I think I can get them to agree to un-exclude |
| 00:17:04 | | lennier1 quits [Ping timeout: 240 seconds] |
| 00:17:27 | <Cook_Me_Plox> | It's my understanding that the Alexa crawls that make up the bulk of Internet Archive, are not themselves purged from anywhere when someone makes an exclude request |
| 00:17:45 | <@JAA> | Cook_Me_Plox: I think the owner would have to contact info@archive.org about that. |
| 00:18:15 | <flashfire42> | Thats usually how that works JAA they have to contact info@archive.org also with evidence they own the domain similar to how an exclusion request is made |
| 00:18:26 | <flashfire42> | From my understanding anyway |
| 00:18:30 | <flashfire42> | I will note |
| 00:19:00 | <Cook_Me_Plox> | Thanks! Do you know if IA would pick up the old entries again? Are you aware of any other cases of a site owner un-blocking a site? |
| 00:19:42 | <flashfire42> | I believe everything is stored and not deleted but made unavailable. Not working there I wouldnt know for sure but AFAIK nothing is truly deleted from IA unless it is straight up illegal in all the world over |
| 00:20:14 | <Cook_Me_Plox> | Yeah I suspect at a bare minimum they're not going through and deleting partitions of the original Alexa crawls |
| 00:21:03 | <flashfire42> | Its all speculation but as far as I know nothing is truly deleted from archive.org except for perhaps CSEM the rest would be darked and made inaccessible but the data safe |
| 00:21:04 | <@JAA> | Yes, I believe it's just an access block in the Wayback Machine. |
| 00:24:02 | | lennier1 (lennier1) joins |
| 00:25:12 | <Cook_Me_Plox> | Thanks! And this is probably a silly question, but there's no way you're aware of to restrict access of certain websites to certain users, or otherwise privately convey the archives? That probably doesn't even make sense as a concept here. |
| 00:25:45 | <Cook_Me_Plox> | Basically, the site owner would very much like me to have access to the archives, but they don't own them. And there's a concern that having them be fully public could have some account security ramifications |
| 00:25:56 | <Cook_Me_Plox> | I'm just trying to figure out the best way to maneuver around that |
| 00:27:50 | <atphoenix> | WBM works as a scan of the public web; i.e. the content there was on the public web |
| 00:29:09 | <Cook_Me_Plox> | Right. The concern is that the stuff that was on the public web back in ~2002 (usernames, etc on hiscores) could have a negative effect on account security |
| 00:32:06 | | lennier1 quits [Ping timeout: 252 seconds] |
| 00:33:47 | <@JAA> | I don't think so, especially not when the snapshots are part of larger archives like Alexa's or IA's web crawls. |
| 00:34:43 | <Cook_Me_Plox> | That makes sense. I really appreciate the help, y'all |
| 00:35:01 | <atphoenix> | I don't see particular security concerns |
| 00:35:17 | <atphoenix> | as this neither example is something that should be used for account security |
| 00:35:37 | <Cook_Me_Plox> | I'm not saying I agree with the concern necessarily, just that the site owner might have it |
| 00:36:28 | <atphoenix> | at least it shouldn't have been used that way in the past, and definitely shouldn't be used in place of actual account security measures like passwords or 2-factor auth. |
| 00:36:29 | <Cook_Me_Plox> | There's a bit of a black market for people trying to recover runescape accounts from long ago, and having some sort of list of old usernames would likely result in more accounts being compromised |
| 00:36:39 | <Cook_Me_Plox> | It's in addition to passwords, of course |
| 00:39:11 | <@JAA> | ~2002 would mean the original Runescape, and the servers for that were shut down a couple years ago, right? |
| 00:39:12 | <atphoenix> | if there are any *current* security measures that might be tied to old information that was publicly accessible...I'd suggest that those current measures be reset/forced to be changed |
| 00:40:00 | <OrIdow6> | Is this the same owner who excluded it in the first place, or someone who bought the domain later? |
| 00:40:10 | <Cook_Me_Plox> | It's the same company from 20 years ago |
| 00:40:16 | <OrIdow6> | Oh |
| 00:40:30 | <Cook_Me_Plox> | Although that sounds hard to validate in the general case, lol |
| 00:41:26 | <Cook_Me_Plox> | And yeah, the original servers were shut down 2 years ago but a lot of the accounts migrated. And any accounts from 2004 onwards still exist |
| 00:42:05 | <@JAA> | Right |
| 00:43:18 | <@JAA> | I'm sure IA has a decent process for validating ownership. Easiest case is if the domain is still registered and owned by the same person/company, of course. Whois history etc. |
| 01:02:41 | <OrIdow6> | So long story short, Freshlive has hundreds of times the videos that were in that list |
| 01:02:54 | <OrIdow6> | And AB didn't even pick up all the videos because it uses JS navigation |
| 01:02:59 | <OrIdow6> | Maybe not hundreds, maybe just tens |
| 01:25:18 | <purplebot> | File:Freshlive-logo-transparent.png uploaded by Arkiver (+0) just now -- https://www.archiveteam.org/?diff=45915&oldid=0 |
| 01:26:18 | <purplebot> | File:Freshlive-icon.png uploaded by Arkiver (+0) just now -- https://www.archiveteam.org/?diff=45916&oldid=0 |
| 01:27:15 | <@arkiver> | let's make a channel for periscope |
| 01:27:17 | <@arkiver> | anyone ideas? |
| 01:28:45 | <@arkiver> | also |
| 01:28:51 | <@arkiver> | a project for freshlive.tv is coming up |
| 01:28:55 | <tech234a> | #microscope |
| 01:29:00 | <@arkiver> | nice |
| 01:29:04 | <@arkiver> | lets do it |
| 01:31:14 | <@arkiver> | I'm expecting freshlive to start tomorrow |
| 01:31:18 | <@arkiver> | we'll try to get all |
| 01:31:29 | <@arkiver> | kiska: could you setup a target for freshlive? |
| 01:31:33 | <@arkiver> | it's |
| 01:31:46 | <@arkiver> | archiveteam_freshlive_ |
| 01:31:48 | <@arkiver> | freshlive_ |
| 01:32:05 | <@arkiver> | Archive Team Freshlive: |
| 01:33:41 | <@arkiver> | no idea on size yet, at least TBs likely |
| 01:36:06 | | Cook_Me_Plox quits [Remote host closed the connection] |
| 01:37:09 | <tech234a> | Has anyone scraped any sites based on the Wicket framework before? Endomondo uses it for profile pages and the challenges list and I've been having some trouble with it. It seems to be that you click on a more link and then it makes a special request to /?=x[some long token] which returns some kind of XML/HTML combination. |
| 01:45:26 | <@kiska> | rsync://rsync.hel1.kiska.pw/freshlive/:downloader/ |
| 01:46:36 | <@arkiver> | kiska: awesome! |
| 01:46:41 | <@arkiver> | will ping you when we start |
| 01:46:54 | <@arkiver> | tech234a: not sure, got URLs? |
| 01:47:01 | <@arkiver> | maybe I can check it tomorrow, today I don't have too much time |
| 01:47:36 | <@kiska> | Have fun! |
| 01:47:39 | <@kiska> | Factory active\ |
| 01:48:08 | <tech234a> | arkiver: Yeah I'm also looking into it, may have possibly figured it out but not sure yet. Here are some URLs: https://www.endomondo.com/challenges https://www.endomondo.com/profile/20515169 |
| 01:49:40 | <@arkiver> | tech234a: looks like it's in the source? |
| 01:49:44 | <@arkiver> | of https://www.endomondo.com/profile/20515169 at least |
| 01:50:16 | <tech234a> | Yeah but it seems like specific headers or cookies are required otherwise it just gives a redirect response or a copy of the homepage |
| 01:51:55 | <@arkiver> | also I see a 'random' param in there blegh |
| 01:52:11 | <@arkiver> | hope it |
| 01:52:16 | <@arkiver> | hope it's not random |
| 01:52:30 | <tech234a> | it's possible that's a cache buster |
| 01:52:33 | <@JAA> | #findelmundo |
| 01:52:43 | <@arkiver> | right |
| 02:00:01 | | Dallas quits [Client Quit] |
| 02:03:47 | | Dallas (Dallas) joins |
| 02:54:44 | | Arcorann_ quits [Ping timeout: 240 seconds] |
| 03:43:00 | | AeonG__ quits [Read error: Connection reset by peer] |
| 04:52:48 | | qw3rty__ joins |
| 04:56:28 | | qw3rty_ quits [Ping timeout: 252 seconds] |
| 06:36:52 | | lennier1 (lennier1) joins |
| 07:54:32 | | godane2 joins |
| 07:56:44 | | godane1 quits [Ping timeout: 240 seconds] |
| 08:05:17 | | Arcorann_ joins |
| 08:18:20 | | DLoader quits [Ping timeout: 252 seconds] |
| 08:27:37 | | DLoader joins |
| 08:41:40 | | hooway joins |
| 08:57:29 | | godane2 quits [Read error: Connection reset by peer] |
| 08:58:30 | | godane2 joins |
| 09:00:19 | | britmob2 quits [Quit: britmob2] |
| 09:23:11 | | honno joins |
| 09:28:44 | | wickedplayer494 quits [Ping timeout: 252 seconds] |
| 09:29:15 | | wickedplayer494 joins |
| 09:48:47 | | DLoader_ joins |
| 09:51:17 | | DLoader quits [Ping timeout: 252 seconds] |
| 09:51:18 | | DLoader_ is now known as DLoader |
| 10:00:32 | | godane2 quits [Client Quit] |
| 10:23:29 | | Matthww quits [Client Quit] |
| 10:32:07 | | Matthww joins |
| 10:48:37 | | honno_ joins |
| 10:51:14 | | honno quits [Ping timeout: 252 seconds] |
| 11:02:54 | <@HCross> | arkiver: Saturday - I've got more hardware coming for a full flash NVMe target |
| 11:06:37 | | honno__ joins |
| 11:09:04 | | honno_ quits [Ping timeout: 240 seconds] |
| 12:03:09 | | honno__ quits [Client Quit] |
| 12:49:51 | | DLoader_ joins |
| 12:52:44 | | DLoader quits [Ping timeout: 240 seconds] |
| 12:52:50 | | DLoader_ is now known as DLoader |
| 13:41:21 | | taka joins |
| 13:49:02 | | britmob2 joins |
| 14:11:42 | | taka leaves |
| 15:01:04 | | Arcorann_ quits [Ping timeout: 240 seconds] |
| 15:21:18 | <@arkiver> | HCross: impressive! |
| 15:21:36 | <@arkiver> | that might be very nice for projects with many smaller files? |
| 15:41:35 | | Stilett0 quits [Remote host closed the connection] |
| 16:38:30 | | godane (godane) joins |
| 17:08:20 | | Doranwen quits [Ping timeout: 252 seconds] |
| 17:12:02 | | Doranwen joins |
| 17:51:29 | | Doranwen is now authenticated as Doranwen |
| 17:58:18 | <purplebot> | Coronavirus/Notable deaths edited by Ka (-18, mb) just now -- https://www.archiveteam.org/?diff=45917&oldid=45902 |
| 18:11:17 | <purplebot> | Deathwatch edited by Switchnode (+161, add niconico metadata purge) just now -- https://www.archiveteam.org/?diff=45918&oldid=45914 |
| 18:15:26 | | wizwiz joins |
| 18:15:57 | <wizwiz> | hi guys I want to become a warrior but I'm not very techy and my PC is not that good. I want to rent a windows VPS, put virtualbox/vmware and go ahead and install Warrior Appliance |
| 18:15:58 | <wizwiz> | can anyone recommend me such a service? as far as I know not a lot of vps providers are accepting vmware installation on vps... |
| 18:15:58 | <wizwiz> | and I guess I can't run warrior appliance directly on a windows vps right? |
| 18:16:38 | <@JAA> | #warrior is *still* the right channel for this. |
| 18:17:09 | | `md joins |
| 18:17:18 | <purplebot> | Niconico edited by Switchnode (+297, add notes about metadata purge) just now -- https://www.archiveteam.org/?diff=45919&oldid=44157 |
| 18:17:29 | <`md> | hi :D |
| 18:17:32 | <`md> | does anyone here seed the libgen fiction torrents? |
| 18:21:18 | <SketchTheCow> | I'm delighted to find out that Tom Cruise has adopted my Archive Team management style |
| 18:23:19 | <thuban> | i'm thinking of editing the deathwatch page to add archivebot job links (where i can find them) for sites that didn't/don't have their own project pages. y/n/q? (will the truncated job id shown by https://archive.fart.website/archivebot/viewer/ be sufficient for this?) |
| 18:23:24 | | godane quits [Ping timeout: 240 seconds] |
| 18:46:31 | <@JAA> | thuban: It's something I've been considering as well, but I think it would get messy quite quickly, especially for sites with a bunch of subdomains, social media accounts, etc. |
| 18:47:55 | <@JAA> | Perhaps the best way would be to collect those relevant URLs rather than AB job IDs, and then putting them in a <ref> with a group attribute. |
| 18:48:16 | <@JAA> | And then link it to the new indexer once I finally finish and launch that. |
| 18:52:56 | <jodizzle> | One question related to this I've had: is there a "threshold" for creating a project page? If the solution to archiving a site was just throwing it in AB, then I don't think that site needs a separate wiki page. But for example, archiving stuff from the XDA DevDB didn't involve a warrior project, but it did involve preparing a few lists of URLs to !ao <, and it might be nice to have a page that |
| 18:53:02 | <jodizzle> | describes that and points to those lists. |
| 18:54:10 | <thuban> | JAA: yes, i had some doubts--i don't think social media generally makes it to deathwatch, but i know we've done some of that stuff through e.g. chromebot and i'm not sure we even have browsable logs for that. what do you mean by "relevant URLs" here? |
| 18:55:14 | <@JAA> | jodizzle: Thresholds include the site's significance, but most importantly, it's 'can anyone be arsed to create a page for it?'. |
| 18:57:00 | <@JAA> | thuban: Lots of social media archival also through AB with socialbot (or previously plain snscrape). chromebot will also be included in my new indexer, but there's no such thing for it currently. I suppose the relevant URLs would more or less be what we were able to find and threw into the bots, similar to the election pages. |
| 19:00:42 | <thuban> | in some cases those would be extremely long lists; i don't know that stuffing all that into deathwatch (already a long page with a lot of references) would be more usable than job ids (or just link(s) to domain(s) in the indexer. |
| 19:00:59 | <thuban> | the new one sounds nice; anything useful i can do? |
| 19:01:06 | <jodizzle> | JAA: Ha, true |
| 19:05:23 | <thuban> | s/domain(s)/domain(s))/ |
| 19:14:38 | <@JAA> | thuban: That's why I mentioned <ref> with groups. Could wrap the <references> inside a collapsible to not display them by default. But yeah, it would bloat an already large page even more. My thinking was that it'd be easy to pass the URL list into the new indexer in the future to get accurate job lists without an explicit dependency on AB's job ID generation. |
| 19:15:38 | <@JAA> | Basically, the new indexer replaces the viewer but also includes the ArchiveBot/* page bot functionality. And it will index chromebot and DPoS projects as well. |
| 19:16:54 | <@JAA> | It's already mostly done, just need to find time and motivation to do the last missing pieces. |
| 19:21:11 | <thuban> | JAA: i'll sit on the deathwatch thing for a while then. i'm free for the immediate future, so if i can be of any help, don't hesitate to ask |
| 19:36:20 | | Stiletto joins |
| 19:40:44 | | spirit quits [Ping timeout: 240 seconds] |
| 19:42:28 | | spirit joins |
| 20:00:47 | <@arkiver> | the freshlive project has started! |
| 20:00:51 | <@arkiver> | kiska is in as target |
| 20:11:06 | <@arkiver> | HCross: no ideas on a rate limit |
| 20:11:12 | <@arkiver> | also the images are served slow |
| 20:11:20 | <@arkiver> | so we might skip them if it becomes a problem |
| 20:11:32 | <@arkiver> | just updated version to 20201216.02 |
| 20:11:37 | <@HCross> | I was planning on my ... usual approach |
| 20:12:22 | <@arkiver> | we'll see what happens |
| 20:13:35 | <@arkiver> | a lot of this is not public |
| 20:13:58 | <@arkiver> | the videos on the site ^ |
| 20:14:05 | <@arkiver> | few thousand are |
| 20:18:07 | <@arkiver> | and there might be multi GB items |
| 20:20:30 | <@HCross> | Can we have Samsung XR back please :p - I want to be able to do 5Gbps on a single item |
| 20:26:08 | <@HCross> | ... how is something in Amazon this slow |
| 20:29:28 | <@HCross> | arkiver: I just broke my server by accident and need to reboot, can you requeue me please |
| 20:51:20 | <@kiska> | RIP did you tell kubernetes to rm -rf --no-preserve-root / |
| 21:04:05 | | LeGoupil joins |
| 21:15:16 | | LeGoupil quits [Client Quit] |
| 21:41:54 | <@arkiver> | HCross: I'll requeue everything at the end |
| 21:42:03 | <@arkiver> | yeaah it's not fast |
| 21:42:11 | <@arkiver> | I'm doing a big one now - at 2.5 GB |
| 21:44:40 | | lunik1 quits [Quit: :x] |
| 21:45:03 | | lunik1 joins |
| 21:54:22 | <@HCross> | I’ve got my machine hooked up to a 10Gbps pipe, 100 concurrent is barely making 400Mbit |
| 21:54:44 | <SketchTheCow> | As seen on the telegrad charts, teamarchive2 was heading for a bit of an issue until getting the history fixed caused me to notice it and set it off. |
| 21:54:55 | <SketchTheCow> | Part of the reason was archive.org keeps having outages and those block up s3 to hell. |
| 22:09:13 | <atphoenix> | channel ideas for freshlive, if a channel is desired -> #deadmeat #freshundead #zombievid #freshzombie |
| 22:39:43 | | Arcorann_ joins |
| 23:17:24 | | balrog quits [Ping timeout: 240 seconds] |
| 23:18:21 | | Matthww quits [Client Quit] |
| 23:18:38 | | Matthww joins |
| 23:27:47 | | wickedplayer494 is now authenticated as wickedplayer494 |
| 23:30:16 | | balrog joins |
| 23:30:17 | | balrog is now authenticated as balrog |
| 23:34:57 | <@arkiver> | HCross: Craigle: scripts have been updated, new minimum is 20201217.01 |
| 23:35:32 | <fuzzy8021> | can a freshlive docker image get built please? |
| 23:35:47 | <@arkiver> | ah I had to ping someone for that |
| 23:35:52 | <fuzzy8021> | think its a manual enable |
| 23:35:53 | <fuzzy8021> | ya |
| 23:36:13 | <fuzzy8021> | dont remmeber who did it last time |
| 23:36:41 | <@arkiver> | kiska: can you please do this ^ |
| 23:36:45 | <@arkiver> | i remembered :) |
| 23:37:04 | <fuzzy8021> | or Fusl i would assume could |
| 23:38:36 | <@Kaz> | hold on |
| 23:38:38 | <@Kaz> | i poke things |
| 23:39:13 | <@kiska> | Yeah kaz can do it as well xD |
| 23:39:27 | <@kiska> | I am doing job interviews since I am unemployed |
| 23:41:26 | <@Kaz> | image is now building: atdr.meo.ws/ArchiveTeam/freshlive-grab |
| 23:41:41 | <@arkiver> | kiska: good luck! |
| 23:41:45 | <fuzzy8021> | thanks |
| 23:42:59 | <@Kaz> | we should add a 'running with docker' section to the readme |
| 23:43:05 | <@Kaz> | which would also remind us to enable the builds |
| 23:43:15 | <@Kaz> | (image build complete btw fuzzy8021) |
| 23:44:29 | <fuzzy8021> | pulling now thanks |