| 00:06:31 | <@arkiver> | Fusl: did you compile the wget-at with gnutls? |
| 00:06:34 | <@arkiver> | or openssl |
| 00:06:40 | <@arkiver> | JAA: ^ |
| 00:06:43 | <@Fusl> | openssl i think |
| 00:06:46 | <@Fusl> | let me check |
| 00:07:06 | <@arkiver> | right since gnutls is first choice in get-wget-lua.sj |
| 00:07:07 | <@arkiver> | sh |
| 00:07:43 | <@Fusl> | ye, openssl |
| 00:07:48 | <@arkiver> | alright good |
| 00:07:53 | <@arkiver> | JAA: so that should be fine then? |
| 00:08:17 | <@JAA> | Yep, unless there's bugs in OpenSSL as well. :-P |
| 00:08:37 | <@JAA> | But at least this one should be fine. |
| 00:08:42 | <@JAA> | (Context: #noteurdomain) |
| 00:09:44 | <jodizzle> | OrIdow6: Interesting, from a quick look a lot of these seem small and probably pretty suitable for straightforward AB runs. |
| 00:10:12 | <jodizzle> | Also, seems like we're using #flashbang, so this should probably go there. |
| 00:10:14 | <@arkiver> | alright |
| 00:10:32 | <@arkiver> | let's put them in AB |
| 00:10:52 | <@arkiver> | or well will leave AB stuff to JAA |
| 00:19:33 | <@arkiver> | JAA: how is NGA images going? |
| 00:19:47 | <@JAA> | arkiver: Good, should finish in 8 hours or so. |
| 00:19:57 | | HP_Archivist quits [Quit: Leaving] |
| 00:19:58 | <@arkiver> | that's awesome |
| 00:20:07 | <@JAA> | I had to split it up into a few jobs and stuff, but worked well as far as I can tell. |
| 00:35:21 | <purplebot> | Archiveteam:IRC edited by JustAnotherArchivist (+148, /* Creating a channel */ Add disabling …) just now -- https://www.archiveteam.org/?diff=46012&oldid=46010 |
| 00:38:22 | <OrIdow6> | https://transfer.notkiska.pw/tFcLF/Smackjeeves_get_redirects_to_dead_comics.sh - capture some redirects to SmackJeeves comics - if someone on the WBM whitelist wants to run & upload the warc, would be appreciated - see #archivebot for more details |
| 01:38:22 | <purplebot> | Archiveteam:IRC edited by Atphoenix (+112, /* Creating a channel */ added …) 14 minutes ago -- https://www.archiveteam.org/?diff=46013&oldid=46012 |
| 02:00:01 | | Dallas quits [Client Quit] |
| 02:03:21 | <purplebot> | File:Adobe-swf-icon.png uploaded by Arkiver (+0) just now -- https://www.archiveteam.org/?diff=46014&oldid=0 |
| 02:03:36 | | Dallas (Dallas) joins |
| 02:12:31 | | XXLuigiMario (XXLuigiMario) joins |
| 02:37:17 | | anelki quits [Quit: servus!] |
| 03:30:13 | | XXLuigiMario quits [Client Quit] |
| 04:37:30 | | qw3rty_ joins |
| 04:41:04 | | qw3rty__ quits [Ping timeout: 240 seconds] |
| 04:48:13 | | HackMii_ (hacktheplanet) joins |
| 04:49:04 | | HackMii quits [Ping timeout: 240 seconds] |
| 04:49:22 | | HackMii_ is now known as HackMii |
| 04:59:49 | | etnguyen03 quits [Client Quit] |
| 05:01:22 | <purplebot> | ArchiveBot edited by Atphoenix (+849, New Usage Caveats section) just now -- https://www.archiveteam.org/?diff=46015&oldid=45719 |
| 05:34:22 | <purplebot> | ArchiveBot edited by Atphoenix (+77, /* Suggested things to focus on …) 22 minutes ago -- https://www.archiveteam.org/?diff=46016&oldid=46015 |
| 06:23:27 | | Domaloma joins |
| 06:24:36 | <Domaloma> | how do you do, fellow refu-jeevers |
| 06:32:23 | <OrIdow6> | Can't say I was a user of the site, just someone that read comics there occasionally, but I'm glad one of you found this channel |
| 06:33:51 | <OrIdow6> | Is there anything I can do for you Domaloma |
| 06:33:52 | <OrIdow6> | ? |
| 06:36:28 | <Lord_Nightmare> | apollo legend posted a suicide note video earlier today, and was found dead by police about an hour ago :( |
| 06:37:04 | <Lord_Nightmare> | I fed his twitter account (which did not link the video) to archivebot but what else needs archiving |
| 06:37:10 | <thuban> | just finished watching that, hadn't heard they found him |
| 06:37:17 | <thuban> | youtube channel at least |
| 06:37:43 | <thuban> | https://www.youtube.com/channel/UC5t8u4kpE0WvLxkPWIp8n-g |
| 06:38:14 | <Lord_Nightmare> | I'm not even sure how to archive that :( |
| 06:38:23 | <thuban> | tubeup? |
| 06:39:02 | <Lord_Nightmare> | i'm not familiar with that project |
| 06:39:12 | <thuban> | https://github.com/bibanon/tubeup |
| 06:39:16 | <thuban> | i'll do it |
| 06:40:27 | <thuban> | "*Dumping entire channels into Community Video is abusive and may get your account locked.* _Talk to the admins first before doing large uploads it's better to ask for guidence or help first than run afowl with the rules._" whoops |
| 06:42:59 | <thuban> | there are 57 total videos, should i email collections-service and/or info in advance or just pull the trigger now? |
| 06:44:30 | <Jake> | The channel has been archived at #youtubearchive but it doesn't have the video published today, which has since been removed. |
| 06:44:31 | <jodizzle> | fwiw, I've dumped a channel or two of about that size (for legitimate reasons) with no problems |
| 06:44:44 | <jodizzle> | dumped to IA I mean, thuban |
| 06:45:07 | <thuban> | hm ok, sounds like it doesn't need to be run again then. |
| 06:45:39 | <jodizzle> | The vids do get de-indexed from IA's search, but that's normal for youtube mirroring on IA |
| 06:45:47 | <thuban> | https://www.mediafire.com/file/i35vo3hcex0r3zu/Untitled.wmv/file the last video |
| 06:49:17 | <Domaloma> | to Orldow6, nothing much. I've just been scouting the net to see who's been doing an archive of SJ. I have a small group of people elsewhere and only very-very recently that I found out you people have done a much bigger job. |
| 06:50:15 | <OrIdow6> | 683.33 GiB total |
| 06:51:41 | <OrIdow6> | Data is here https://archive.org/details/archiveteam_smackjeeves?tab=collection , but you should be able to browse it in the Wayback Machine, e.g. https://web.archive.org/web/20201230003727/https://www.smackjeeves.com/discover/detail?titleNo=176093&articleNo=4 |
| 06:52:50 | <OrIdow6> | Basic approach was to try all comic and user IDs up to 350k |
| 06:56:29 | <OrIdow6> | I have heard of 3 or 4 (lost count) other efforts, which was yours? |
| 07:05:50 | <Domaloma> | That's quite impressive, and honestly much more than what we ended up attempting. Did you automate this? Or use a program/app/thingy. Most of my friends have only been using HakuNeko |
| 07:06:34 | <Domaloma> | in a discord server, we've mostly been selecting nostalgic sprite comics that we know haven't been moved to other places - though archiving things already archived doesn't hut at all. I know someone used a python script to save comments aswell, but I forgot how that works. |
| 07:12:09 | <Domaloma> | in a discord server, we've mostly been selecting nostalgic sprite comics that we know haven't been moved to other places - though archiving things already archived doesn't hurt at all. I know someone used a python script to save comments aswell, but I forgot how that works. |
| 07:12:16 | <Domaloma> | Woops. |
| 07:12:40 | <OrIdow6> | This used the ArchiveTeam warrior/"DPoS" software, a general website-saving system used here |
| 07:13:03 | <OrIdow6> | The site-specific script was here https://github.com/ArchiveTeam/smackjeeves-grab , and this ran for each comic/user |
| 07:13:25 | <Domaloma> | Ah, I see. I was looking into other methods and found out about wget and HTTrack. Were those methods useful/would you recommend them, or did they not work? |
| 07:15:58 | <OrIdow6> | This project used a wget fork, as do most such ArchiveTeam projects; from my understanding HTTrack would have been difficult to scale to this much data, though in any case I don't think it produces WARC files and therefore would not have been suitable here |
| 07:16:10 | <OrIdow6> | If you are looking to make a personal copy of websites, I think both work well |
| 07:16:53 | <Domaloma> | Right, right. Thanks. I'm honestly an amateur at this, it's my first time attempting to archive a website (especially since I don't have Linux), so I'll look more into it... tommorow. Yeah. |
| 07:18:28 | <OrIdow6> | People usually seem satisfied with HTTPTrack |
| 07:19:34 | <OrIdow6> | But I have never used it |
| 07:21:33 | <OrIdow6> | Myself |
| 07:23:18 | <atphoenix> | I've use HTTrack to pretty good effect for some personal stuff, but not recently. It also needs guidance on what to ignore (like AB). |
| 07:23:54 | <Domaloma> | Ah, okay. |
| 07:44:28 | | ragu_ joins |
| 07:47:44 | | ragu__ quits [Ping timeout: 240 seconds] |
| 07:48:00 | | ragu__ joins |
| 07:48:10 | | ragu__ is now authenticated as ragu |
| 07:50:16 | | ragu_ quits [Ping timeout: 252 seconds] |
| 07:58:16 | | thuban quits [Quit: What price Glory ?] |
| 09:01:43 | | HackMii quits [Ping timeout: 240 seconds] |
| 09:19:45 | <brad> | I know of one guy who was trying to use HTTrack to archive the SWRPG forums that are being shut down in February, but I believe he kept getting banned by the site. I’ve got my own wget based archive I’m trying on one machine, and I’m trying to get grab-site installed on another machine so that it can at least try to download with multiple threads in parallel to a WARC. |
| 09:21:17 | <brad> | If I can get a container based installation of grab-site working, then I might be able to take that basic configuration to multiple different systems that I might run. |
| 09:22:55 | <brad> | One copy might run on my local machines in the house, and other copies might run on service providers outside of the house. I might try AWS Fargate just to get some experience with it, but more likely to run longer-term on something less expensive. |
| 09:25:22 | | hooway joins |
| 09:27:29 | | hooway_ joins |
| 09:30:22 | | thuban joins |
| 09:30:24 | <@kiska> | You'll not use if you like having an intact wallet. After their 14 cents per GB bandwidth charge |
| 09:30:24 | | HackMii (hacktheplanet) joins |
| 09:31:24 | | hooway quits [Ping timeout: 240 seconds] |
| 09:32:00 | <thuban> | https://www.youtube.com/channel/UCqNJ_TkbZQQfgI4l8QfagGA more apollo legend-related content |
| 09:47:24 | | Arcorann quits [Ping timeout: 240 seconds] |
| 09:47:41 | <betamax> | Ryz: are you around? need some help archiving another board of the sploder forums and can't figure out the best way to do it without re-archiving all the other boards |
| 09:50:19 | | Arcorann (Arcorann) joins |
| 09:55:12 | <betamax> | the problem is that the URLs are of the form <baseURL>/index.php/topic,0123456.0.html , so the specific board isn't in the URL (and therefore I don't see a way to force archivebot to only download a single board) |
| 10:05:33 | | thuban quits [Client Quit] |
| 10:09:40 | <betamax> | Ryz: fyi since I don't see a way around the above issue (and since the forums are closing tomorrow), I've re-queued them in archivebot |
| 10:11:47 | <betamax> | I used all the same ignoresets as you did, but increased concurrency to 12 since there are an extra ~47,000 topic to be archived (~580,000 posts) |
| 10:13:28 | | thuban joins |
| 10:14:44 | | HackMii quits [Ping timeout: 240 seconds] |
| 10:16:35 | | HackMii (hacktheplanet) joins |
| 10:29:38 | | HackMii quits [Client Quit] |
| 10:29:58 | | HackMii (hacktheplanet) joins |
| 10:57:19 | <OrIdow6> | brad: "the SWRPG forums that are being shut down in February" - can you give details? |
| 11:04:04 | <OrIdow6> | Sounds like the sort of thing ArchiveTeam is interested in |
| 11:07:04 | <jodizzle> | It's a site we're running in AB, so we already know about it |
| 11:08:12 | <jodizzle> | Unfortunately not running too successfully because of bans, though |
| 11:08:30 | <@kiska> | Oh we are most interested in that |
| 11:08:37 | <@kiska> | We can run a project if needed |
| 11:09:04 | <@kiska> | Is this on deathwatch? |
| 11:09:10 | <OrIdow6> | Oh, apparently I can't search |
| 11:11:01 | <jodizzle> | Ooh wait, my apologies, I think brad is referring to a different forum. I wasn't paying attention to the acronym: 'SWRPG' |
| 11:11:17 | <@kiska> | brad: Can we have a link to this forum? |
| 11:12:00 | <jodizzle> | kiska: The other forum that we're having trouble with currently in AB is https://community.fantasyflightgames.com/. And yes, it's on deathwatch. |
| 11:12:12 | <@kiska> | I see... |
| 11:12:40 | <jodizzle> | (also shutting down in February) |
| 11:12:50 | <@kiska> | Argh |
| 11:15:49 | <@kiska> | How harsh is the rate limit for that forum? |
| 11:16:16 | <@kiska> | Also is it perimeterx? |
| 11:16:32 | <OrIdow6> | I wonder if he could actually be referring the first thing Google gives me for "SWRPG forums"... https://community.fantasyflightgames.com/forum/306-star-wars-edge-of-the-empire-rpg/ |
| 11:18:44 | <@kiska> | Possibly |
| 11:18:51 | <@kiska> | Lets wait on that info |
| 11:19:09 | <OrIdow6> | Yes |
| 11:20:00 | <@kiska> | Feels like Xenforo or Invision |
| 11:20:03 | <OrIdow6> | A luxury we have for something shutting down in February, whatever that "something" is |
| 11:22:09 | <@kiska> | Ah we do have something for Invision https://github.com/ArchiveTeam/wget-lua-forum-scripts |
| 11:22:12 | <@kiska> | Not sure how upto date it is |
| 11:23:58 | <jodizzle> | I'm not totally sure how the rate limit on the forum we're currently crawling works, but it seems like after crawling it for a day, we get hit by day-ish of 403s |
| 11:43:48 | <OrIdow6> | Doesn't look like those scripts are for a distributed crawl |
| 12:02:43 | | honno joins |
| 12:14:33 | | spirit quits [Client Quit] |
| 12:34:12 | | Nullificator joins |
| 12:40:03 | <Nullificator> | hi |
| 12:46:58 | | Nullificator quits [Remote host closed the connection] |
| 13:03:09 | | taufik joins |
| 13:03:59 | <taufik> | Hello! I was told from Wikimedia that Archive Team can help with something. |
| 13:04:04 | <@arkiver> | hi |
| 13:04:54 | <taufik> | There is this local newspaper that is closing. I'm currently archiving it manually in Internet Archive. |
| 13:06:28 | <taufik> | This is the newspaper http://www.newsabahtimes.com.my/ |
| 13:07:05 | <taufik> | Few of the last remaining newspaper that offers a column in an indigenous language here. |
| 13:08:42 | <taufik> | Maybe Archive Team could tell me any faster way to archive this or help archive this |
| 13:09:00 | <@arkiver> | when is it going away? |
| 13:09:47 | <@arkiver> | ah dec 31 |
| 13:09:55 | <@arkiver> | its in archivebot |
| 13:09:58 | <@arkiver> | archivebot.com |
| 13:10:09 | <@arkiver> | (top one) |
| 13:10:19 | <taufik> | I'm just not sure when the website will be closed |
| 13:10:53 | <@arkiver> | "It was earlier scheduled to close on Dec 20, but Gusti said the company wanted to finish its newsprint before closing tomorrow (Dec 31)." |
| 13:10:58 | <@arkiver> | http://www.newsabahtimes.com.my/nstweb/fullstory/41705 |
| 13:11:38 | <taufik> | Yup |
| 13:12:17 | <taufik> | I've already told to some Wikipedians here to help archive this using Internet Archive |
| 13:12:38 | <@arkiver> | they should just come to us next time |
| 13:12:44 | <@arkiver> | archiving page by page is a pain |
| 13:12:54 | <@arkiver> | using save page now on web.archive.org I assume |
| 13:12:54 | <taufik> | Yeah :( |
| 13:13:06 | <@arkiver> | we'll put it in wayback machine as well |
| 13:13:21 | <taufik> | Is there anything you can do? |
| 13:13:39 | <@arkiver> | its in archivebot |
| 13:13:41 | <@arkiver> | see archivebot.com |
| 13:13:53 | | honno quits [Client Quit] |
| 13:16:54 | | taufik quits [Remote host closed the connection] |
| 13:19:26 | | ragu_ joins |
| 13:20:15 | | taufik37 joins |
| 13:21:07 | <Arcorann> | It doesn't look like archivebot is getting the full archives |
| 13:22:04 | <Arcorann> | http://www.newsabahtimes.com.my/nstweb/archives <-- page only shows one day by default, need to submit a form to get more |
| 13:22:51 | <Arcorann> | The URL format seems easy to check though (e.g. http://www.newsabahtimes.com.my/nstweb/fullstory/41688) |
| 13:23:01 | | ragu__ quits [Ping timeout: 252 seconds] |
| 13:23:11 | <OrIdow6> | I noticed that, see #archivebot |
| 13:26:32 | | taufik joins |
| 13:36:41 | | taufik quits [Ping timeout: 244 seconds] |
| 13:39:38 | | taufik37 quits [Remote host closed the connection] |
| 13:43:14 | | taufik joins |
| 13:49:07 | <OrIdow6> | So it doesn't get lost in the bot noise: can someone put https://transfer.notkiska.pw/TWJwc/www.newsabahtimes.com.my_pages_not_already_in_wbm into AB as an ao <, preferably at max concurrency (which I don't think I can do, lacking voice)? |
| 13:49:45 | <betamax> | OrIdow6: I can |
| 13:51:44 | <OrIdow6> | Thank you betamax |
| 13:52:47 | <betamax> | I set it to use 10 workers, I don't know if there actually is a max, but if it looks to be going too slow after an hour or so I (or someone) can increase it |
| 13:58:18 | <OrIdow6> | I didn't know it be set above 9 |
| 13:59:00 | <OrIdow6> | *it could be |
| 14:04:53 | | Datechnom5 (Datechnom) joins |
| 14:06:04 | | Datechnom quits [Ping timeout: 240 seconds] |
| 14:06:04 | | Datechnom5 is now known as Datechnom |
| 14:16:44 | | Lord_Nightmare quits [Quit: ZNC - http://znc.in] |
| 14:19:47 | | taufik46 joins |
| 14:23:42 | | taufik quits [Ping timeout: 244 seconds] |
| 14:30:05 | | etnguyen03 (etnguyen03) joins |
| 14:48:24 | | Arcorann quits [Ping timeout: 240 seconds] |
| 15:12:16 | | taufik46 quits [Ping timeout: 244 seconds] |
| 15:12:17 | | Ajay quits [Remote host closed the connection] |
| 15:23:27 | | HP_Archivist (HP_Archivist) joins |
| 15:49:12 | | systwi_ is now known as systwi |
| 16:06:38 | | spirit joins |
| 16:36:23 | <purplebot> | ArchiveBot edited by Atphoenix (+162, /* Usage, Dashboards, and Completed …) just now -- https://www.archiveteam.org/?diff=46017&oldid=46016 |
| 17:11:16 | <@JAA> | betamax: There isn't a limit to that setting, but there's a hard limit of 6 connections per host (= protocol + domain + port). |
| 17:24:27 | <@kiska> | Well anyway time for me to try my hand at writing FFG forums |
| 17:34:14 | <@JAA> | The images.nga.gov AB jobs finished this morning. Apart from possibly some errors that weren't retried successfully, it should all be done. |
| 18:18:26 | | ragu__ joins |
| 18:21:47 | | ragu_ quits [Ping timeout: 265 seconds] |
| 18:54:59 | | ragu__ is now authenticated as ragu |
| 19:47:06 | | DogsRNice (Webuser299) joins |
| 20:01:49 | <brad> | @OrIdow6: Yes, I can confirm that @jodizzle is correct. There is an ArchiveBot job that has been created to archive the SWRPG forums, with ident 5l4qk9brqjo7emnpmuugp7owq. |
| 20:02:28 | <brad> | But this job has been on-again/off-again. It’s currently paused. |
| 20:27:26 | <brad> | I’ve got my own private wget archive that I’m running, and I’m working on trying to get my own private copy of grab-site up and running so that I can create a proper WARC that I could hand over to the IA and WBM. |
| 20:28:04 | <jodizzle> | Okay, thanks. But keep in mind that the WARCs you generate could go to IA, but they probably wouldn't get into the WBM |
| 20:28:14 | <brad> | But we definitely seem to be having problems in getting everything from the SWRPG forums. |
| 20:29:33 | <brad> | I’m okay with my WARCs not getting into the WBM. I want to create them for the benefit of anyone who wants them, as well as for the experience of creating them. What they do with them is not something I can control. |
| 20:32:06 | <brad> | Also speaking entirely for myself, I would love to see what a distributed project could do for archiving the SWRPG forums, and what I might be able to do to help that process along. |
| 20:34:58 | <brad> | There is a community of people on the SWRPG Discord that would like to find a way to self-host our own copy of those forums, but we are pretty clueless on that end. Any pointers that could be provided would be appreciated, I’m sure. |
| 20:36:41 | <@kiska> | Ok now that we have confirmed which forum it is, I guess I could go and create a repo for a project since this appears to require one |
| 20:37:45 | <brad> | Longer-term, I would like to be able to do what I can to positively contribute to the ArchiveBot and the ArchiveTeam in general. But I also realize that I need to earn that trust. |
| 20:41:03 | <@kiska> | So for this forum, what is the highest thread id? |
| 20:41:06 | <brad> | @kiska: I would certainly appreciate that, and I would love to learn what I can do to further help with this project. Pointers to FAQs, web pages, repos, etc... would be most welcome. |
| 20:41:27 | <@kiska> | Just to confirm it is this one right? https://community.fantasyflightgames.com/ |
| 20:41:51 | <brad> | Yup, that’s the one getting shut down in February. |
| 20:41:52 | <@kiska> | I suggest you browse https://www.archiveteam.org/ |
| 20:42:22 | <@kiska> | I'll write code, but its unlikely to start before the middle of January since people are still posting and making threads, etc |
| 20:43:43 | <brad> | I have browsed that site, which lead to these irc channels, among other things. |
| 20:44:19 | <@kiska> | There should be a couple pages relating to the warrior and infras |
| 20:44:32 | <brad> | I can confirm that members of the greater FFG Community Forums will probably continue to try to post there will into 2021 and probably even 2022. |
| 20:45:20 | <brad> | Almost all of the people on those forums are more interested in playing the games than they are the technology behind the community site that allows them to post threads. |
| 20:45:59 | <brad> | So, this will be a game of catch-up, right to the moment when the forums get completely shut down. |
| 21:20:22 | <purplebot> | Endomondo edited by Tech234a (+116, Add source code and tracker link) just now -- https://www.archiveteam.org/?diff=46018&oldid=45893 |
| 21:42:23 | <purplebot> | Smack Jeeves edited by Wickedplayer494 (-5, From what I've heard in -BS, this …) just now -- https://www.archiveteam.org/?diff=46019&oldid=45995 |
| 22:07:13 | | Lord_Nightmare (Lord_Nightmare) joins |
| 22:25:23 | <purplebot> | Current Projects edited by Wickedplayer494 (+655, + .EU, Flash domains to scripts, …) just now -- https://www.archiveteam.org/?diff=46020&oldid=45992 |
| 22:31:22 | <purplebot> | Flash edited by Wickedplayer494 (+477, Infobox and navbox) just now -- https://www.archiveteam.org/?diff=46021&oldid=45884 |
| 22:35:24 | | lukash7 quits [Ping timeout: 240 seconds] |
| 22:36:17 | | lukash7 joins |
| 22:43:22 | <purplebot> | Endomondo edited by Wickedplayer494 (+22, In progress, and navbox) 9 minutes ago -- https://www.archiveteam.org/?diff=46022&oldid=46018 |
| 23:00:03 | | HackMii quits [Remote host closed the connection] |
| 23:00:26 | | HackMii (hacktheplanet) joins |
| 23:09:10 | | wickedplayer494 quits [Remote host closed the connection] |
| 23:27:46 | | wickedplayer494 joins |
| 23:27:49 | | wickedplayer494 is now authenticated as wickedplayer494 |