| 00:02:24 | | Ka is now authenticated as Ka |
| 00:15:34 | | HackMii quits [Remote host closed the connection] |
| 00:16:41 | | HackMii (hacktheplanet) joins |
| 00:26:51 | | Doranwen is now authenticated as Doranwen |
| 00:37:41 | | HackMii quits [Remote host closed the connection] |
| 00:38:31 | | HackMii (hacktheplanet) joins |
| 01:01:58 | <SketchTheCow> | Hi, I'm back. |
| 01:02:04 | <SketchTheCow> | If there's something that needs me, let me know. |
| 01:04:49 | <@JAA> | Hey Jason. Uh something's going on with teamarchive1 and FOS. No longer reporting to Telegraf since a few days, and I think the uploader on 1 broke a while ago as its disk space never dropped anymore. |
| 01:15:37 | | AeonG joins |
| 01:15:48 | <atphoenix> | I think the ArchiveBot description in WBM needs editing to point to the wiki instead of to EFnet |
| 01:42:09 | <SketchTheCow> | Sure, let me check. |
| 01:42:35 | <SketchTheCow> | So, first, FOS has 4.4tb free. So no panic there. |
| 01:43:02 | <SketchTheCow> | TEAMARCHIVE1 has 1.9 free |
| 01:43:43 | <SketchTheCow> | Both think they're running telegraf |
| 01:44:21 | <SketchTheCow> | What's the service name for telegraf again? I'll stop and start. |
| 01:45:09 | <SketchTheCow> | systemctl restart telegraf |
| 01:45:35 | <SketchTheCow> | Done on both. |
| 01:46:08 | <SketchTheCow> | So, if this works, great. If it fails, it's because someone at IA changed the network access. |
| 01:46:12 | <SketchTheCow> | And I'll have to discuss it with them. |
| 01:48:37 | <SketchTheCow> | OK, looks strongly like a blockage. |
| 01:50:55 | <@JAA> | Doesn't look like it works. Last datapoint was on the 12th at 11:40 UTC. |
| 01:53:02 | <@JAA> | And yeah, no space issues so far, but ta1 dropped to near-zero before and then just climbed to 18 % and stalled there nearly two weeks ago. |
| 01:54:20 | <@JAA> | https://atdash.meo.ws/d/000000058/telegraf-detail?viewPanel=21&orgId=1&from=1606689287342&to=1607371773077&var-includeall=false&var-user=sketchcow&var-host=teamarchive1.fnf.archive.org |
| 01:54:48 | | britmob joins |
| 01:55:36 | <@JAA> | NB, nothing's uploading to ta1 since then because pipeline hardware issues and stuff. |
| 01:57:17 | | britm0b quits [Ping timeout: 252 seconds] |
| 01:58:52 | | britm0b joins |
| 02:00:01 | | Dallas quits [Client Quit] |
| 02:00:02 | | britmob quits [Ping timeout: 252 seconds] |
| 02:01:01 | | britmob joins |
| 02:03:04 | | britm0b quits [Ping timeout: 240 seconds] |
| 02:03:43 | | Dallas (Dallas) joins |
| 02:03:58 | | britm0b joins |
| 02:07:44 | | britmob quits [Ping timeout: 240 seconds] |
| 02:08:37 | | Ka quits [Remote host closed the connection] |
| 02:08:50 | | britm0b quits [Ping timeout: 252 seconds] |
| 02:09:19 | <SketchTheCow> | I just had our guy check. |
| 02:09:29 | <SketchTheCow> | Ports are all open on our side. |
| 02:09:43 | <SketchTheCow> | Packets absolutely leaving heading to atdash.meo.ws |
| 02:10:34 | <@JAA> | Huh |
| 02:10:41 | <@JAA> | Fusl? |
| 03:14:32 | <@Fusl_> | grep telegraf /var/log/syslog |
| 03:30:09 | <SketchTheCow> | Dec 15 03:29:33 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:33Z W! [outputs.influxdb] Metric buffer overflow; 138 metrics have been dropped |
| 03:30:12 | <SketchTheCow> | Dec 15 03:29:33 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:33Z E! [outputs.influxdb] When writing to [http://atdash.meo.ws:8086]: 401 Unauthorized: authorization failed |
| 03:30:17 | <SketchTheCow> | Dec 15 03:29:33 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:33Z E! [agent] Error writing to outputs.influxdb: could not write any address |
| 03:30:20 | <SketchTheCow> | Dec 15 03:29:41 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:41Z W! [outputs.influxdb] Metric buffer overflow; 276 metrics have been dropped |
| 03:30:23 | <SketchTheCow> | Dec 15 03:29:41 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:41Z E! [outputs.influxdb] When writing to [http://atdash.meo.ws:8086]: 401 Unauthorized: authorization failed |
| 03:30:27 | <SketchTheCow> | Dec 15 03:29:41 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:41Z E! [agent] Error writing to outputs.influxdb: could not write any address |
| 03:30:30 | <SketchTheCow> | Dec 15 03:29:48 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:48Z W! [outputs.influxdb] Metric buffer overflow; 138 metrics have been dropped |
| 03:30:33 | <SketchTheCow> | Dec 15 03:29:49 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:49Z E! [outputs.influxdb] When writing to [http://atdash.meo.ws:8086]: 401 Unauthorized: authorization failed |
| 03:30:37 | <SketchTheCow> | Dec 15 03:29:49 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:49Z E! [agent] Error writing to outputs.influxdb: could not write any address |
| 03:30:40 | <SketchTheCow> | Dec 15 03:29:54 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:54Z W! [outputs.influxdb] Metric buffer overflow; 138 metrics have been dropped |
| 03:30:43 | <SketchTheCow> | Dec 15 03:29:55 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:55Z E! [outputs.influxdb] When writing to [http://atdash.meo.ws:8086]: 401 Unauthorized: authorization failed |
| 03:30:47 | <SketchTheCow> | Dec 15 03:29:55 teamarchive1.fnf.archive.org telegraf[686745]: 2020-12-15T03:29:55Z E! [agent] Error writing to outputs.influxdb: could not write any address |
| 03:32:48 | | taka80 quits [Ping timeout: 244 seconds] |
| 03:51:35 | <SketchTheCow> | So there you go |
| 04:00:42 | <@Fusl> | SketchTheCow: can you dm me the password field from your telegraf.conf? |
| 04:10:46 | | lukash quits [Quit: The Lounge - https://thelounge.chat] |
| 04:13:25 | | lukash (lukash) joins |
| 04:54:00 | | qw3rty_ joins |
| 04:57:24 | | qw3rty__ quits [Ping timeout: 240 seconds] |
| 05:12:20 | | wickedplayer494 quits [Ping timeout: 252 seconds] |
| 06:04:02 | | nerdguy1138 quits [Ping timeout: 252 seconds] |
| 06:19:28 | | godane2 joins |
| 06:21:44 | | godane1 quits [Ping timeout: 240 seconds] |
| 06:41:17 | | wickedplayer494 joins |
| 06:41:20 | | wickedplayer494 is now authenticated as wickedplayer494 |
| 06:53:05 | | pie_ joins |
| 07:10:27 | | flirora joins |
| 07:10:38 | | flirora is now authenticated as flirora |
| 07:11:24 | <flirora> | so I took a chance at appealing to atwiki and it seems that the support team is willing to give me a second chance, but I think it's better if I go away from archiving for a while |
| 07:11:36 | <flirora> | chan ops: please ban me for the next 3 months |
| 07:12:45 | | flirora leaves |
| 08:37:11 | | hooway joins |
| 08:38:19 | | taka joins |
| 08:38:57 | | taka quits [Remote host closed the connection] |
| 08:50:09 | | godane2 quits [Client Quit] |
| 08:59:09 | | c00k13 joins |
| 09:13:22 | | nerdguy1138 (nerdguy1138) joins |
| 09:22:34 | | AeonG_ joins |
| 09:24:36 | | AeonG quits [Ping timeout: 252 seconds] |
| 09:43:44 | <OrIdow6> | What's the context of that? |
| 09:47:51 | | ddd123427 joins |
| 09:50:05 | <ddd123427> | Hello. |
| 09:50:23 | <ddd123427> | You will need to log in to your account to view the pages of videos that have been deleted in the past. |
| 09:50:52 | <ddd123427> | There are approximately 38000000 video pages in total. |
| 09:51:02 | <OrIdow6> | ddd123427: What website are you talking about? |
| 09:51:23 | <ddd123427> | https://www.nicovideo.jp/ |
| 09:52:07 | <OrIdow6> | What is being deleted? The entire website, or just a piece of it? |
| 09:52:18 | <purplebot> | Coronavirus/Notable deaths edited by Ka (+227, /* Table */ fixing format) just now -- https://www.archiveteam.org/?diff=45902&oldid=45237 |
| 09:52:18 | <purplebot> | Deathwatch edited by Ka (+661, /* 2020 */) just now -- https://www.archiveteam.org/?diff=45903&oldid=45900 |
| 09:54:31 | <ddd123427> | 過去に動画データが削除された動画のメタデータが削除されます。 |
| 09:54:36 | <ddd123427> | Metadata for videos whose video data has been deleted in the past will be deleted. |
| 09:55:55 | <OrIdow6> | When will the deletion happen? |
| 09:57:03 | <ddd123427> | The deletion will happen by spring 2021. |
| 09:59:09 | <OrIdow6> | https://blog.nicovideo.jp/niconews/141893.html - shutdown notice |
| 10:01:16 | <OrIdow6> | Page claims (to quote Google Translate) "The specific schedule will be announced on NicoNico Info as soon as it is decided." |
| 10:03:26 | <OrIdow6> | ddd123427: Is the site very slow? For example, does it take more than a few seconds to load a page? Does the site stop working if a large number of people use it at the same time? |
| 10:06:03 | | Arcorann (Arcorann) joins |
| 10:07:49 | <OrIdow6> | I have some things to do, so I'll just say it all now: |
| 10:08:30 | <OrIdow6> | - If the site is fast enough, ArchiveTeam probably won't get to it until spring. If or when they announce the exact shutdown dates, you should come on here and tell us. |
| 10:08:34 | | ddd123427 quits [Ping timeout: 244 seconds] |
| 10:08:40 | <OrIdow6> | Oh, never mind then. |
| 10:09:42 | <OrIdow6> | If they come back and someone else has the opportunity to be the receptionist... tell them about Deathwatch |
| 10:10:55 | <OrIdow6> | And maybe give the rundown on how to add to it, i.e. you need an account & a link to the shutdown notice is preferable |
| 10:12:13 | <thuban> | we should do some digging and figure out what the "My Memory" function (also being deleted) is--could easily be either private or public |
| 10:13:37 | | ddd1234 joins |
| 10:14:40 | <ddd1234> | The site is fast. It's not often that the site stops with a lot of people using it at the same time. |
| 10:14:42 | <ddd1234> | However, continuous access in a short period of time is not possible. |
| 10:16:10 | <thuban> | ddd1234: do you know what the "My Memory" feature is? is it also something that can or should be archived? |
| 10:18:01 | <Arcorann> | What site is this, and has it been added to Deathwatch? |
| 10:18:51 | <thuban> | Arcorann: nicovideo.jp (not the whole site, just metadata pertaining to already-deleted videos); no, but i'll get to it tomorrow if no one else does |
| 10:20:02 | <Arcorann> | I see |
| 10:20:35 | <ddd1234> | "My Memory" is a feature that allows you to record the comments at that point in time. It does not need to be archived. |
| 10:23:34 | <Arcorann> | When you look at that don't forget to update the page the wiki already has if necessary--> https://archiveteam.org/index.php?title=Niconico |
| 10:28:20 | <thuban> | the good news is that it looks like video ids are sequential. the bad news is that there are a lot of deleted videos. i'll check out the api at some point |
| 10:34:16 | <ddd1234> | Arcorann: ok |
| 10:37:55 | <thuban> | ddd1234: i believe that was for me, although you're welcome to edit the wiki if you wish! thanks for letting us know about this. |
| 10:41:10 | <ddd1234> | thuban: Oh, I see. I don't have editing privileges, so please edit. |
| 10:49:45 | <Arcorann> | You might as well create an account on the wiki, it's free |
| 10:53:32 | | @Fusl quits [Ping timeout: 252 seconds] |
| 10:59:07 | | AeonG__ joins |
| 11:01:04 | | AeonG_ quits [Ping timeout: 240 seconds] |
| 11:03:40 | | AeonG__ quits [Read error: Connection reset by peer] |
| 11:05:03 | | AeonG__ joins |
| 11:06:50 | | Fusl (Fusl) joins |
| 11:06:50 | | @ChanServ sets mode: +o Fusl |
| 11:08:38 | | qw3rty_ quits [Read error: Connection reset by peer] |
| 11:09:15 | | qw3rty_ joins |
| 11:10:08 | | seatsea quits [Quit: Ping timeout (120 seconds)] |
| 11:10:18 | | seatsea joins |
| 11:23:47 | | @Fusl quits [Ping timeout: 252 seconds] |
| 11:33:04 | <OrIdow6> | ddd1234: What do you mean, "continuous access in a short period of time is not possible"? |
| 11:33:26 | <OrIdow6> | And another useful wiki page: https://www.archiveteam.org/index.php?title=Deathwatch |
| 11:33:58 | <OrIdow6> | This is a list of shutdowns that ArchiveTeam knows about |
| 11:41:02 | <ddd1234> | OrIdow6: If you access the site many times in a short period of time, an error message will be displayed. It is necessary to allow an interval for access. |
| 11:41:20 | <ddd1234> | https://dic.nicovideo.jp/a/%E7%9F%AD%E6%99%82%E9%96%93%E3%81%A7%E3%81%AE%E9%80%A3%E7%B6%9A%E3%82%A2%E3%82%AF%E3%82%BB%E3%82%B9%E3%81%AF%E3%81%94%E9%81%A0%E6%85%AE%E3%81%8F%E3%81%A0%E3%81%95%E3%81%84 |
| 11:42:03 | | Arcorann_ joins |
| 11:45:14 | | Arcorann quits [Ping timeout: 252 seconds] |
| 11:50:14 | <VerifiedJ> | My download of archiviolastampa.it is almost finished, just downloading items with missing images. |
| 11:52:48 | <VerifiedJ> | ~69,500 files totaling 916GB |
| 12:01:51 | | Fusl (Fusl) joins |
| 12:01:51 | | @ChanServ sets mode: +o Fusl |
| 12:11:40 | | @Fusl quits [Client Quit] |
| 12:21:39 | | Fusl (Fusl) joins |
| 12:21:39 | | @ChanServ sets mode: +o Fusl |
| 12:37:53 | | ddd1234 quits [Ping timeout: 244 seconds] |
| 13:09:58 | | dd joins |
| 13:10:48 | | dd leaves |
| 13:15:26 | <betamax> | Sploder.com, a one-popular flash-based game making website, will be shutting down around the end of December |
| 13:15:29 | <betamax> | there are around 4.5 million user-made games, the forums, member profiles, member groups, and the game makers themselves |
| 13:16:02 | <betamax> | I don't know much about flash archiving, but as a former user of the site (~10 years ago), I'd like to see it preserved in some form |
| 13:16:33 | <OrIdow6> | Put it on the Deathwatch |
| 13:18:46 | <OrIdow6> | So, update on where I am: working on .eu domains, next will do Smackjeeves, I think |
| 13:20:34 | <OrIdow6> | Unless someone else does it, obviously |
| 13:21:57 | <OrIdow6> | Anyone started work on https://www.archiveteam.org/index.php?title=Freshlive ? |
| 13:22:43 | <OrIdow6> | 3 days until that shuts down |
| 13:23:10 | <betamax> | OrIdow6: added (pending moderation) |
| 13:23:19 | <OrIdow6> | Ok, good |
| 13:23:45 | <betamax> | I will try and make a separate page with the site structure, but others familuar with flash will be needed to work out how to go about the actual archiving process |
| 13:24:49 | <OrIdow6> | Hopefully it's simple enough |
| 13:25:25 | | ugche80 (ugche) joins |
| 13:25:29 | <OrIdow6> | I don't know much about Flash, that's for sure; all I can do is watch what it does in Wireshark |
| 13:25:34 | | ugche80 leaves |
| 13:25:45 | <OrIdow6> | Sanqui: I see you did some work on Freshlive, what's the status of that? |
| 13:26:25 | | ugche (ugche) joins |
| 13:26:27 | <Sanqui> | OrIdow6: no clue frankly, I just helped ragu__ put their m3u8 videos in archivebot and it ran through |
| 13:26:55 | <OrIdow6> | Oh |
| 13:27:07 | <Sanqui> | also there's two articles for some reason, they need merging :( |
| 13:30:19 | <OrIdow6> | Hm |
| 13:32:52 | <OrIdow6> | That list only has 7832 lines, maybe it'll be possible to do with AB or something like that |
| 13:33:01 | <Sanqui> | I already ran that list through archivebot |
| 13:33:06 | <Sanqui> | it's archived |
| 13:34:28 | <OrIdow6> | I mean, the videos themselves |
| 13:34:38 | <OrIdow6> | Transport segments |
| 13:34:52 | <OrIdow6> | That is |
| 13:34:57 | <Sanqui> | well, yes, they were downloaded |
| 13:35:00 | <Sanqui> | it was a few gb? |
| 13:35:58 | <OrIdow6> | Which job? |
| 13:39:11 | <OrIdow6> | https://archive.fart.website/archivebot/viewer/job/bulhc was the m3u8s, got up to about 800mb |
| 13:41:20 | <OrIdow6> | https://archive.fart.website/archivebot/viewer/job/8mdqc got the website, and went up to ~107 GB, but I don't see any of the .ts files in question in the meta warc |
| 13:42:15 | <Sanqui> | ok, sorry, no clue |
| 13:42:27 | <Sanqui> | if you can get another list I'd be happy to put it in ab |
| 13:42:33 | <OrIdow6> | Maybe I'm skipping something over |
| 13:43:03 | <OrIdow6> | Ok, I'll see about that tomorrow |
| 13:43:19 | <VerifiedJ> | archiviolastampa.it is now offline with a message saying it will be available again from February 15, 2021. |
| 13:43:27 | <OrIdow6> | (Again, if it doesn't turn out that I'm missing something obvious) |
| 13:43:46 | <Sanqui> | nah sorry I really don't know much about that project idk why I put myself as leader |
| 13:43:52 | <Sanqui> | maybe because I knew something when I did put it but lol |
| 13:47:51 | <OrIdow6> | Whoever decides the direction of a canoe, is still the captain |
| 13:47:53 | <OrIdow6> | No worries |
| 13:48:07 | <OrIdow6> | Thank you |
| 13:48:18 | <Sanqui> | in that case you have my full authorization to steer |
| 13:48:20 | <Sanqui> | :P |
| 13:49:26 | <Sanqui> | in news of what I DO know though, I'm working on a new tool for medium-scale Discord server archival operations, interested people should join #discard for updates & fruitful discussion |
| 13:49:41 | | ugche leaves |
| 14:21:24 | | Arcorann_ quits [Ping timeout: 240 seconds] |
| 16:03:24 | | xit quits [Ping timeout: 240 seconds] |
| 16:03:24 | | lukash quits [Read error: Connection reset by peer] |
| 16:03:30 | | lukash7 joins |
| 16:03:44 | | mupf quits [Ping timeout: 240 seconds] |
| 16:04:05 | | xit joins |
| 16:04:42 | | mupf joins |
| 16:18:16 | <betamax> | I'm trying to upload a file to the wiki, and I get the error "Internal error: Server failed to store temporary file." |
| 16:19:05 | <betamax> | (this isn't important, just trying to add logo / image for the wiki page for Sploder I've just created) |
| 16:42:55 | | honno joins |
| 16:43:48 | <@JAA> | I still think that's an interference with the moderation extension. |
| 16:44:15 | <@JAA> | betamax: Try again. |
| 16:45:01 | <honno> | Ahh sorry I used the wrong chat heh, uh JAA you got any ideas on my q? I'll just post what I said before here |
| 16:45:19 | <purplebot> | Deathwatch edited by Betamax (+216, /* 2020 */ Added Sploder to 2020 …) just now -- https://www.archiveteam.org/?diff=45904&oldid=45903 |
| 16:45:19 | <purplebot> | Sploder created by Betamax (+6439, New page for Sploder, closing December …) just now -- https://www.archiveteam.org/?diff=45905&oldid=0 |
| 16:46:02 | <@JAA> | No worries. (Context: #archiveteam) |
| 16:46:06 | <honno> | Bit confused on how one would deal with the files that result from a megawarc-factory job (specific example I'm interested in: https://archive.org/details/archiveteam_gamemaker). I get how to use warcs individually (e.g. set up a pywb session), but I'm not sure how I should proceed to locally run an instance of megawarcs. I get that there's the megawarc python script that can "restore" megawarcs from the |
| 16:46:06 | <honno> | .warc.gz/.tar.json.gz files in each of the collection items, and this results in a tar file. I'm currently doing this for each folder locally but wondering once it's all done, how can I use the resulting tars? Do they get merged somehow? Or are restoring the megawarcs actually pointless if I want to use the archive? |
| 16:46:30 | <@JAA> | I've never used megawarc restores, but I think it produces a tar of the WARCs that went into one megawarc. |
| 16:47:03 | <@JAA> | But the fundamental question is: what are you trying to do exactly? |
| 16:47:03 | <honno> | Is a megawarc actually infact just a big WARC split up into chunks for IA purposes? |
| 16:47:20 | <@JAA> | A megawarc is effectively just concatenated WARCs. |
| 16:47:24 | <honno> | Ahh Im trying to 1) browse the WARC collections locally 1.5) to scrape it |
| 16:47:29 | <@JAA> | It's a valid WARC itself. |
| 16:47:45 | <honno> | Oo |
| 16:47:49 | <@JAA> | So you could import it into pywb directly. |
| 16:48:19 | <purplebot> | File:Sploder-logo.png uploaded by Betamax (+0) just now -- https://www.archiveteam.org/?diff=45906&oldid=0 |
| 16:48:19 | <purplebot> | File:Sploder-homepage.png uploaded by Betamax (+0) just now -- https://www.archiveteam.org/?diff=45907&oldid=0 |
| 16:48:37 | <betamax> | JAA: that's it working fine now, thanks |
| 16:48:37 | <@JAA> | Heh, well, that proves it then I guess. |
| 16:48:53 | <@JAA> | jrwr: Moderation extension is blocking file uploads. Can that be fixed? |
| 16:49:20 | <@JAA> | betamax got that same error as others reported before, added to automoderated, now it works. |
| 16:50:25 | <honno> | For pywb, what does that exactly mean? I still need to restore the megawarcs? or I can just copy and paste the .warc.gz files into archive/ and the idx stuff into /indexes <- I did that and running wayback did not recognize any URLs |
| 16:52:02 | <@JAA> | honno: https://pywb.readthedocs.io/en/latest/manual/usage.html#using-existing-web-archive-collections |
| 16:52:14 | <honno> | Ayup thats what I followed |
| 16:52:23 | <@JAA> | Did you run cdx-convert? |
| 16:52:26 | <honno> | Yep |
| 16:52:29 | <@JAA> | Hmm |
| 16:52:46 | <honno> | Hmm maybe it gave a false positive, it said everythings alright, but to clarify I used the files with the (lemme check the extension) |
| 16:52:59 | <honno> | .cdx.gz files |
| 16:54:08 | <honno> | And /archive was full of the .megawarc.warc.gz files |
| 16:54:15 | <@JAA> | Maybe it needs uncompressed CDX files. |
| 16:54:27 | <honno> | Ah yea Ill try that now |
| 16:56:19 | <honno> | Ah so now it is converting the cdx to cdxj files, which seems promising behaviour |
| 16:56:44 | <@JAA> | Yeah, that sounds good. |
| 17:04:11 | <SketchTheCow> | Fusl_: Sent |
| 17:04:20 | <honno> | JAA the URLs are recognised now, thanks so much :D guess I asummed coz tools worked with compressed warcs itd be same for these files heh. got a problem with actually loading the pages but think need to restructure my /archive folder |
| 17:04:20 | <SketchTheCow> | Do you need it again |
| 17:08:00 | <@JAA> | honno: Nice! :-) Yeah, since the CDX is the index, it doesn't really make sense to store it compressed since you need to actively access it all the time and decompressing constantly is expensive. I believe the CDX files inside the items aren't really used at all when you access something through the WBM; they're produced by the derive and then added to the full index of the WBM, which I believe is |
| 17:08:06 | <@JAA> | stored in plain for exactly that reason. |
| 17:08:47 | <honno> | ahh cool, I'm getting why the warc format exists now lol |
| 17:09:47 | <Sanqui> | has anybody thought about adding WARC support to something like VCR.py? |
| 17:11:30 | <@JAA> | Hah, turns out I basically reimplemented VCR.py a while ago for my snscrape tests as I didn't know about it. lol |
| 17:12:02 | <Sanqui> | oh damn. I'll be using VCR.py for my new Discord archival tool, with a custom serializer |
| 17:12:39 | <@JAA> | Well, I'm sure VCR.py does it better than my thing and covers more cases etc. I like to reinvent wheels anyway. |
| 17:13:01 | <Sanqui> | well, it doesn't cover websockets |
| 17:13:33 | <Sanqui> | I'm not planning to output WARCs at all at the moment because nobody would be using WARC replay tools for this anyway |
| 17:13:52 | <Sanqui> | I might use the default serializer (yaml) but I'll have to see how much it would balloon file sizes |
| 17:14:18 | <Sanqui> | want to be a bit pragmatic too. no need to store every single header coming from discord... |
| 17:14:18 | <purplebot> | Sploder edited by Betamax (+57, Added logo / image to Sploder page) 23 minutes ago -- https://www.archiveteam.org/?diff=45908&oldid=45905 |
| 17:21:06 | <honno> | it works now woo! Im wondering if folk here have done web scraping projects on large local pywb instances? It's really slow right now. |
| 17:22:29 | <honno> | the pywb docs talk about indexing but that's the same thing the cdx files have already done right? |
| 17:23:20 | <honno> | oo I see, I guess its when I load a page its requesting lots of different things found in different locations or even different warc files... so is there a huge overnight job I could to produce a "static site" of my warc collection? |
| 17:27:24 | | sknebel (sknebel) joins |
| 17:48:51 | <@Fusl> | SketchTheCow: didnt get any |
| 18:47:22 | | VerifiedJ quits [Quit: The Lounge - https://thelounge.chat] |
| 18:50:18 | <purplebot> | ISP Hosting edited by Atphoenix (+354, added users.telenet.be) just now -- https://www.archiveteam.org/?diff=45909&oldid=45720 |
| 18:50:40 | | VerifiedJ (VerifiedJ) joins |
| 19:16:18 | <purplebot> | ISP Hosting edited by Atphoenix (+45) 24 minutes ago -- https://www.archiveteam.org/?diff=45910&oldid=45909 |
| 19:26:09 | <@kiska> | https://server8.kiska.pw/uploads/399891ba737b8fc5/image.png RIP drafts? |
| 19:44:23 | <atphoenix> | lastampa website says (translated) "The service will be available again from February 15, 2021." |
| 19:53:53 | <SketchTheCow> | Oh nice |
| 19:54:02 | <SketchTheCow> | I see hackint needs me to log in to message you |
| 19:54:06 | <SketchTheCow> | Fuck it, you'll get it on slack |
| 19:58:29 | <SketchTheCow> | It's messaged to you on the archive slack |
| 19:58:38 | <SketchTheCow> | This login thing with hackint is boring and dull |
| 20:00:03 | <@JAA> | Ah yes, unlike the login requirement on Slack, right? |
| 20:38:04 | <@Fusl> | JAA: telegraf for ta1 and ta2 fixed |
| 20:49:23 | <@JAA> | Yay, thanks! |
| 20:52:18 | <purplebot> | Deathwatch edited by Vicarage (+143) just now -- https://www.archiveteam.org/?diff=45911&oldid=45904 |
| 21:34:23 | <@kiska> | Damn periscope :D |
| 21:36:56 | | Arcorann_ joins |
| 21:53:27 | <tech234a> | Additional FAQ: https://help.twitter.com/en/using-twitter/periscope-faq |
| 21:53:51 | <tech234a> | It seems that it will go read-only: "Periscope web at periscope.tv/periscopeco, will remain available as a read-only archive of public broadcasts." |
| 22:01:55 | | honno quits [Remote host closed the connection] |
| 22:09:15 | <@arkiver> | yeah periscope |
| 22:09:32 | <@arkiver> | we'll at least wait likely until it is read only |
| 22:13:18 | <purplebot> | Deathwatch edited by Tech234a (+229, Add Periscope), Tech234a (+207, Add additional information about …), Tech234a (+22, Another Periscope clarification) 10 minutes ago -- https://www.archiveteam.org/?diff=45914&oldid=45911 |
| 22:49:33 | | Stilett0 joins |
| 22:51:24 | | Stiletto quits [Ping timeout: 240 seconds] |
| 23:03:20 | | godane (godane) joins |
| 23:12:30 | | godane1 joins |
| 23:15:24 | | godane quits [Ping timeout: 240 seconds] |
| 23:50:24 | <EggplantN_> | Hrm perisocope so sad |
| 23:50:41 | <EggplantN_> | I mean if we can get scripts done (i may be able to feb time) we could go ahead low concurrent |
| 23:50:58 | <EggplantN_> | especially if we can get discovery done early |
| 23:51:00 | | lennier2 joins |
| 23:53:59 | | lennier1 quits [Ping timeout: 252 seconds] |
| 23:54:01 | | lennier2 is now known as lennier1 |
| 23:57:12 | | lennier2 joins |
| 23:59:28 | | lennier1 quits [Ping timeout: 252 seconds] |
| 23:59:32 | | lennier2 is now known as lennier1 |