HomeSearchPrevious dayNext day

00:01:08<@JAA>susu: The pipeline has a --keep-data option to make it not delete the data after upload. Can't tell you how to apply that with Docker though or how to get the data out of the container afterwards.
00:02:03<@EggplantN>hey taka
00:02:07<@EggplantN>we good now?
00:03:22<@EggplantN>we are 1/3rd done
00:06:11taka quits [Ping timeout: 244 seconds]
00:07:10<@JAA>Welp
00:07:24<susu>JAA Thank you, i will consult the seesaw doc I think. I would have liked to avoid touching the project code, but i will use it in a fork, i will see with the maintainer later, it is too specific for this channel now. And afterwards I think a docker cp will work, anyway data is accessible from the virtual fs in var/lib/docker/. Thank you
00:07:42<@JAA>susu: Do *not* modify any code.
00:08:42<@EggplantN>what project is this susu
00:08:59<@EggplantN>*i repeat do not modify code*
00:09:00<@JAA>True, accessing it through /var/lib/docker is dirty but probably easiest. You just need to figure out how to pass the extra option into the Docker container. I'd assume that instead of `docker run ... DOWNLOADER`, you'd run `docker run ... -- --keep-data DOWNLOADER`, but I don't really know.
00:09:04<susu>JAA I may not have understood, --keep-data is in the pipeline code or the pipeline call ?
00:09:23<@JAA>It's an option for seesaw's run-pipeline script.
00:09:44<Craigle>Honestly if you want the project data, why not just grab ALL of the data from IA once we get it uploaded?
00:09:46<@JAA>Which is the Docker container's entrypoint, so the above or some variation of it should work.
00:09:54<@EggplantN>in the docker command after you specific the image i.e atdr.meo.ws/archiveteam/urls-grab --concurrent 20 --keep-data sus
00:10:08<Craigle>That would allow you to get everything vs just a small percentage of it
00:10:14<susu>JAA EggplantN Ok thank, you , and don't worry, I was not using in prod env.. I am not a monster :D
00:10:16<@JAA>(Ignore me, just listen to EggplantN who actually knows how to use Docker.)
00:10:35<@EggplantN>i was forced JAA
00:10:43<@EggplantN>YOU DONT KNOW WHAT THEY PUT YOU THROUGH
00:10:48<@EggplantN>YOU'D HAVE NO IDEA OF THE PAIN
00:11:30<@JAA>Oh, I know. I was also forced. I just minimised my use of it and stopped running workers. Not that I ever ran many anyway.
00:12:10<@EggplantN>tbf its like a drug
00:12:15<@EggplantN>you get hooked
00:12:27<@EggplantN>i deployed my first docker-compose at work last week
00:14:05<susu>Craigle just to verify content quality that comes out my infrastructure
00:15:08<Craigle>Gotcha. Just wanted to save you some trouble if you wanted a personal copy of all of the files
00:16:28<susu>Craigle thankyou, but even if I wanted to make a copy, i could'nt because I regularly ran out of space for such "cold" data
00:17:26<Craigle>Hahaha, I understand that situation completely
00:18:35hooway quits [Client Quit]
00:22:55<susu>Thank you all for the help and your commitment over the projects, and again, I am not a monster, please forgive me for this stressful blow about the idea of project modification :D
00:32:57taka2 joins
00:33:02<taka2>Perhaps niconico is now puzzled as to why the load is so heavy.
00:33:03<taka2>So why don't you try sending a DM to Twitter support?
00:33:03<taka2>https://twitter.com/nico_nico_talk
00:34:03taka2 quits [Remote host closed the connection]
00:34:11<thuban>back from afk... my understanding is that at this pint we do _not_ need/want additional non-jp ips, is that correct?
00:34:32<thuban>*point
00:34:37<thuban>(also is #niconino [sic] the canonical channel now?)
00:37:25<@hook54321>oh huh, finally hit the channel limit.
00:56:19<@kiska>And that is?
00:58:18<@JAA>CHANLIMIT=#:123
00:58:34<hexa->oh :3
00:59:32<@JAA>Wild hexa- appeared!
00:59:45<hexa->I do that sometimes x)
00:59:48<@JAA>:-)
01:00:34<hexa->I've hit that limit on freenode, it's roughly 120 over there as well
01:01:45<@hook54321>guess i could just connect twice, although it'd only be a matter of time until i hit it again :P
01:02:04<hexa-> CHANLIMIT=#:120
01:02:12<hexa->hook54321: what would be a sensible limit?
01:02:25<@hook54321>idk
01:03:28<@hook54321>1000 /s
01:04:48<hexa->rip :D
01:04:52<@JAA>1337 :-P
01:05:25<@JAA>Maybe we should abandon old project channels after a while.
01:07:23<hexa->anyway, it's a per server setting, that we'd need to sync up network wide
01:07:40<hexa->and I'm not sure whether there is an i-line for loosening that up
01:08:15<@hook54321>JAA: i guess now we can actually close channels
01:08:43<@JAA>Yeah, /kickban everyone, unregister, done.
01:09:06<@hook54321>or just forward everyone to #archiveteam or here
01:09:25<@JAA>Yeah, but I think that would require keeping at least one client in the channel.
01:09:36<thuban>might be a good time to fix the wiki's project template like we've talked about
01:09:38<@JAA>You can't MLOCK modes with arguments.
01:09:58<@JAA>And I believe the relevant ChanServ setting just blocks access the channel rather than redirecting.
01:10:06<@JAA>access to*
01:11:26<hexa->hm, idk /cs help close?
01:12:44<hexa->maybe soper only, idk
01:13:09<@JAA>I get the help text at least.
01:13:22<@JAA>Also wouldn't redirect like a +b *!*@*$#archiveteam-bs though.
01:13:28<hexa->true
01:13:50<hexa->TIL: opers have 3*max_chans_per_user limit
01:14:02<@JAA>Cheaters!
01:14:17<hexa->I honestly thought I could just evade the limit altogether
01:14:24tzt quits [Ping timeout: 260 seconds]
01:14:35<@JAA>Yeah, true.
01:15:06<hexa->https://github.com/charybdis-ircd/charybdis/blob/master/modules/core/m_join.c#L283-L292
01:15:12<hexa->doesn't look like there is an exception either
01:15:29<hexa->eh max_chans_per_user_large
01:15:40<hexa->that looks interesting
01:16:15<hexa->> "Maximum extended number of channels a user can join",
01:16:28<@JAA>Set via this priv: https://github.com/charybdis-ircd/charybdis/blob/f679aa459f5fa384b7b5cac4ef1b214c7582d729/modules/m_privs.c#L78
01:16:42<hexa->https://github.com/charybdis-ircd/charybdis/commit/a4721f5e9fd9dedc68534e736075a755645621f2
01:16:57<hexa->huh, that looks iline-able
01:17:08<hexa->but requires charybdis 4.x
01:18:01<@JAA>Ah. There's always a catch, isn't there?
01:18:28<hexa->right
01:27:14<hexa->guess we'll look into solanum some day
01:27:21<hexa->not sure when people wake up to discuss that
01:28:10<@hook54321>~~\probably quicker than things happen on efnet~~
01:28:35<hexa->welp
01:28:38<hexa->probably!
01:30:28ragu__ quits [Remote host closed the connection]
01:31:28ragu_ joins
01:32:10<@JAA>Isn't that arkiver's tagline? :-)
02:16:36tzt joins
02:32:19<@kiska>Yes :D
02:51:43daalekz (daalekz) joins
02:51:53us3rrr joins
03:09:36susu quits [Ping timeout: 244 seconds]
03:45:45marked19573 quits [Client Quit]
03:46:18marked19573 joins
03:55:33<tech234a>We may be able to resume the Endomondo project... it seems the maintenance page was removed for profile pages for some reason, and the rate limit seems a bit more lenient. I would assume the existing scripts would work. Could someone look into this? #findelmundo
03:56:40Jonboy3451 joins
03:59:48Mineroboter_ joins
04:00:07Jonboy345 quits [Ping timeout: 264 seconds]
04:01:19Mineroboter quits [Ping timeout: 264 seconds]
04:24:02taka (taka) joins
04:38:49qw3rty_ joins
04:42:39qw3rty__ quits [Ping timeout: 260 seconds]
04:49:45Arcorann (Arcorann) joins
04:51:39auctus quits [Client Quit]
04:56:28auctus joins
05:03:09igloo22225 quits [Remote host closed the connection]
05:03:13igloo222251 joins
05:09:18linuxgemini2 (linuxgemini) joins
05:11:14linuxgemini quits [Ping timeout: 260 seconds]
05:11:14linuxgemini2 is now known as linuxgemini
05:11:29ave0 joins
05:11:49ave quits [Ping timeout: 260 seconds]
05:11:49ave0 is now known as ave
05:23:33igloo222251 is now known as igloo22225
05:23:37igloo22225 quits [Client Quit]
05:23:44igloo22225 (igloo22225) joins
06:04:01DogsRNice quits [Read error: Connection reset by peer]
07:03:59lennier1 quits [Read error: Connection reset by peer]
07:04:25lennier1 (lennier1) joins
07:15:48<atphoenix>https://scalewheels-gr.com/ has a different message from before. Old message had a date that suggested a March 1 return. No date now. I guess we shall see. "ScaleWheels Greece are down for maintenance. Extreme makeover are in progress. We will be back online soon. Thanks for your passion. See you soon"
07:16:17<atphoenix>that was crawled (in part) on or about 2/14 in AB 1i41nj2p5q78xkwh8jcooaxmn
07:36:24sliccricc_ quits [Remote host closed the connection]
07:37:42sliccricc_ (sliccricc) joins
07:48:58sliccricc_ quits [Remote host closed the connection]
07:49:17sliccricc_ (sliccricc) joins
08:13:59Arcorann_ joins
08:17:31Arcorann quits [Ping timeout: 264 seconds]
08:25:04<yawkat>ive completed dumping bintray, and now i have a 4.2TB .warc.gz. do i just... upload it to archive.org? or what is the best way
08:26:35<yawkat>s/bintray/jcenter
08:27:46<mgrandi>damn
08:27:54<mgrandi>uhhh, not sure honestly
08:29:31daalekz quits [Ping timeout: 264 seconds]
08:29:40<mgrandi>i've heard that 1TB per IA item is usually best, and to be sure to use the IA cli
08:35:24<yawkat>so if i upload it as warc.gz, will people be able to browse the individual files in it?
08:43:20BlueMaxima quits [Read error: Connection reset by peer]
08:47:29<yawkat>but it looks like theres tools for splitting warc files, so it should be possible to split the warc into 1tb chunks
09:01:39Arcorann_ quits [Ping timeout: 260 seconds]
09:20:38hooway joins
09:50:23<OrIdow6>Isn't 1 TB somewhat high? I can't keep track of it
09:51:21<OrIdow6>Also, if by "files" you mean individual records, not as far as I know; and it will not go into the Wayback Machine either
10:01:37<yawkat>yes i mean individual records. itd be kind of pointless if i upload it but nobody could use it because theyd have to download the full 4tb just to load a single artifact
10:02:35<OrIdow6>They can do range requests
10:02:37<OrIdow6>But AFAIK ther
10:03:10<OrIdow6>e's nothing in the web interface that lets you see individual records, like it does do for files in zips
10:13:43<@EggplantN>s3's limit is 1TB
10:13:50<@EggplantN>it wont accept anything over 1TB
10:18:42Arcorann_ joins
10:28:19<themadpro>Still muted in #archiveteam so I'm sending here
10:28:25<themadpro>> March 1: Niconico will delete past comments (regardless of whether the video is deleted or not).
10:28:35<themadpro>> March 15: Niconico will delete metadata (including title, description, view count) for deleted videos.
10:29:12<themadpro>Better late than never, a project was never set up for this was it?
10:35:41<themadpro>https://wiki.archiveteam.org/index.php/Deathwatch
10:36:26<yawkat>OrIdow6: so maybe it would be better if i zipped it and uploaded that?
10:37:05<OrIdow6>themadpro: #niconino
10:37:53<OrIdow6>yawkat: If it's a copy of a file-uploading site or something like that, I think it'd make sense (though purists will say that it should always be in warc regardless of accessiblility)
10:38:20<yawkat>it's jcenter, i.e. a repository of binary build artifacts
10:38:38<OrIdow6>Oh, yeah, I think that would make sense then
10:38:45<yawkat>the warc shouldnt contain much useful info, since it's a pure file tree, yea
10:38:52<themadpro>Orldow6: thank you!
10:40:13<yawkat>would it make sense to prod someone from archive.org for this? i dont want to stress their infrastructure if it turns out there was a better way
10:51:40daalekz (daalekz) joins
10:58:59<purplebot>Niconico edited by Wickedplayer494 (-6, New IRC channel) just now -- https://www.archiveteam.org/?diff=46370&oldid=46367
11:02:00<purplebot>Current Projects edited by Wickedplayer494 (+141, Webs is FINALLY a go, and so is …) just now -- https://www.archiveteam.org/?diff=46371&oldid=46329
12:00:07Arcorann_ quits [Ping timeout: 264 seconds]
12:03:08Arcorann_ joins
12:06:56taka quits [Ping timeout: 244 seconds]
12:25:49yawkat quits [Ping timeout: 260 seconds]
12:30:15yawkat (yawkat) joins
12:44:29yawkat quits [Ping timeout: 260 seconds]
12:54:21daalekz quits [Client Quit]
12:58:26taka (taka) joins
13:03:47Arcorann joins
13:05:31Arcorann_ quits [Ping timeout: 264 seconds]
13:10:22yawkat (yawkat) joins
13:18:19yawkat quits [Ping timeout: 260 seconds]
13:44:55yawkat (yawkat) joins
13:51:01yawkat quits [Client Quit]
13:52:48yawkat (yawkat) joins
13:55:49<AK>Is it worth switching the default warrior to reddit instead of urls to help clear the backlog?
14:20:54Matthww9 joins
14:22:19Matthww quits [Ping timeout: 264 seconds]
14:22:19Matthww9 is now known as Matthww
14:23:36<@Kaz>AK: yes
14:23:47<@Kaz>(done)
14:26:05ragu__ joins
14:30:04ragu_ quits [Ping timeout: 260 seconds]
14:31:19Arcorann quits [Ping timeout: 264 seconds]
14:49:08poro197470 joins
14:56:58lennier1 quits [Read error: Connection reset by peer]
14:57:19lennier2 joins
14:57:25lennier2 is now known as lennier1
15:04:19sliccricc_ quits [Ping timeout: 264 seconds]
15:16:12taka quits [Remote host closed the connection]
15:23:58us3rrr quits [Client Quit]
15:24:21onetruth quits [Client Quit]
15:50:45spirit quits [Quit: Leaving]
15:52:54spirit joins
15:53:25spirit quits [Client Quit]
15:56:31spirit joins
17:03:32Mateon2 joins
17:05:31Mateon1 quits [Ping timeout: 264 seconds]
17:05:31Mateon2 is now known as Mateon1
17:06:27yawkat quits [Client Quit]
17:08:36yawkat (yawkat) joins
17:38:07poro197470 quits [Ping timeout: 244 seconds]
17:47:54<yawkat>when sunsetting is done in multiple steps, how do you list that on the deathwatch page?
18:13:52<atphoenix>I'd list the closest date, and what is being removed at that date. I guess another entry could be made for the later sunset dates. Probably would be useful to mention 'Additional content is scheduled to sunset on <date>'.
18:20:38jonboy3452 joins
18:24:07<@EggplantN>nico needs help default project moved
18:24:43Jonboy3451 quits [Ping timeout: 264 seconds]
18:26:30<yawkat>atphoenix: ok, will do
18:58:43nerdguy1138 quits [Remote host closed the connection]
19:01:28nerdguy1138 (nerdguy1138) joins
20:07:49<@Kaz>EggplantN: moved defaultproject image too?
20:08:24<@Kaz>looks like it's still on reddit, not updated
20:08:26<@Kaz>now*
20:23:37icedice joins
20:24:03<icedice>Is there a project for archiving MangaDex?
20:25:17<icedice>It's the only manga reading site that hosts user submitted scanlations in original quality
20:26:27<icedice>They respond to DMCA, but given how Internet censorship is ramping up each and every year it wouldn't surprise me if they'll be forced to shut down their site due to legal pressure sooner or later
20:27:07<icedice>Most of the manga on there isn't even available in English outside of fan translations
20:27:27<icedice>And they also host scanlations in other languages
20:39:07Stiletto quits [Ping timeout: 264 seconds]
20:48:53Stiletto joins
21:26:54Barto quits [Remote host closed the connection]
21:27:07Barto (Barto) joins
22:11:37Guest69 joins
22:16:44BlueMaxima joins
22:42:18DLoader quits [Remote host closed the connection]
23:22:17Arcorann joins
23:28:57<mgrandi>First step is to create a wiki page for it and maybe do research on how the site is laid out @icedice
23:30:13<icedice>All right
23:30:17<mgrandi>@yawkat: so it's not going to be in a super accessible format regardless of what you do, I would maybe split it into smaller chunks that make sense
23:30:32<icedice>I know they use DDoS-Guard and Path Network
23:31:16<mgrandi>I don't know if IA runs a CDX job on a WARC when uploaded, no matter what you should generate that for your final file setup so it's easier to find the files @yawkat
23:31:33<mgrandi>Maybe @JAA can comment when they have time
23:32:09<mgrandi>@icedice: good stuff to include, basically, search pages, how are the urls structured, etc. Bonus points if you are a coder and can figure out how to scrape all the relevant urls
23:36:34pome joins
23:36:57pome quits [Remote host closed the connection]
23:37:10pome joins
23:37:49pome leaves
23:51:23<icedice>Correction: Looks like they ignore DMCA, but respond to C&D
23:54:19<@JAA>yawkat: Don't do 1 TB chunks. We generally use 50 GB in projects. If it's primarily large files, you can probably get away with 100 or even 200, but yeah, definitely don't do 1 TB. That's the hard limit for total item size, and if you get close to it, stuff *will* break. As for discoverability, if your account gets whitelisted (not sure what the process is, it just happened 'magically' to mine), it'd
23:54:25<@JAA>get into the Wayback Machine with its fancy tools. Otherwise, yeah, the CDX will at least provide an index of the WARCs, which can be used to download any parts of it with range requests. It's not very pleasant/accessible though. There used to be a WARC-to-ZIP thingy that converted individual WARC records to a ZIP file, but that broke many moons ago.
23:55:01hooway quits [Client Quit]

HomeSearchPrevious dayNext day