HomeSearchPrevious dayNext day

00:00:34<luckcolors>wessel1512: we picked #waspscorp in the end
00:00:41<luckcolors>but thanks anyways :)
00:00:54<wessel1512>no problem
00:30:43<SketchTheCow>Hellooooo
00:30:56<SketchTheCow>I've got Archivist being cranky. What's going on.
00:34:05<SketchTheCow>Something department homeland security something something tor, something ban
00:36:15<@Kaz>threw toys out of pram, wouldn't actually say what the issue was
00:36:17<@Kaz>then he left
00:36:54<@Kaz>but yes, also something about a sinkhole domain in reddit url list
00:37:05<@Kaz>that apparently means we're all going to jail (again)
00:38:09<flashfire42>wtf is a sinkhole domain and why does it mean I am going to jail lol
01:04:04@kiskaWeebChat quits [Ping timeout: 240 seconds]
01:47:15systwi_ (systwi) joins
01:47:50systwi quits [Ping timeout: 252 seconds]
02:12:20VADemon joins
02:29:36<@arkiver>any channel ideas for parler?
02:48:33<jodizzle>#falter?
02:48:35<jodizzle>#failer?
02:48:39<jodizzle>I don't think those quite fit though
02:49:07<@arkiver>nope :P
02:49:18<purplebot>File:Kinja-logo-black.jpg uploaded by Arkiver (+0) just now -- https://www.archiveteam.org/?diff=45757&oldid=0
03:12:23<@Fusl_>#parlerment?
03:32:07<VADemon>#neparlepas
03:33:06<@arkiver>last two are nice
03:33:49<@arkiver>I especially like #neparlepas since it keeps things french
03:35:00<VADemon>Now I know 4 years of french in school didnt go to waste ... :P
03:39:46<jodizzle>Yeah, I like it too
04:36:16<@Fusl>me2
04:38:01Stilett0 joins
04:39:26Stiletto quits [Ping timeout: 252 seconds]
04:40:32VADemon quits [Ping timeout: 252 seconds]
04:56:37qw3rty__ joins
05:00:04qw3rty_ quits [Ping timeout: 240 seconds]
05:03:35<voltist>When wiki pages talk about integrating WARC records into the Wayback Machine, what do they mean? Is there some special partnership going on there, or can anyone upload bulk scrapes onto the machine?
05:08:42<ivan>if you have mediatype:web and are whitelisted your WARCs go into wayback, otherwise not
05:11:16<voltist>ivan: Ah got it. What kinds of folks are whitelisted?
05:16:48<ivan>archivebot and probably some other organizations IA trusts
05:17:29<ivan>mark or jeff at IA might know
05:27:58<@JAA>'ArchiveBot' isn't really whitelisted as it doesn't exist as an account on IA. It's just uploaded through Jason's account.
05:28:15<@JAA>(Well mostly, some items went through other accounts.)
05:29:15<@JAA>IA probably won't tell you the details, just like they won't really talk about inclusion criteria. You just are or aren't whitelisted.
05:33:33<tech234a>Fusl: tracker is 403ing for me?
05:33:41<@Fusl>what url?
05:33:49<tech234a>https://tracker.archiveteam.org/
05:33:55<@Fusl>oh
05:33:56<@Fusl>uh
05:33:57<@Fusl>1s
05:35:24<@Fusl>that should do
05:35:24<tech234a>Homepage is fixed, but https://tracker.archiveteam.org/google-sites 404s
05:35:33<@Fusl>urgh
05:35:55<@Fusl>fixed as well
05:36:02<tech234a>thanks!
06:07:59<SketchTheCow>https://archive.org/details/SonyPSPManualsv1.2
07:25:30<nico_32>voltist: if you are know on a project, you can request a change of mediatype
07:25:50<nico_32>that's how I got my yahoogroup' grab into wbm
07:29:26<nico_32>after the mediatype change, it take some days before wbm see the data
08:02:08<Zopolis4>is there a form of blank script for archiving? as in, one that reads off a list of urls, with no project-specific requirements/code
08:02:29<Zopolis4>essentially a script that will automatically work with a list of urls
08:02:37<Zopolis4>closest one ive found is pastebin
08:04:52kiwec (kiwec) joins
08:05:22<jodizzle>Zopolis4: `wget -i <filename>`, if that's what you mean.
08:05:47<jodizzle>I'm a bit confused what you mean by 'script'
08:06:15<Zopolis4>something like this https://github.com/ArchiveTeam/pastebin-grab
08:06:45<OrIdow6>What are you trying to do?
08:07:09<OrIdow6>If anything
08:08:18<Zopolis4>i have a bunch of plans for scraping websites/groups of websites that are either too big for archivebot or are dynamic
08:10:57<jodizzle>So the pastebin-grab project that you linked is specifically designed to work in a distributed setup, but I guess in principle you could adapt it to your needs. A more general tool though is https://github.com/ArchiveTeam/grab-site, which is kind of like a local archivebot.
08:12:02<Zopolis4>the sites are too big for me to handle alone
08:12:11<Zopolis4>i meant something that connected to a tracker and all that
08:14:14<OrIdow6>Something like that
08:14:35<Zopolis4>?
08:15:05<OrIdow6>Something simple like pastebin, yes
08:15:12<Zopolis4>ah ok
08:17:51<Zopolis4>considering the project-specific tasks sections of pastebin-grab (and fotoalbum-grab) are empty, ill use pastebin
08:19:43<OrIdow6>"project-specific tasks"?
08:21:09<Zopolis4>############################################################################ This section defines project-specific tasks.#
08:21:43<OrIdow6>Oh
08:21:43<Zopolis4>(from pipeline.py)
08:22:04<OrIdow6>Beware that the lua script still has project-specific parts, and that's normally where the subtleties are
08:22:20<Zopolis4>yup
08:22:25<Zopolis4>im looking through some now
08:22:59<Zopolis4>apart from pipeline and the lua script, are there any files that are different across projects?
08:23:07<Zopolis4>not counting ignore-list
08:24:38VADemon joins
08:25:25<Zopolis4>and how would i be able to test the scripts
08:39:49<jodizzle>Zopolis4: I think you would probably need to run your own tracker
08:41:11<Zopolis4>ah
08:42:41<Zopolis4>how would i do that?
08:43:25<@Kaz>Zopolis4: it sounds like what you actually want is grab-site
08:43:36<Zopolis4>scroll up a bit
08:44:20<@Kaz>i don't understand what you're trying to achieve
08:44:30<@Kaz>hijack the pastebin project to save arbitrary urls?
08:44:39<@Kaz>how big are these sites that they're too big for archivebot
08:44:53<Zopolis4>some of them archivebot might be able to scrape
08:45:04<Zopolis4>some are around 500gb+
08:45:31<Zopolis4>some have url patterns that would screw up archivebot
08:45:49<@Kaz>I can currently see a 7.6TB job in archivebot
08:46:03<Zopolis4>whoah
08:46:27<@Kaz>i think just assuming things are 'too big' might be a bit quick to jump to a conclusion
08:46:33<Zopolis4>true
08:47:10<Zopolis4>in some cases i have big lists of sites that are all at risk of dissapearing
08:47:21<@Kaz>arrrrrrrrrrchivebot is your friend
08:47:24<@Kaz>trust me
08:47:37<Zopolis4>i tried
08:47:37<@Kaz>then throw in ignore patterns as required
08:47:46<Zopolis4>but its not ignore patterns
08:47:48<@Kaz>if they're truly too big, throw them in the deathwatch page on the wiki
08:47:51<Zopolis4>but ill try again
08:48:08<jodizzle>Note that at the moment we're pretty focused on elections stuff
08:48:36<jodizzle>So please be mindful of how much you queue
08:49:06<Zopolis4>yh that was the main issue
08:49:17<jodizzle>I see
08:49:54<jodizzle>Well, that's not a permanent condition. It just takes a while to churn through elections content.
08:50:45<jodizzle>How at-risk are these sites?
08:52:24<OrIdow6>Yeah, if nothing compels you to run a secret archival process, AT can take it up
08:52:58<OrIdow6>(If sufficiently at risk etc.)
09:00:18britmob2 quits [Quit: britmob2]
09:06:32<Zopolis4>one of the sites isint super at risk, but it hosts a lot of content available nowhere else, and its having hosting, funding and dmca problems
09:07:11<Zopolis4>others arent at risk of dissapearing, but they have large portions of their content removed regularly
09:08:18<Zopolis4>theres a massive chunk which are likely to delete most if not all of their content after dec 1 2020
09:09:50<Zopolis4>and theirs a big chucnk which are unlikely to disssapear, but host large amounts of content
09:12:16<Zopolis4>and one is an index of otherwise unavailable links
09:33:44<Zopolis4>another one is full of metadata i need for a much much larger project i have in the works
09:52:39<jodizzle>Zopolis4: Well, feel free to make your case for them in #archivebot. If the sites really are large, we might want to put them on the Deathwatch, like Kaz said.
09:53:21<Zopolis4>k
09:53:35<Zopolis4>i might wait until the election stuff has gone through
10:42:52kiskaWeebChat (kiska) joins
10:42:53@ChanServ sets mode: +o kiskaWeebChat
11:40:44<nico_32>wget --load-cookies "${COOKIE}" -e robots=off --keep-session-cookies --warc-file="${WARC_NAME}" --input-file="${URL_LIST}" --user-agent="${USER_AGENT}" --tries=3 --warc-header="operator: Archive Team" --random-wait --warc-max-size=1G -o "${WARC_NAME}.wget-log" --waitretry 30 --timeout 30 --output-document "${WARC_NAME}.wget-tmp" -nv
11:41:41<nico_32>Zopolis4: that what I used for a grab
12:30:55Zerote quits [Read error: Connection reset by peer]
12:34:08Zerote joins
13:03:14<nico_32>*sigh*
13:03:41nico_32 is adding some page to wbm via the Live Web Proxy
13:04:27<nico_32>there is a link from a RE page on github to a folder on another site
13:04:40<nico_32>WBM have the "index of" 38 times
13:04:44cadence quits [Ping timeout: 240 seconds]
13:04:45<nico_32>but 0 time the files
13:05:34cadence (cadence) joins
13:05:41yano quits [Quit: WeeChat, The Better IRC Client, https://weechat.org/]
13:05:48<nico_32>fixed now
13:06:03yano (yano) joins
13:28:53britmob2 joins
13:29:16cadence quits [Ping timeout: 252 seconds]
13:29:24cadence (cadence) joins
14:58:32@arkiver quits [Excess Flood]
14:59:25arkiver (arkiver) joins
14:59:25@ChanServ sets mode: +o arkiver
15:01:44Arcorann quits [Ping timeout: 240 seconds]
16:00:17@arkiver quits [Excess Flood]
16:00:56arkiver (arkiver) joins
16:00:56@ChanServ sets mode: +o arkiver
16:12:49Zerote quits [Read error: Connection reset by peer]
16:14:41Zerote joins
17:00:44<@JAA>Zopolis4, Kaz: AB doesn't really care much about the data size; we've had a job of over 18 TB before. What matters much more is the number of URLs and outlinks. (Although of course if individual downloads are huge, they might fill the disk, and parallel downloads of big files are broken, so it'll take a long time to do it sequentially.)
17:15:52VADemon quits [Ping timeout: 252 seconds]
17:28:32katocala quits [Remote host closed the connection]
17:39:54@arkiver quits [Excess Flood]
17:40:36arkiver (arkiver) joins
17:40:36@ChanServ sets mode: +o arkiver
18:25:24katocala (katocala) joins
19:50:54Stiletto joins
19:53:10Stilett0 quits [Ping timeout: 252 seconds]
19:57:16HP_Archivist (HP_Archivist) joins
19:58:31Stilett0 joins
19:59:46Stiletto quits [Ping timeout: 252 seconds]
20:38:25@arkiver quits [Excess Flood]
20:39:06arkiver (arkiver) joins
20:39:06@ChanServ sets mode: +o arkiver
20:54:20teej joins
21:35:24<kiwec>finished downloading the twitch clips, most of them were 403s. final size 1.5Tb for 90317 clips
22:10:15VADemon joins
22:40:55kiwec quits [Remote host closed the connection]
23:27:17godane (godane) joins

HomeSearchPrevious dayNext day