HomeSearchPrevious dayNext day

00:25:42User quits [Client Quit]
00:30:49<FalconK>SketchTheCow: I've added a 60 second delay after each WARC upload from ananiel for the moment, let me know if that's not enough.
00:48:54<OrIdow6>JAA: What channel was this message you saw on EFNet? Don't see it in my logs
00:50:17<@kiska>#archiveteam :D
00:50:18<@kiska>I do
00:53:39<OrIdow6>Oh, looks like I was disconnected for 2 days
00:54:25<OrIdow6>Whoops
00:55:19<@kiska>https://server8.kiska.pw/uploads/0c620cb9dc6420d3/image.png Here
01:02:45<OrIdow6>Thanks
02:25:40<mgrandi>Is there a way to see if archivebot or similar has backed up a twitter account before I request itm
02:29:59<@JAA>Not easily (yet).
02:30:37<@kiska>Just queue it™
02:30:46<@JAA>Or grep -Fi your logs.
02:38:45<mgrandi>I'm assuming not everyone has access to queue stuff?
02:38:53<mgrandi>https://twitter.com/Manga4Congress is the account I was looking at saving
02:44:23<jodizzle>mgrandi: According to my logs, that was saved just recently
02:44:52<mgrandi>Ok thanks
02:45:06<mgrandi>Hence why I wanted to check rather than spam
02:45:21<mgrandi>Do we post in #archivebot to request? Or here
02:45:36<jodizzle>Probably here is better, since #archivebot is cluttered with machine messages
02:46:03<jodizzle>But it looks like you're in #archivebot. Can you search your logs yourself?
02:52:18<mgrandi>i can, although irccloud doesn't have a search all history feature yet sadly >.>
02:53:03<mgrandi>also, i asked this question before, i'm uploading a twitch stream and it has the file extension .ts, should i rename it to be .mp4? i think it is technically .mp4 video
02:55:01<mgrandi> "Codec: H264 - MPEG-4 AVC (part 10) (h264)" for video, "Codec: ADTS" for audio
03:00:11<@JAA>.ts is normally MPEG-TS, not MP4.
03:01:01<mgrandi>hmm, what i posted is what vlc says, i'll leave it as a .ts file extension then
03:01:52<@JAA>You posted the video and audio codecs, not the container type.
03:02:37<@JAA>H.264 can be stored in MP4, MTS, M2TS, MKV, etc.
03:05:38<mgrandi>i just used ffprobe and it says `mpegts`, so you were right
03:06:18<mgrandi>makes sense since twitch stores VODs in like 30 second clips and then this program concatenates them together and other traditional formats probably wouldn't like that
03:06:26<mgrandi> /s/formats/containers
03:06:45<@JAA>Yep
03:47:26<@JAA>It would probably be a good idea to archive InfoWars. https://tv.infowars.com/ lists ~11k entries with direct links to video and audio files. We had an AB job for this site (under another domain but serving the same content, http://tv.us-west-1c.infowars.com/ ) which retrieved 1.5k URLs from pravda.infowars.com, which serves the video/audio files, for a total of a bit over 2 TB, and almost 33k such
03:47:32<@JAA>URLs were ignored later. So as a rough estimate, this is 40-50 TB. However, there are duplicates in here, namely URLs with and without some timestamp (cache buster?) parameter and higher/lower resolution. Still probably very roughly 15-20 TB in total. Unfortunately, the site is fairly low on metadata; for example, the description is 'to be added...' for many episodes from the past year or so.
03:58:07<OrIdow6>You're saying that the AB job didn't get them because of ignores?
03:58:26<@JAA>Well, because of the size, we added the ignore.
03:58:46<OrIdow6>Oh
03:59:08<@JAA>Not going to grab 40-50 TB through AB, *especially* not on the pipeline it was running on, which has had upload issues to everywhere for the past few weeks.
03:59:34<@JAA>Also, this should probably be in IA items, not in random WARCs.
03:59:55<@JAA>Perhaps there's better metadata somewhere else.
04:05:29<OrIdow6>What's the risk? Is this something imminent?
04:08:22<@JAA>Not that I'm aware of.
04:08:39<@JAA>More of a 'we should probably keep a copy of this somewhere' thing.
04:12:44<mgrandi>infowars is 50 tb? what the hecko
04:12:56<OrIdow6>Well, looks simple enough, notwithstanding the problem of getting more metadata
04:13:02<OrIdow6>Hours and hours of hd video
04:13:18<mgrandi>the guy has been banned from so many platforms i dont know what he is on anymore
04:13:24<mgrandi>that might have more metadata
04:17:32<mgrandi>are you not able to have folders when you upload something to archive.org?
04:34:28<mgrandi>guess you can't, whoopsie
04:35:08<@JAA>You can.
04:38:49<mgrandi>i'm using the ia command line with --spreadsheet, let me see what column i need
04:40:34<@JAA>Never used --spreadsheet, but basically what you do is set the --remote-name to a relative path. The directories get created automatically as you upload something into them.
04:41:08<mgrandi>not sure if i should attempt to fix this or if i can be deleted and i start again, i apparently also didn't specify the collection right and its in the 'opensource' collection rather than something more appropriate
04:41:28<@JAA>You can also `ia upload` an entire directory, which will then end up as a directory in the item. (Internally, it really just uploads each file in the directory with the corresponding path, and the directory itself is recreated.)
04:43:33<mgrandi>hmm , the documentation sorta implies that its only really available with the --spreadsheet option which is what i tried to use
04:43:41<mgrandi>https://archive.org/services/docs/api/internetarchive/cli.html#bulk-uploading
04:47:10<mgrandi>https://archive.org/details/aoc_among_us_twitch_stream_2020_10_20 is what i created, i guess if i could edit it i would just get rid of the 'history' folder and then just have 1 copy of the chat log (the 776770697.* files), and then edit the hashes.json/readme.md to fix the two copies of the chat log not being there)
04:47:55qw3rty_ joins
04:48:38<mgrandi>first upload, i'm bad at this :)
04:51:28qw3rty__ quits [Ping timeout: 252 seconds]
04:53:16Zopolis4 joins
04:54:16<@JAA>Ah, the history directory, always fun.
04:56:44<@JAA>`ia delete identifier history/files/foobar -H x-archive-keep-old-version:0`
04:57:05<@JAA>List each of the files you want to delete there (can be multiple in the same command).
04:58:46<@JAA>(When you upload a file and there is already a file with the same name/path, the existing file gets moved to history/files to prevent accidental overwrites. This happens even when you delete a file. The special header disables that.)
05:00:01treora quits [Client Quit]
05:00:16<mgrandi>yeah, i had two files with the same name but in different folders, but i guess didn't specify the remote name so it just overwrote it
05:00:30<mgrandi>i tried deleting them but i think its freaking out cause its still deriving it, it said no such directory
05:01:14treora joins
05:03:30Zopolis4 quits [Remote host closed the connection]
05:03:38<@JAA>Right, that was actually an issue I reported a while ago, but Jake decided not to read properly: https://github.com/jjjake/internetarchive/issues/266
05:04:21<mgrandi>whoops
05:04:40<@JAA>I assume the same thing applies on --spreadsheet.
05:04:51<mgrandi>yeah
05:05:13<mgrandi>i'll add this to my list of things to fix, as well as the documentation that leaves out that you can just upload a folder directly
05:13:28<mgrandi>unrelated: now that youtube-dl is back up, did we download all the issues and all that?
05:18:32<SketchTheCow>JAA: It worked
05:18:35godane2 joins
05:19:10<SketchTheCow>Right now there's a small set sitting in the inbox due to the way the system worked. I am probably going to force them over into the last item JUST to keep it so something doesn't suffer in limbo.
05:19:14<SketchTheCow>But it worked.
05:19:25<SketchTheCow>So feel free to pop something more aggressive over there so we see how it holds up
05:20:44godane1 quits [Ping timeout: 240 seconds]
05:21:08<SketchTheCow>Forcing the waiting 40gb just so we're clean for the next batch
05:22:24Zopolis4 joins
05:22:39@arkiver quits [Excess Flood]
05:23:40arkiver (arkiver) joins
05:23:40@ChanServ sets mode: +o arkiver
05:26:27fionera quits [*.net *.split]
08:15:04nerdguy1138 quits [Ping timeout: 240 seconds]
08:27:37hooway joins
10:02:29nerdguy1138 (nerdguy1138) joins
10:52:57VADemon joins
12:04:04Doranwen quits [Ping timeout: 240 seconds]
12:05:08Doranwen joins
12:09:12benjinsmith joins
12:11:28benjins quits [Ping timeout: 252 seconds]
12:11:48benjinsmith is now known as benjins
12:12:21Doranwen quits [Remote host closed the connection]
12:12:44Doranwen joins
12:17:01britmob2 quits [Client Quit]
12:17:20britmob2 joins
12:27:04Doranwen quits [Ping timeout: 240 seconds]
12:47:44Doranwen joins
13:01:09Doran joins
13:01:09Doranwen quits [Read error: Connection reset by peer]
14:17:36balrog quits [Ping timeout: 252 seconds]
14:18:53balrog joins
14:32:20Arcorann quits [Ping timeout: 244 seconds]
14:43:04nimaje quits [Remote host closed the connection]
14:43:24nimaje joins
15:10:51systwi quits [Read error: Connection reset by peer]
15:14:32systwi (systwi) joins
16:08:51fionera (Fionera) joins
16:58:51<@JAA>SketchTheCow: Cool, thanks! Will switch the fastest pipeline to it for now. There's no /pipeline.html or Telegraf on it though, so can't really monitor for out of disk space until it goes boom.
17:10:31<SketchTheCow>We can approach getting telegraf on it.
17:10:49<SketchTheCow>Send instructions, I'll try
17:11:09<@JAA>I have no idea how it works.
17:11:17<@JAA>Fusl_: ^
17:12:34<@Fusl>SketchTheCow: what distro and version is it?
17:33:29<SketchTheCow>Oh, was it fusl
17:33:33<SketchTheCow>You all look the exact same
17:33:37<@Fusl>hi
17:34:27<SketchTheCow>Ubuntu 14.04.6 LTS (Trusty Tahr)
17:34:34<@Fusl>ancient technology
17:34:35<@Fusl>:P
17:34:40<@Fusl>let me see what i can dig out
17:34:47<SketchTheCow>Let me see what they said to me
17:34:50<SketchTheCow>Maybe it's not that
17:35:17<SketchTheCow>focal/20.04
17:35:24<SketchTheCow>It WAS Trusty, now Focal
17:36:17<@Fusl>nice
17:36:20<@Fusl>easy
17:38:41<@Fusl>does it have docker?
18:10:18<purplebot>ITunesU created by Kyndigs (+1413, Creation) just now -- https://www.archiveteam.org/?diff=45766&oldid=0
18:49:03Doran is now known as Doranwen
18:50:09Zopolis4 quits [Ping timeout: 244 seconds]
19:59:58<@Fusl>SketchTheCow: just in case you havent noticed, i dm'd you stuff
20:14:57<SketchTheCow>Saw
20:14:59<SketchTheCow>Off it goes
20:18:05<@Fusl>SketchTheCow: isnt this teamarchive2.fnf.archive.org?
20:18:12<@Fusl>https://atdash.meo.ws/d/000000058/telegraf-detail?orgId=1&refresh=30s&var-includeall=false&var-user=sketchcow&var-host=teamarchive2.fnf.archive.org&var-ival=1s&from=now-2d&to=now
20:19:30<SketchTheCow>There is teamarchive2 and teamarchive1
20:19:34<@Fusl>ah
20:19:38<SketchTheCow>I just installed on 1
20:19:42<SketchTheCow>2's been going for a while
20:20:09<SketchTheCow>But then JAA got all "blah blah blah uncomfortable shooting hundreds of gigabytes into a potential black hold"
20:20:16<SketchTheCow>So here we are
20:24:34<@JAA>:-)
20:46:16sembiance quits [Ping timeout: 252 seconds]
21:02:06sembiance (sembiance) joins
22:42:04balrog quits [Ping timeout: 240 seconds]
22:42:34<@JAA>mgrandi: I don't think anyone looked into it yet. I could grab it with qwarc probably, but I don't have Flash, so someone would have to tell me what needs to be grabbed.
22:43:08<mgrandi>Doesn't chrome still have it built in? Or did it get removed
22:43:13<mgrandi>Regardless I'll take a look
22:43:22<@JAA>I don't know, I don't use Chrome. :-)
22:43:33<mgrandi>Neither do I, but I have it installed at least
22:44:59<mgrandi>Flash however is bugging me and wanting to uninstall itself, it's time is near. o7
22:46:29Arcorann (Arcorann) joins
22:46:32<@JAA>SketchTheCow, Fusl: So uh, what's the status? Doesn't show up on atdash as far as I can see.
22:50:27<SketchTheCow>I mean, my thing thinks it's going
23:03:56balrog joins
23:28:41<@kiska>systemctl restart telegraf?
23:42:59VADemon quits [Client Quit]
23:50:10@arkiver quits [Excess Flood]
23:50:58arkiver (arkiver) joins
23:50:58@ChanServ sets mode: +o arkiver

HomeSearchPrevious dayNext day