HomeSearchPrevious dayNext day

00:00:03HackMii quits [Remote host closed the connection]
00:00:20HackMii (hacktheplanet) joins
00:23:15etnguyen03 quits [Quit: Konversation terminated!]
00:34:32ducky quits [Ping timeout: 260 seconds]
00:36:20ducky (ducky) joins
00:48:29ericgallager quits [Quit: This computer has gone to sleep]
00:58:27ericgallager joins
01:00:51Hackerpcs quits [Quit: Hackerpcs]
01:01:32<ericgallager>has anyone archived Zillow's climate risk scores? https://bsky.app/profile/volts.wtf/post/3m6upiqs67k2c
01:04:54<pokechu22>IIRC zillow is pretty anti-scraping
01:06:56Hackerpcs (Hackerpcs) joins
01:08:42Webuser421296 joins
01:09:01Webuser421296 quits [Client Quit]
01:17:29etnguyen03 (etnguyen03) joins
01:18:01<nicolas17>looking at parti-livestream
01:18:35jason joins
01:19:09<nicolas17>listing taking me 1 second per page ugh
01:19:45Czechball quits [Quit: Quit: Leaving]
01:21:47<nicolas17>200k files and I don't know how far I am
01:23:33<nicolas17>there are duplicated filessss
01:24:03<@JAA>I once listed a bucket that took almost two months and produced a couple hundred GB of compressed JSONL. Yep, that's how it goes. :-)
01:24:39<nicolas17>example (2MB):
01:24:40<nicolas17>c20d4d80a75defb0364ca43f14486f58cd210597 423581_557ce44a-1597-4555-8221-557bec85c2f0.png
01:24:41<nicolas17>c20d4d80a75defb0364ca43f14486f58cd210597 423581_713af7b0-2fbd-43ef-a963-d9821cf2a95a.png
01:25:02@JAA pretends to be surprised.
01:25:19<@JAA>.png sounds like it might be a thumbnail or similar?
01:26:39<nicolas17>https://media.parti.com/423581_713af7b0-2fbd-43ef-a963-d9821cf2a95a.png (bad content-type)
01:28:03<nicolas17>...really curious what that decimal number before the uuid means
01:28:22etnguyen03 quits [Client Quit]
01:29:57<nicolas17>1 million files, 630GiB, still going
01:30:19<@JAA>Yeah, I wouldn't be surprised if this were pretty big, especially if it's HLS with segments.
01:30:43<nicolas17>so far it's all images
01:30:44<nicolas17>???
01:34:41<@JAA>I just found that netcup employs Anubis, by the way. Custom message 'This site is protected by <a href="https://www.anexia.com/">ANEXIA</a>.' and the anime girl image is a slow 403.
01:38:51<nicolas17>1TiB
01:40:24<nexussfan>JAA: custom versions of anubis exists, but AFAIK they all use .within-website
01:42:19<nicolas17>oh god I got to the video streams, this is huge
01:45:19<nicolas17>/channels contains 298 dev-channel-<id> subdirectories which seems... strangely low?
01:47:00<nicolas17>"the web3 creator economy & live stream platform" okay having <300 active users makes more sense now
01:47:42<nicolas17>how come they're using google cloud storage instead of filecoin? :P
01:52:18<mystique_altrosky>cause they need something that actually works
01:57:30etnguyen03 (etnguyen03) joins
02:09:10tzt quits [Remote host closed the connection]
02:09:29tzt (tzt) joins
02:11:09nathang2184 quits [Quit: Ping timeout (120 seconds)]
02:11:28nathang2184 joins
02:19:00nathang2184 quits [Ping timeout: 256 seconds]
02:26:16<nicolas17>list has been running for an hour, 7M files, 5.2TiB, still going
02:26:41beardicus quits [Ping timeout: 272 seconds]
02:26:43<nicolas17>I started from scratch on my VPS which has better latency to google, I'm optimistic it will catch up with my local PC long before it finishes
02:27:50beardicus (beardicus) joins
02:30:17nathang2184 joins
02:30:48Czechball joins
02:31:13Doomaholic (Doomaholic) joins
02:34:21jason quits [Read error: Connection reset by peer]
02:34:46jason joins
02:38:27Wohlstand quits [Quit: Wohlstand]
02:38:27nathang2184 quits [Read error: Connection reset by peer]
02:38:38nathang2184 joins
02:45:40ducky quits [Ping timeout: 260 seconds]
02:46:19nathang2184 quits [Ping timeout: 272 seconds]
02:47:28ducky (ducky) joins
02:48:02sg72 quits [Remote host closed the connection]
02:49:11sg72 joins
02:59:40ducky quits [Ping timeout: 260 seconds]
03:01:35ducky (ducky) joins
03:04:10<nicolas17>21M files, 17.6TiB, still going
03:06:40ducky quits [Ping timeout: 260 seconds]
03:07:50Island quits [Read error: Connection reset by peer]
03:09:11nathang2184 joins
03:11:53ducky (ducky) joins
03:13:33cultpony quits [Ping timeout: 272 seconds]
03:14:16cultpony (cultpony) joins
03:14:49nathang2184 quits [Ping timeout: 272 seconds]
03:15:56<nicolas17>I estimate 60TB but I could be way off
03:16:56ducky quits [Ping timeout: 260 seconds]
03:24:23<h2ibot>BlankEclair edited List of websites excluded from the Wayback Machine/Partial exclusions (+55, …): https://wiki.archiveteam.org/?diff=58207&oldid=58146
03:27:50etnguyen03 quits [Client Quit]
03:30:30nathang2184 joins
03:32:11Lord_Nightmare quits [Quit: ZNC - http://znc.in]
03:33:24ducky (ducky) joins
03:35:49Lord_Nightmare (Lord_Nightmare) joins
03:37:37nathang2184 quits [Ping timeout: 272 seconds]
03:40:27etnguyen03 (etnguyen03) joins
03:44:51nathang2184 joins
03:59:21<nicolas17>ok finished channels/ and it seems there's more directories, so all bets are off now
03:59:36DogDisco joins
04:01:19<@JAA>Welcome to S3 bucket listing.
04:04:19etnguyen03 quits [Remote host closed the connection]
04:13:24ducky quits [Ping timeout: 260 seconds]
04:17:59<nicolas17>https://storage.googleapis.com/parti-livestream/?prefix=ivs 9% on this directory now
04:23:05<nicolas17>anyway I'm not sure if archiving this is feasible or useful
04:23:54Webuser669590 joins
04:24:04<nicolas17>by the time I'm done running the list, some videos from >30 days ago may have gotten removed, and there may be some new ones
04:24:14Webuser669590 quits [Client Quit]
04:24:15<nicolas17>I think *live* streams are in this same bucket even (so new HLS segment files are being added every 2 seconds)
04:30:38<Guest>nicolas17: ive seen that livestreams have ids assigned to them (maybe sequentially?), so i think it might have something to do with that
04:31:10<nicolas17><nicolas17> ...really curious what that decimal number before the uuid means
04:31:15<nicolas17>I suspect it's the *user* ID
04:31:27<Guest>also possible
04:31:48<nicolas17>because under channels/ there's 300 directories with similar numbers, each of them having multiple videos with a timestamp
04:32:02<nicolas17>so that number is the user/channel ID, not the video ID
04:35:01ducky (ducky) joins
04:36:50SootBector quits [Remote host closed the connection]
04:37:57SootBector (SootBector) joins
04:39:59<Guest>only 300?
04:40:34<nicolas17><nicolas17> "the web3 creator economy & live stream platform" okay having <300 active users makes more sense now
04:40:42<Guest>thats like what kick.com tried to do with twitch
04:41:29<nicolas17>note that if a user hasn't streamed in the last month (so saved streams already expired) and/or doesn't save stream recordings, I won't see the directory as existing at all
04:41:41<Guest>imo theres not much of a point in archiving (especially if theres only 300 channrls)
04:41:49<Guest>i thought the bucket was a lot smaller
04:42:24<nicolas17>I don't even know what's in ivs/, maybe it's similar to "clips"? in which case there's a *ton*
04:42:57<Guest>what are the file formats?
04:44:40ducky quits [Ping timeout: 260 seconds]
04:45:20ducky (ducky) joins
04:48:43<nicolas17>HLS
04:48:56<nicolas17>might be they used a different system for past streams
04:49:01<nicolas17>ivs might mean https://aws.amazon.com/ivs/
04:54:37<nicolas17>maybe the last month of streams is in channels/<id>/archive/<timestamp> but older stuff is in ivs? maybe they migrated systems around that time? 2025/10/27 is the most recent timestamp I see in ivs
04:57:40<Guest>parti was founded in 2017 and ivs was created in 2020, unless they changed the architecture since then (the site is really empty for an 8 year old company)
04:59:46<Guest>its kind of pointless whether they used ivs or not though
05:02:07<nicolas17>it seems ivs/ has stream recordings older than ~2025-10-27... whether they changed systems at that date, or they move them there after a month, doesn't really matter
05:03:44<nicolas17>either way my extrapolation so far says ivs/ is 100TB :p
05:05:26<nicolas17>and considering https://tracker.archiveteam.org/twitch/ was 550TB...
05:06:22<Guest> based on https://ivs.rocks/calculator i dont think they would use ivs
05:06:40<nicolas17>ah well that's for streaming
05:06:53<nicolas17>when the stream is over you can shove it into a regular S3 bucket
05:12:11<nicolas17>paying for 100TB of GCP storage is still no laughing matter tbh
05:15:55sec^nd quits [Remote host closed the connection]
05:16:15sec^nd (second) joins
05:23:52ducky quits [Ping timeout: 260 seconds]
05:26:04ducky (ducky) joins
05:31:20ducky quits [Ping timeout: 260 seconds]
05:31:23<nicolas17>Guest: does 1508 users sound more reasonable?
05:34:09jason quits [Ping timeout: 272 seconds]
05:34:35jason joins
05:36:37ducky (ducky) joins
06:09:36nexussfan quits [Quit: Konversation terminated!]
06:31:10jason quits [Read error: Connection reset by peer]
06:31:35jason joins
06:34:03jason quits [Read error: Connection reset by peer]
06:34:18jason joins
06:58:31Wohlstand (Wohlstand) joins
07:18:53hexagonwin quits [Read error: Connection reset by peer]
07:19:47jason quits [Remote host closed the connection]
07:19:59hexagonwin joins
07:23:00gosc joins
07:26:12<gosc>I can just make a page for the wiki right? I want to add Microsoft Store on there
07:26:29<gosc>it could also be a project in the future, like chrome extensions
07:51:57Dada joins
08:38:56ducky quits [Ping timeout: 260 seconds]
08:40:52ducky (ducky) joins
08:51:13AK (AK) joins
08:59:53ducky_ (ducky) joins
08:59:56ducky quits [Ping timeout: 260 seconds]
09:00:31ducky_ is now known as ducky
09:10:12Wohlstand quits [Client Quit]
09:25:19Stagnant_ quits [Ping timeout: 272 seconds]
09:32:16ducky_ (ducky) joins
09:33:04ducky quits [Ping timeout: 260 seconds]
09:33:13ducky_ is now known as ducky
09:44:44ducky quits [Ping timeout: 260 seconds]
09:46:55ducky (ducky) joins
09:54:32ducky quits [Ping timeout: 260 seconds]
09:57:12ducky (ducky) joins
10:14:55Stagnant_ (Stagnant) joins
10:34:21gosc quits [Ping timeout: 272 seconds]
10:44:29Wohlstand (Wohlstand) joins
10:45:52ducky quits [Ping timeout: 260 seconds]
10:47:39ducky (ducky) joins
11:03:08ducky quits [Ping timeout: 260 seconds]
11:05:28ducky (ducky) joins
11:12:56ducky quits [Ping timeout: 260 seconds]
11:12:57ducky_ (ducky) joins
11:13:35ducky_ is now known as ducky
11:18:12evergreen59 joins
11:21:18evergreen5 quits [Ping timeout: 256 seconds]
11:21:18evergreen59 is now known as evergreen5
11:22:26Dada quits [Ping timeout: 256 seconds]
11:25:16arch quits [Ping timeout: 256 seconds]
11:28:14arch (arch) joins
11:34:24ducky quits [Ping timeout: 260 seconds]
11:35:15ymgve__ joins
11:36:23ducky (ducky) joins
11:38:35Webuser446493 joins
11:38:42<Webuser446493>Hi. May I ask for this site to be archived? http://www.whhansu.com. This is a contractor related to the big HK fire and want to preserve evidence related to the fire before they start deleting.
11:38:43<Webuser446493>[1] https://www.bbc.com/news/articles/cp34yk7z6yxo
11:39:35ymgve_ quits [Ping timeout: 272 seconds]
11:50:49<@JAA>Webuser446493: I've started an ArchiveBot job for it.
11:53:41<Webuser446493>Thank yuo
11:53:43<Webuser446493>Thank you!
11:56:56APOLLO03a joins
11:59:50APOLLO03 quits [Ping timeout: 256 seconds]
12:09:44<h2ibot>Manu edited Discourse/archived (+100, Queued forum.sublimetext.com): https://wiki.archiveteam.org/?diff=58208&oldid=58196
12:09:59BornOn420 quits [Ping timeout: 272 seconds]
12:14:04ducky quits [Ping timeout: 260 seconds]
12:15:55ducky (ducky) joins
12:23:37hexagonwin_ joins
12:25:11hexagonwin quits [Ping timeout: 272 seconds]
13:03:04ducky quits [Ping timeout: 260 seconds]
13:05:04ducky (ducky) joins
13:08:13Webuser446493 quits [Client Quit]
13:24:04ducky quits [Ping timeout: 260 seconds]
13:24:51Webuser539401 joins
13:25:02Webuser539401 quits [Client Quit]
13:27:05ducky (ducky) joins
13:34:20ducky quits [Ping timeout: 260 seconds]
13:44:59<nicolas17>parti listing took ~7 hours
13:45:39<nicolas17>174523 GiB, 197M files
13:47:55ducky (ducky) joins
14:07:00twiswist quits [Read error: Connection reset by peer]
14:07:12twiswist (twiswist) joins
14:10:04croissant` joins
14:12:25Wohlstand quits [Remote host closed the connection]
14:13:34croissant_ quits [Ping timeout: 256 seconds]
14:29:50Webuser278032 joins
14:30:02Webuser278032 quits [Client Quit]
14:45:56sg72 quits [Remote host closed the connection]
14:47:06sg72 joins
15:02:07Bestest joins
15:03:31lennier2 quits [Ping timeout: 272 seconds]
15:03:58<Bestest>Apologies if this is the wrong channel to ask; in regards to https://wiki.archiveteam.org/index.php/Fileplanet https://archive.org/details/archiveteam-fileplanet , is there any better way to get in contact with Spirit besides their email?
15:04:16lennier2 joins
15:04:23<Bestest>They did respond eventually, but it took 3 months and as a result I missed it at the time. I was wondering if there was any better place to communicate with them
15:11:25BornOn420 (BornOn420) joins
15:13:03dendory quits [Quit: The Lounge - https://thelounge.chat]
15:14:03phuzion (phuzion) joins
15:15:16<h2ibot>Manu edited Discourse (+43, Active Discourses: Add GitLab Forum): https://wiki.archiveteam.org/?diff=58209&oldid=57815
15:15:19sg72 quits [Remote host closed the connection]
15:17:02nathang2184 quits [Ping timeout: 256 seconds]
15:18:07sg72 joins
15:20:12phuzion quits [Client Quit]
15:21:11phuzion (phuzion) joins
15:22:41phuzion quits [Client Quit]
15:22:49phuzion (phuzion) joins
15:35:05dendory (dendory) joins
16:05:44nathang2184 joins
16:16:03justauser joins
17:08:49<justauser|m>Poked the remains of https://whyfiles.org.
17:09:42<justauser|m>Its former NS still answers the queries for this name with its last IP, but the server on this IP, as well as others on the same subnet, don't respond usefully.
17:09:52APOLLO03a quits [Quit: .]
17:17:49Wohlstand (Wohlstand) joins
17:29:44Wohlstand quits [Client Quit]
17:39:50justauser|m quits [Quit: Reconnecting]
17:40:01justauser|m (justauser|m) joins
17:48:11Webuser319042 joins
17:48:21NeonGlitch quits [Quit: Textual IRC Client: www.textualapp.com]
17:48:35Webuser319042 quits [Client Quit]
18:06:45Dada joins
18:19:12phuzion quits [Client Quit]
18:19:45kansei quits [Quit: ZNC 1.10.1 - https://znc.in]
18:21:49kansei (kansei) joins
18:50:53Guest58 quits [Ping timeout: 272 seconds]
18:53:20Guest58 joins
19:08:35DogDisco leaves [Ooops, wrong browser tab.]
19:11:15@imer quits [Quit: Oh no]
19:12:22sec^nd quits [Remote host closed the connection]
19:12:44sec^nd (second) joins
19:13:25NeonGlitch (NeonGlitch) joins
19:14:55imer (imer) joins
19:14:55@ChanServ sets mode: +o imer
19:39:45APOLLO03 joins
19:52:09phuzion (phuzion) joins
19:56:36<phuzion>Hey, I noticed this: https://wiki.archiveteam.org/index.php/Running_Archive_Team_Projects_with_Docker#Can_I_run_a_Docker_container_on_ARM_or_some_other_unusual_architecture? Is there a bug that I can follow somewhere in case this endianness issue does get resolved?
20:05:54kansei quits [Client Quit]
20:11:03kansei (kansei) joins
20:13:51pokechu22 quits [Ping timeout: 272 seconds]
20:18:17Guest58 quits [Ping timeout: 272 seconds]
20:19:53Guest58 joins
20:22:37pokechu22 (pokechu22) joins
20:30:05<@JAA>phuzion: Probably https://github.com/ArchiveTeam/warrior-dockerfile/issues/56
20:34:42PredatorIWD25 quits [Read error: Connection reset by peer]
20:39:00ducky quits [Ping timeout: 260 seconds]
20:41:51PredatorIWD25 joins
20:46:47Guest58 quits [Ping timeout: 272 seconds]
20:49:00DogsRNice joins
20:52:57Guest58 joins
20:56:17nathang2184 quits [Ping timeout: 272 seconds]
20:57:24<phuzion>JAA: Thanks, subscribed!
20:59:47PredatorIWD25 quits [Read error: Connection reset by peer]
21:02:11Webuser873658 joins
21:02:17Webuser873658 quits [Client Quit]
21:02:36PredatorIWD25 joins
21:07:56PredatorIWD25 quits [Read error: Connection reset by peer]
21:10:04PredatorIWD25 joins
21:16:49nathang2184 joins
21:19:43Guest58 quits [Ping timeout: 272 seconds]
21:21:39ATinySpaceMarine quits [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.]
21:22:09Guest58 joins
21:32:13^ quits [Read error: Connection reset by peer]
21:32:23^ (^) joins
21:39:35Dada quits [Remote host closed the connection]
21:49:49<nicolas17>phuzion: well ARM has the same endianness as x86
21:50:26<nicolas17>the issue is more "if we *know* it broke due to big endian in the past, there may be other arch-specific bugs that we don't know about"
22:01:38ducky (ducky) joins
22:02:02<Guest>nicolas17: depends on the duration and resolution of each stream
22:02:58<nicolas17>Guest: sorry I meant if "1508 users" sounds like a more realistic number than "300 users"
22:03:07<nicolas17>in terms of archival it makes things worse :P
22:17:24kansei quits [Client Quit]
22:19:14kansei (kansei) joins
22:20:10Webuser915956 joins
22:20:36Webuser9159563 joins
22:21:31Webuser915956 quits [Client Quit]
22:21:31Webuser9159563 quits [Client Quit]
22:27:20<Guest>for 1508 users thats 115GiB/user
22:28:24<Guest>based on https://snxpstudio.co/resources/video-file-size-calculator/ @ 1080p, 30fps, H.264, 8mbps ("default") video bitrate, stereo sound, 192kbps ("default") audio bitrate, and 3h of video thats 10.79GiB/video
22:30:29<Guest>so like 10 videos/user, but theres only 25 live right now
22:31:48<Guest>i think the number of active streamers is a lot lower and maybe a few hundred for people who tried it out on a few streams and quit
22:33:20<Guest>so tl;dr there may be 1508 users, but there definitely isnt 1508 streamers
22:37:59<@arkiver>Guest: is this going away?
22:41:08<nicolas17>arkiver: my understanding is: "I found this open bucket, can we archive it?" "seems it has stream VODs and it's 100TB" "oh x_x nevermind"
22:45:01<@arkiver>right yeah
22:45:32<Guest>no the parti website isnt shutting down soon
22:45:53<Guest>i think nicolas17 explained it well
22:52:18Webuser239872 joins
22:52:38Webuser239872 quits [Client Quit]
22:53:36ATinySpaceMarine joins
22:55:21Guest58 quits [Ping timeout: 272 seconds]
22:56:43Guest58 joins
23:02:16nexussfan (nexussfan) joins
23:04:06etnguyen03 (etnguyen03) joins
23:05:15<klea>JAA: should i mark AT-github/warrior-dockerfile#56 as a <ref> on some part of that page?
23:05:26<klea>ig no point in not doing so
23:05:59<klea>Guest: preferably not say you subscribed (it could be spammy)
23:08:01<Guest>what?
23:08:01Guest58 quits [Ping timeout: 272 seconds]
23:08:30<h2ibot>Klea edited Running Archive Team Projects with Docker (+113, /* Can I run a Docker container on ARM or some…): https://wiki.archiveteam.org/?diff=58210&oldid=56992
23:08:44<klea>Guest: sorry, i thought you said https://github.com/ArchiveTeam/warrior-dockerfile/issues/56#issuecomment-2314348097
23:09:07<Guest>oh ok
23:09:22<klea>it'd be neat if someone marked it as spam, but idk who can :(
23:11:43Guest58 joins
23:14:30<h2ibot>Nicolas17v2 edited 짱공유닷컴 (+1, update rate limit): https://wiki.archiveteam.org/?diff=58211&oldid=57883
23:21:37klea is thinking of maybe setting up her own irc logger if AT authorizes, which would collect private irc logs, usable by current ops (i would give a bunch of passwords to one of the current ones, who would then be able to share one with each user who wants or smth), and would potentially have a way for authorized people to tell the bot to join, leave, or make channel
23:21:39klea logs (or sections) public
23:24:28<nicolas17>we have https://irclogs.archivete.am/
23:24:31<klea>i'll likely make a channel specific for that bot, and then give !archiveteam-core +*SF, and !archiveteam-ops +v, making a log public, or asking the bot to archive logs for a channel on archive.org, would require +o access
23:24:39<klea>but that doesn't archive projects
23:24:42<klea>which was most of the idea
23:24:51<@JAA>We don't want to log project channels.
23:25:03<nicolas17>if we wanted to log project channels, we might as well add that bot to more channels rather than set up a whole new logger
23:25:05klea wonders why it wouldn't be ok to log them and publish them ONLY after the website is dead
23:25:09<klea>yeah true
23:25:17<@JAA>Yeah, it'd be a one-line diff to add it.
23:25:29<@JAA>(+ a git push and a SIGUSR1)
23:25:43klea wonders where the source code for that is
23:25:52<klea>wait, JAA it's using h2ibot ?
23:26:18<@JAA>Not quite, but close. It's based on the same code.
23:26:49<klea>is it public?, i guess no
23:26:53<@JAA>https://gitea.arpa.li/ArchiveTeam/irclog
23:27:29<klea>oh nice
23:32:17<klea>btw, if y'all have any discord cdn urls and don't have a valid expiry thingy, you can send it trough discord, and it creates a updated expiry time, if wanted i can try to create webhooks or smth to have something to be able to make accesible urls
23:33:04<nicolas17>DigitalDragon: https://fixit.digitaldragon.dev/ fix it D:
23:33:14<klea>lfmao
23:38:12<klea>oh, im stupid
23:38:19<klea>i was wondering what the bot's name was

HomeSearchPrevious dayNext day