HomeSearchPrevious dayNext day

00:19:40<tzt>http://ge.tt is shutting down 28th of February
00:27:45<@kiska>Here is a test file I uploaded: http://ge.tt/6CMOUZB3
00:28:10<@kiska>Seems to be 8 chars with 0-9 and A-Z
00:28:32<@kiska>Here is another http://ge.tt/3u8QUZB3
00:28:56<@kiska>So 8 characters of 0-9 A-Za-z
00:30:36<@kiska>arkiver: fun times?
00:33:00<purplebot>Deathwatch edited by Kiska (+128, Add ge.tt) just now -- https://www.archiveteam.org/?diff=46365&oldid=46359
00:36:13hooway quits [Client Quit]
00:38:37<SCSi>doesnt give us much time to archive
00:43:07<@JAA>[0-9a-zA-Z]{8} = ~218 trillion combinations. Yeah, not going to happen.
00:46:30britmob2 quits [Client Quit]
00:46:50britmob2 joins
00:51:40<Ryz>It may be shutting down on 2021 February 28, but the files will not available on 2021 March 10,
00:51:47<Ryz>So that's more time than just 2 days
01:09:19daalekz (daalekz) joins
01:20:07daalekz quits [Client Quit]
01:35:21Arcorann (Arcorann) joins
01:43:34Arcorann_ joins
01:46:14Arcorann quits [Ping timeout: 260 seconds]
01:48:20@Fusl quits [Excess Flood]
01:48:50Fusl (Fusl) joins
01:48:50@ChanServ sets mode: +o Fusl
01:49:49<SCSi>so when we rallying the troops and knocking this one out
01:49:57<SCSi>i wanna show up on the leaderboards not in the bottom 50
02:05:30<tech234a>http://api.ge.tt/
02:05:38<tech234a>http://api.ge.tt/main/getStats/
02:05:50<tech234a>still not enough but possibly helpful
02:11:46<tech234a>http://ge.tt/developers/docs/rest
02:12:08<tech234a>http://ge.tt/developers/rest
02:18:08<@arkiver>hi
02:18:09<@arkiver>ah
02:18:19<@arkiver>fun fun fun
02:18:54<@arkiver>lets go 218 trillion URLs :)
02:19:07<@arkiver>JAA: could you do a quick twitter scan?
02:20:41<@arkiver>i'll send them an email ow
02:20:42<@arkiver>no
02:20:43<@arkiver>ow
02:20:45<@arkiver>now*
02:25:15<atphoenix>multiple files can be behind one base URL, e.g. http://ge.tt/64SMJpf has 5 files including http://ge.tt/64SMJpf/v/81
02:28:47<atphoenix>^ url found via a search engine searching for site:get.tt
02:28:55Arcorann_ quits [Ping timeout: 264 seconds]
02:30:22<tech234a>Only 53 million files
02:33:04<atphoenix>the example I gave is for SweetFX ReShade files. I wonder if we can find forums that liked to use this file hoster.
02:33:57<tech234a>Plenty of URLs already in Google and CDX
02:34:29<tech234a>Bing too
02:35:03<atphoenix>duckduckgo too. My URL example is used here: http://forum.doom9.org/showthread.php?t=170357
02:41:42<@arkiver>53 million is not much, but these may be large
02:48:46Sylirana quits [Read error: Connection reset by peer]
02:48:55Sylirana (Sylirana) joins
02:55:45Arcorann (Arcorann) joins
02:57:30fuzzy8021 quits [Read error: Connection reset by peer]
02:57:51fuzzy8021 joins
02:57:51fuzzy8021 quits [Changing host]
02:57:51fuzzy8021 (fuzzy8021) joins
03:18:59Arcorann quits [Ping timeout: 260 seconds]
03:22:07ragu_ joins
03:22:13ragu__ quits [Read error: Connection reset by peer]
03:31:04DopefishJustin quits [Remote host closed the connection]
04:01:26Mineroboter_ joins
04:03:54Mineroboter quits [Ping timeout: 260 seconds]
04:24:07katocala quits [Ping timeout: 264 seconds]
04:24:40katocala joins
04:37:55DogsRNice quits [Read error: Connection reset by peer]
04:40:14qw3rty__ joins
04:43:55qw3rty_ quits [Ping timeout: 264 seconds]
05:00:17<atphoenix>there are a bunch of forums and reddit posts that reference my example. https://duckduckgo.com/?q=http%3A%2F%2Fge.tt%2F64SMJpf . Hopefully they can just give us the list of 53m...
05:51:00<purplebot>URLTeam edited by Aarchi (+245, Update git.io) just now -- https://www.archiveteam.org/?diff=46366&oldid=46362
06:02:41DopefishJustin joins
08:02:26lennier2 joins
08:02:42BlueMaxima quits [Client Quit]
08:05:59lennier1 quits [Ping timeout: 260 seconds]
08:06:06lennier2 is now known as lennier1
08:49:42<OrIdow6^2>So NicoNino now has about a day and a half before deleting past comments
08:53:17<OrIdow6^2>The script I wrote is at https://github.com/OrIdow6/niconino-grab
08:53:53<OrIdow6^2>As far as I know, it is completely done except for zstd and a version number bump
08:55:49<OrIdow6^2>There are about 23 million videos this would need to cover to be complete, and it is heavily rate-limited during the day in Japan
08:59:47<OrIdow6^2>Can one of the ops look at this? I can add a static dictionary if need be.
09:30:57Sylirana quits [Ping timeout: 244 seconds]
09:31:55Sylirana (Sylirana) joins
09:35:42<@EggplantN>jfc tech234a thats a lotta items 🙃
09:37:18<@kiska>arkiver: See OrIdow6^2 pls
09:39:33auctus joins
09:44:55<@EggplantN>oh hey OrIdow6^2
09:44:56<@EggplantN>are you here?
09:44:58<atphoenix>if you want a project channel, may I suggest nicotino (going up in smoke) or NicoNinoNoMore or
09:45:01<@EggplantN>whatcha need from us ops
09:45:27<@EggplantN>need a tracker + items loading? or need someone to do a code review.
09:47:22<@EggplantN>Tracker is up
09:47:40<@EggplantN>set min version to 20210227.01
09:49:00<@EggplantN>target added (rsync only)
09:49:37<@EggplantN>also r.e ZSTD was this something required? or just would like?
09:49:51<@EggplantN>also r.e rate limiting we can deploy 2k+ IPs + hetzner cloud
09:53:03<OrIdow6^2>Away for like 20 more minutes
09:54:05<@EggplantN>okie fyi zstd isnt much use on video afaik
09:54:11<@EggplantN>so I believe we can ignore that (?)
09:55:35<@EggplantN>gonna go for a shower will be back similar time
09:58:21<@kiska>Also :D https://server8.kiska.pw/uploads/5fbfe2ea82053400/image.png
10:03:00<@EggplantN>Is this what were archiving
10:04:04<@kiska>no this is from the IA talk that Jonah did
10:04:56treora quits [Read error: Connection reset by peer]
10:05:07treora joins
10:15:34<@EggplantN>Okay few questions for OrIdow6^2 when you're back
10:15:34<@EggplantN>a) are we archiving video or just text
10:15:34<@EggplantN>b) if video then not much point for zstd?
10:15:34<@EggplantN>c) if video any reason for the multi item size of 30?
10:17:12sliccricc_ (sliccricc) joins
10:19:24sliccricc quits [Remote host closed the connection]
10:21:43Ruthalas quits [Ping timeout: 264 seconds]
10:27:05Ruthalas (Ruthalas) joins
10:31:16<OrIdow6^2>EggplantN: Zstd is useful, because it's not getting videos themselves, just metadata pages & comments
10:31:30<OrIdow6^2>So (a) is "just text"
10:31:37<@EggplantN>okie all is fine
10:31:51<@EggplantN>we would need to likely wait for arkiver for zstd + deploy a different target
10:31:53<OrIdow6^2>Multi-item may actually be useful being increased
10:33:02<OrIdow6^2>Hm
10:34:28<@EggplantN>Do you have the item list at least for me to load that into the tracker ready?
10:34:29<@kiska>Any reason why we aren't archiving video?
10:34:48<OrIdow6^2>That's why I wondered why a static dict was possible
10:35:17<OrIdow6^2>EggplantN: No, going to work on that, need to decide whether to try all possible IDs or get a subset from the search page
10:35:31<@EggplantN>We can just throw all possible in no problem
10:35:41<OrIdow6^2>kiska: Because it's not being deleted; only old comments and metadata of deleted videos are being removed
10:35:47<@kiska>I see...
10:36:41<@EggplantN>okay no worries. The targets to zstd the warcs uploaded but if you want to do zstd on the worker that needs arkiver to configure it on researcher7
10:37:34<OrIdow6^2>Thinking about it, it may not be too big with gzip
10:37:53<@EggplantN>How big would per item be?
10:38:51<OrIdow6^2>I'm going to say something on the order of 30 KB
10:39:12<OrIdow6^2>Good chance it's less than that
10:39:40<@EggplantN>at 30KB 23mil items its only 660GB
10:39:54<OrIdow6^2>Hard to tell exactly, because I don't know the distribution of comments/video and used IDs
10:39:55<OrIdow6^2>Yeah
10:41:55<OrIdow6^2>Running item generation now
10:41:58<@kiska>Lets put this on atr3 if you're going to do gz or with a static dict
10:42:26<OrIdow6^2>As EggplantN has made me realize, probably just gz
10:43:17<@EggplantN>perfect
10:44:03<@EggplantN>Please PR the code to AT GH repo when ready, kiska has prepared docker to build. target is ready to just accept data (not send to IA yet). Send me the items or make a GH repo called niconino-items (preferred) and I'll fork to AT
10:44:06<@EggplantN>and add to tracker
10:44:22<@EggplantN>I'll also default this project on the warrior
10:46:43<@EggplantN>wait hold up OrIdow6^2
10:46:48<OrIdow6^2>What
10:46:52<@EggplantN>is this project called niconico
10:46:54<@EggplantN>or niconino
10:47:11<@EggplantN>wiki says niconico
10:47:29<OrIdow6^2>Oh, did I mispell it again?
10:47:43<@EggplantN>lmao we've set everything for niconino lol
10:47:50<@EggplantN>I can move it over to niconico
10:48:32<OrIdow6^2>I'll change the repo
10:48:43<OrIdow6^2>I did this before, caught it myself though
10:49:07<@EggplantN>lol I already called it ninonino on tracker before realising now noticed when getting the logo its niconico
10:51:44<@EggplantN>okie added to tracker frontpage
10:51:56<@EggplantN>peoples notifications will be alerting shortly
10:52:20<@EggplantN>https://irc.fu.is/uploads/711b06336a00afb8/image.png
10:52:21<@EggplantN>there we go
10:54:00<purplebot>Niconico edited by Tglass (+47, Update IRC channel) just now -- https://www.archiveteam.org/?diff=46367&oldid=46321
10:54:08Arcorann (Arcorann) joins
11:00:47<OrIdow6^2>Good thing you caught that
11:00:53<OrIdow6^2>Here you are EggplantN: https://github.com/ArchiveTeam/niconino-grab/pull/1
11:01:01<OrIdow6^2>Should probably rename the repo, too
11:01:06<@EggplantN>yeah
11:01:07<@EggplantN>lemme do that
11:01:30<@EggplantN>merged changed name
11:02:09Arcorann quits [Ping timeout: 260 seconds]
11:02:29<AK>Is there a niconinoniconino channel yet?
11:04:08OrIdow6^2 is now known as OrIdow6
11:04:10<OrIdow6>Thank you
11:04:56<@EggplantN>Not yet AK
11:04:59<@EggplantN>just here for now ;)
11:07:11hooway joins
11:09:22<OrIdow6>Having to take a bit of a detour with items
11:09:45<@EggplantN>okie lemme know if you wanna just yeet all possible in
11:11:04<OrIdow6>There is a more efficient way than all possible
11:11:28<OrIdow6>But it will take time, and I don't know whether this will go fast enough that prioritization isn't necessary
11:12:42<OrIdow6>Alright, here you have it: https://github.com/OrIdow6/niconico-items
11:13:15<OrIdow6>Go slow at first, and it's possible that I omitted something with multiitems
11:14:55<@EggplantN>Okie I'll test it first real quick
11:16:11<OrIdow6>Ok
11:18:32<OrIdow6>By the way, don't test with the very early IDs - sm8 has millions of comments
11:19:36<@EggplantN>its running and redis is random so i'll get whatever it gives
11:20:12<@EggplantN>58=503 https://www.nicovideo.jp/watch/sm95642
11:20:12<@EggplantN>Server returned 503 (RETRFINISHED). Sleeping.
11:20:55<OrIdow6>That's niconico's version of a 429
11:21:04<@EggplantN>Ah right
11:21:18<@EggplantN>so 503 is 429 here jesus im running 1 concurrent
11:21:23<@EggplantN>it slept for the right amount of time though
11:21:45<OrIdow6>I have it set for 1 minute on 503s
11:21:50<@EggplantN>perfect
11:22:00<@EggplantN>so this is a 1-2 concurrent per IP?
11:22:07<OrIdow6>Probably
11:23:06<@EggplantN>its bloody slow lol
11:23:23<AK>I get a 503 on that and I haven't visited niconico before
11:23:28<AK>I think that one might actually be a 503
11:23:51<AK>*404
11:23:58<@EggplantN>yeah 404's are fine
11:24:04<@EggplantN>but 503 = 429 here
11:24:09<@EggplantN>🙃
11:24:14<@kiska>OOF
11:24:17<@EggplantN>least they give error codes and not 200's
11:24:19<@EggplantN>for everything
11:24:23<@kiska>At least its better than halo
11:24:49<@EggplantN>okay it worked OrIdow6
11:25:00<@EggplantN>3.46MiB for 30 items
11:25:12<OrIdow6>Good
11:25:13<@EggplantN>~110kb but thats a small data set
11:25:20<@EggplantN>You happy for us to run with this?
11:25:25<@kiska>Maybe we should do 10 multi items if they give 503 for 429 :D
11:25:36<@EggplantN>i only got 1 503 kiska
11:25:41<@kiska>I see
11:25:41<@EggplantN>for a multi item of 30
11:25:48<OrIdow6>Can you loook at your logs? How many "watch_dll" and "watch_app" scripts did it get?
11:26:11<@EggplantN>instead
11:26:12<@EggplantN>https://paste.ee/p/dhCQw
11:26:16<@EggplantN>enjoy the whole log
11:26:16<OrIdow6>Oh, thanks
11:28:40<OrIdow6>So I think this looks like it's running ok for now
11:29:02<@EggplantN>ready for me to open the flood gates?
11:29:11<OrIdow6>I think so
11:29:13<OrIdow6>Start slowly
11:29:41<OrIdow6>The main site has a lot of capacity (at least at certain times of day), but I'm not sure about the history endpoint
11:29:50<@HCross>If you can hold off 20 minutes I can scale
11:29:55<OrIdow6>I.e. all the http://nmsg.nicovideo.jp/api requests
11:30:03<@EggplantN>ah right i've not put a rate limit on
11:30:06<@EggplantN>but we will as needed
11:30:47<@EggplantN>deployed 50 concurrent
11:32:16<AK>Error response from daemon: manifest for atdr.meo.ws/archiveteam/niconico-grab:latest not found: manifest unknown: manifest unknown
11:32:24<AK>It's niconico right?
11:32:30<@EggplantN>ah crap
11:32:31<@EggplantN>kiska
11:32:40<@kiska>Yes?
11:32:46<@EggplantN>i renamed repo
11:32:49<@EggplantN>can you redeploy drone
11:32:57<@EggplantN>also OrIdow6 they're intelligent and rate limiting me using a /24
11:32:59<@EggplantN>🙃
11:33:02<@kiska>Sync'ing
11:33:05<@kiska>One moment
11:33:26<@kiska>Building
11:34:07<@EggplantN>hrm
11:34:10<OrIdow6>EggplantN: Is it possible it could be by UA or something?
11:34:20<@EggplantN>its possible
11:34:22<@EggplantN>want me to try?
11:34:31<OrIdow6>But I suppose /24 rate limiting is what I'd expect from them
11:34:32<OrIdow6>Not yet
11:34:51<@EggplantN>i tried doing 50 concurrent 1/IP from the same /24
11:35:26<@HCross>I’ll deploy the usual solution in a bit
11:36:17<@EggplantN>even fucking 8 IPs
11:36:20<@EggplantN>instant 503
11:37:19<OrIdow6>Looking into it
11:37:42<AK>Exception: No usable Wget+At found.
11:37:43<AK>Looking for Wget+AT in ./wget-at
11:37:43<AK>./wget-at: Incorrect Wget+AT version (want ['GNU Wget 1.20.3-at.20210212.02']).
11:37:46<AK>Welp
11:37:54<@EggplantN>was that the docker CT?
11:37:56<AK>Yep
11:38:20<@EggplantN>uh]
11:38:26<@EggplantN>thats the latest verison
11:39:09<AK>https://pastebin.com/fvdVmj5P
11:39:28<@EggplantN>gimme a sec hrm,
11:40:05<@EggplantN>also OrIdow6 if you want an example of random UA's see reddit
11:40:15<OrIdow6>Ok
11:43:15<OrIdow6>Is anyone besides Eggplant running yet? Would help narrowing down the cause of the 503s
11:43:24<@EggplantN>yeah AK seems docker is using the old wget-at?
11:43:36<AK>Ooh
11:44:02<AK>grab-base vs grab-base-df?
11:44:15<AK>reddit uses
11:44:15<AK>FROM atdr.meo.ws/archiveteam/grab-base:gnutls
11:44:23<AK>This uses FROM atdr.meo.ws/archiveteam/grab-base-df
11:44:32<AK>Urls-grab uses FROM atdr.meo.ws/archiveteam/grab-base
11:44:35<@EggplantN>ah yes
11:44:39<@EggplantN>lemme update that
11:46:47ragu__ joins
11:50:31ragu_ quits [Ping timeout: 264 seconds]
12:03:13<@EggplantN>we're just debugging this now AK
12:03:16<@EggplantN>please hold off
12:20:32<AK>No worries, happy for me to leave them running with watchtower? Or turn them off?
12:21:12<@EggplantN>watchtower is good
12:45:12ragu__ quits [Read error: Connection reset by peer]
12:45:30ragu__ joins
13:16:35<@arkiver>OrIdow6: why is it running? i thought we wanted to wait for zstd and all
13:16:50<@arkiver>ah crap a day and a half
13:16:59<OrIdow6>arkiver: It was pointed out to me that it will likely be under a TB
13:17:07<@arkiver>OrIdow6: please feel free to spam me next time :)
13:17:47<OrIdow6>arkiver: Will do :)
13:17:48<@arkiver>i see the static user_id is still in, is this correct OrIdow6 ?
13:17:55<@arkiver>223 in Lua
13:20:32<@kiska>arkiver
13:20:33<@kiska>:D
13:20:42<@arkiver>hi
13:23:55<@EggplantN>hey arkiver we've gone ahead and just started seems smallish dealing with bugs right now with OrIdow6
13:23:58<@kiska>Would you like me to ping you once an hour for time critical things for this?
13:24:18<@EggplantN>also you'll be glad to know another people deserve to go to hell arkiver
13:24:27<@EggplantN>randomly returning 302/200 instead of 404's
13:27:27<@arkiver>yeah TB is fine without ZSTD
13:28:38<Jake>https://paste.ubuntu.com/p/Qhk38268RD/
13:29:22<Jake>Orldow6 ^
13:30:10<OrIdow6>Will look
13:30:29<Jake>Thank you!
13:31:46<@EggplantN>will need more items also soon OrIdow6
13:31:47<@EggplantN>:D
13:31:50<@arkiver>JAA: idea for your bot, how about it warns us for approaching deadlines on the deathwatch pages? :)
13:32:24<Jake>I'm not sure it's doing comment threads correctly?
13:32:26<@kiska>Ping at least 2 months in advanced
13:32:40<@kiska>And once per day
13:36:04<OrIdow6>Jake: Do the other getwaybackkey requests work? I.e., is it just this one time it's failing for you, or is it doing this all the time?
13:36:16<OrIdow6>And what do you think the problem with comment threads is?
13:36:50<OrIdow6>Wow, that was fast
13:36:59<Jake>I'm seeing very few getwaybackkey requests, maybe actually only the ones with the lua error?
13:37:14<OrIdow6>It is nighttime in Japan, so they have excess capacity
13:37:18<Jake>maybe every 100th video id it's calling getwaybackkey?
13:37:28<OrIdow6>That's expected
13:37:54<Jake>well, they are all erroring out like that paste I posted above.
13:38:02<OrIdow6>Anyone else having this problem?
13:41:28<OrIdow6>EggplantN: How many items to a batch?
13:41:37<@EggplantN>uh
13:41:42<@EggplantN>1 mil?
13:41:45<OrIdow6>Ok
13:43:17<@arkiver>OrIdow6: how many items in total?
13:43:38<OrIdow6>arkiver: 20-30 million
13:45:53<@arkiver>guess we can add all of them
13:45:58<@arkiver>take the upper range
13:46:05<@arkiver>upper limit of the range
13:46:13<OrIdow6>That's what's going on right now
13:46:25<OrIdow6>They're not all sequential (there are other types besides "sm"), but most are
13:47:04<OrIdow6>vid:sm[N] where [N] is from 1 to 38400400
13:48:11<OrIdow6>So do you want one big file, or 39 small ones?
13:48:36<OrIdow6>Anyhow, some more nuanced discovery will be needed for non-SMs
13:49:19<@arkiver>what are the non-SMs
13:49:38<@arkiver>OrIdow6: whatever is easier for you, i'd keep it to ~1 million per file?
13:49:42<OrIdow6>Video IDs have prefixes, the most common type are "sm" (submitted by regular user)
13:49:50<OrIdow6>Ok
13:50:00<OrIdow6>The page on the AT wiki discussed it
13:51:14<@EggplantN>also OrIdow6 yes I saw the bug Jak e did
13:51:39<OrIdow6>Looks like it's something transient
13:51:57<Jake>I got it again as well, more full logs: https://paste.ubuntu.com/p/42MtG6R3bX/
13:52:51<OrIdow6>How often is this happening?
13:53:29<OrIdow6>And is it necessary that I generate all 38 million items, versus someone loading them into the tracker directly?
13:53:32<Jake>from my limited logs, every time it tries to do getwaybackkey
13:53:43<OrIdow6>650 MB of sequential numbers
13:53:57<OrIdow6>Then retrying isn't useful
13:54:19<OrIdow6>EggplantN: When this happens to you, does it happen all the time?
13:54:24<OrIdow6>Or just occasionally?
13:54:30<@EggplantN>occasionally
13:58:43<OrIdow6>Should there be a channel for this?
13:59:18<OrIdow6>How about #niconino
14:00:15<@arkiver>OrIdow6: please do generate all of them and put them in -items
14:00:17<@arkiver>for logs
14:01:48<OrIdow6>arkiver: https://github.com/ArchiveTeam/niconico-items/pull/1
14:01:59<OrIdow6>(I'm sure Github loves this kind of commit)
14:02:15<OrIdow6>Also, I see that 00 has not been marked added
14:03:28<OrIdow6>Anyhow, I seem to have a habit of starting "small" projects that spam up -bs, please come to #niconino
14:03:38<@arkiver>we're cooperating with github and IA on #gitgud
14:04:03<@arkiver>also
14:04:09<@arkiver>feel free to ZSTD things :P
14:04:28<OrIdow6>Are static dicts fine in the future?
14:04:36AK quits [Client Quit]
14:04:43<@arkiver>no
14:04:47<@arkiver>do we need dicts here?
14:04:51AK (AK) joins
14:04:53<OrIdow6>No
14:05:01<OrIdow6>Oh, I see what you mean, just for the compression savings
14:05:03<OrIdow6>Will do
14:05:21<OrIdow6>*just for the algorithm
14:08:34<OrIdow6>And sorry for just declaring that people should "come to #niconino", but I am going (going away from IRC, that is) here soon
14:08:50<OrIdow6>If you pick a more interesting channel name... just send me the logs
14:09:47<@arkiver>queuing
14:10:11<@EggplantN>i added 00_vid
14:11:10<OrIdow6>I mean moved to "added" in the items repo
14:11:24<OrIdow6>Though for all I know that's not something usual
14:11:45<@arkiver>00_vid is added?
14:11:50<@arkiver>ah ok
14:11:54<@arkiver>adding everything
14:15:32<@arkiver>nice speed EggplantN
14:15:35<@EggplantN>>_>
14:15:50<@arkiver>also HCross is gaining on you
14:15:51<@arkiver>i think
14:16:59<@EggplantN>oh this is a single Dual E5
14:17:01<@EggplantN>and 1 /24
14:23:59<OrIdow6>Please reduce your rate
14:24:09<@EggplantN>yes
14:24:09<OrIdow6>Site as a whole is having problems
14:26:11trinsic joins
14:28:10<OrIdow6>I don't want to have disrupted anyone's actual use of this site
14:28:42<taka>I think we should reduce the connection bandwidth to Niconico. They have blocked connections from foreign IPs in the past when we experienced a lot of foreign access.
14:29:05<@kiska>Ah good lemme go and get some jp IPs
14:30:42trinsic quits [Client Quit]
14:33:14<OrIdow6>Why the non-multi items?
14:34:04<@arkiver>because of limits OrIdow6
14:34:14<OrIdow6>Oh
14:34:15<@arkiver>10k items/min
14:41:29<wessel1512>what is the Niconico docker link
14:41:46<wessel1512>atdr.meo.ws/archiveteam/niconico-grab:latest isint working
14:52:46<wessel1512>arkiver can you help
14:53:48<@arkiver>kiska: does it need to be poked? ^
14:53:56<@kiska>Sorry?
14:54:14<@JAA>arkiver: Twitter scrape for ge.tt running now. Going back to 2010 since it appears that the domain was registered in October 2010.
14:55:01<wessel1512>the docker container cant be pulled
14:57:02<@kiska>One moment lemme try it
14:57:27<@kiska>https://server8.kiska.pw/uploads/dc3d43fbdcfec960/image.png No problems here?
14:58:17<wessel1512>it works hear as well
14:58:57<wessel1512>got 407 couple of minutes ago
15:03:13<@arkiver>JAA: thank you!
15:13:40<@JAA>Older uploads on ge.tt have 7-char IDs. Also, I see lots of IDs with the same few trailing characters (note that kiska's examples from last night both end with UZB3). Don't think it's enough to make bruteforcing feasible though.
15:13:46<@JAA>arkiver: https://transfer.notkiska.pw/32AgL/twitter-search-ge.tt-links.zst
15:18:14Atom quits [Ping timeout: 260 seconds]
15:24:54<@JAA>Raw scrape data just in case it's needed: https://transfer.notkiska.pw/jYlYo/twitter-search-ge.tt.jsonl.zst
15:30:42<@arkiver>that was fast...
15:30:46<@arkiver>thanks!
15:32:47<@JAA>parallel snscrape ftw. :-)
15:35:47lunik1 quits [Quit: :x]
15:46:27<@EggplantN>Nico limit now 25k
15:54:48<@EggplantN>So what's next arkiver mediafire Google sites periscope or something off deathwatch :P
16:09:11lunik1 joins
16:14:16Mineroboter_ quits [Client Quit]
16:16:22Mineroboter joins
16:34:04Mineroboter quits [Ping timeout: 260 seconds]
16:39:59<@arkiver>periscope
16:40:01<@arkiver>:)
16:41:57<@kiska>arkiver: ge.tt first pls
16:42:04<@kiska>Its terminating service on the 10 Marhc
16:42:06<@kiska>March*
16:42:20<@arkiver>#microscope for periscope btw
16:42:28<@arkiver>yeah JAA gave the list
16:45:19Mineroboter joins
17:04:11Atom joins
17:36:47marked195736 joins
17:37:04marked19573 quits [Ping timeout: 260 seconds]
17:37:04marked195736 is now known as marked19573
18:11:42marked19573 quits [Client Quit]
18:24:11marked19573 joins
19:20:05<SCSi>meh is the docker image not setup for webs?
19:20:37Hackerpcs quits [Quit: Hackerpcs]
19:21:53Hackerpcs (Hackerpcs) joins
19:23:25<Craigle>SCSi: atdr.meo.ws/archiveteam/webs-grab
19:26:31<SCSi>gotcha, thx
19:51:51<Craigle>SCSi: Forgot to mention, the project channel is #webbed as well
19:54:52<SCSi>rgr
20:18:31jlu5 quits [Quit: Leaving]
20:18:46jlu5 (jlu5) joins
21:17:06DogsRNice (Webuser299) joins
21:37:29<mgrandi>https://www.griddy.com/ is shutting down
21:39:53<mgrandi>(famous for: the texas weather apocalypse )
21:42:45<mgrandi>looks like from the logs (thanks JAA) that it was run on feb 18th
21:43:53<mgrandi>it looks like we didn't get their zendesk instance (https://griddy.zendesk.com/hc/en-us ) though, i'll download their youtube
21:45:16<@JAA>https://i.imgur.com/VtRIt3t.gif
21:46:02<@JAA>Zendesk's been annoying lately, but I'll try.
21:47:49<mgrandi>i can download their android app, maybe someone can attempt to grab their ios app?
21:49:53<mgrandi>i also don't see a link to their mobile apps so maybe we can throw https://play.google.com/store/apps/details?id=com.app.griddy&hl=en_US&gl=US in ? i have no idea how to prevent that from infinitely recursing though
21:51:19<@JAA>I've thrown https://apkpure.com/griddy/com.app.griddy/ in, which includes past versions.
21:51:35<mgrandi>https://apps.apple.com/us/app/griddy-wholesale-electricity/id1196099593
21:59:07<mgrandi>i was more thinking about the comments, it seems to only have a subset and then you have to scroll down / click a button to show more, i can just, right click -> save webpage as, unless there is a handy way to do this as a warc somewhere
22:00:14<@JAA>Browser with warcprox, brozzler, crocoite.
22:04:59<mgrandi>is that at all easy to set up
22:10:11<@JAA>Not really, no.
22:12:50<mgrandi>lol
22:13:02<mgrandi>is warcprox sufficient maybe?
22:13:10<mgrandi>i can just manually scroll down
22:13:41<@JAA>Yeah, but you need to be careful not to also capture all your other browser traffic, which is virtually impossible due to all the crap that gets transmitted in the background these days.
22:13:53<mgrandi>luckily i have 5 browsers installed
22:14:33<mgrandi>i'll just configure only one to use the proxy
22:19:25susu joins
22:31:48BlueMaxima joins
23:29:35<taka>Currently, Niconico is having difficulty viewing videos, probably due to the AT load, and they are performing emergency maintenance.
23:29:53<taka>https://blog.nicovideo.jp/niconews/148271.html
23:30:26<taka>Can you make the load a little smaller?
23:34:55<Jake>EggplantN: ^^ or maybe someone else with tracker access?
23:35:25<@EggplantN>LOL
23:35:27<@EggplantN>sec
23:35:45<Jake>yup sorry ;)
23:35:49<@EggplantN>halved
23:44:48<Jake><3
23:55:55Larsenv quits [Ping timeout: 264 seconds]
23:57:56<susu>Hi all... I have launched a docker runner fore webbed. I would like to check crawl data, it is possible to rsync back a copy of the ouput folder (RSYNC_SRV/NICKNAME) to my local filesystem to check the warc, without putting the mess ?
23:58:54<susu>(i put it here because.. it may apply for another projects in the future)
23:59:15Larsenv (Larsenv) joins

HomeSearchPrevious dayNext day