HomeSearchPrevious dayNext day

00:01:31godane quits [Ping timeout: 252 seconds]
00:02:22<Ryz>So... maybe a project on archiving Google Play app pages, particularly reviews? https://twitter.com/TheQuartering/status/1354909724869853188
00:02:36<@EggplantN>woah
00:02:37<@EggplantN>ok
00:02:40<@EggplantN>this shits getting wild
00:03:24<@EggplantN>https://usercontent.irccloud-cdn.com/file/kvg2i8On/image.png
00:03:35<@EggplantN>https://usercontent.irccloud-cdn.com/file/6hmmyTbg/image.png
00:03:49<@EggplantN>Unsure if its >100k but its enough to swing its rating
00:04:07<Ryz>Help with archiving those pages like that and in general? s:
00:04:34<daalekz>can't you get taken off the store if you get review bombed hard enough
00:06:01<Ryz>Well, there's this Google cached thing that we'll at least save: https://twitter.com/TheQuartering/status/1354909724869853188
00:06:08<Ryz>Not that one, this one: https://webcache.googleusercontent.com/search?q=cache:PMQUE_NvgEwJ:https://play.google.com/store/apps/details%3Fid%3Dcom.robinhood.android%26hl%3Den_US%26gl%3DUS+&cd=1&hl=en&ct=clnk&gl=us
00:10:05rsn joins
00:11:58rsn_ quits [Ping timeout: 252 seconds]
00:11:59<@JAA>Looks like reviews are loaded through disgusting POST requests.
00:12:27<@JAA>Seems even worse than the YouTube APIs.
00:12:33<@JAA>Well, 'APIs'.
00:27:40godane2 joins
00:30:04godane1 quits [Ping timeout: 240 seconds]
00:30:25godane (godane) joins
00:32:39godane2 quits [Ping timeout: 265 seconds]
00:36:09godane1 joins
00:38:56godane quits [Ping timeout: 265 seconds]
01:09:41astri quits [Ping timeout: 244 seconds]
01:11:20josephwdye joins
01:11:32<josephwdye>Hello :D
01:11:38<@kiska>Henlo!
01:11:54<@JAA>ohai
01:12:07<@kiska>We have services
01:12:10<josephwdye>How is it going beautiful people?
01:12:43<@kiska>How goes this fine day. Its wonderfully cool in Sydney
01:12:44BlueMaxima joins
01:14:38<josephwdye>My friend is showing of their new 3d printer and i'm soooo jelly.
01:16:39vela quits [Remote host closed the connection]
01:16:56vela (vela) joins
01:17:59astri (astri) joins
01:18:30<@JAA>Aye, pretty well. Royal entertainment is being provided by Reddit. :-)
01:18:54<@JAA>We have another channel for off-topic chat, by the way: #archiveteam-ot
01:18:55<josephwdye>Shit is pretty good right now.
01:19:19<@EggplantN>Heyo.
01:22:42AK quits [Quit: Ping timeout (120 seconds)]
01:22:53AK (AK) joins
01:26:01<OrIdow6>So my thing to explore the Storyfire follower/followee graph finished
01:29:05<OrIdow6>Here is the data - https://transfer.notkiska.pw/rJQ6p/storyfire_follows.py.zst - there may be duplicates due to an oversight involving threading
01:29:34<@JAA>arkiver: ^
01:29:38<OrIdow6>Next step is to order by # followers and produce a size estimate
01:30:20<OrIdow6>About half a million users, by the way
01:30:36<@JAA>Nice
01:31:18<OrIdow6>Again, this file doesn't contain video lists, or even user human-readable names, just info on who follows whom
01:49:50nerdguy1138 quits [Client Quit]
01:50:09nerdguy1138 (nerdguy1138) joins
01:50:40nerdguy1138 quits [Client Quit]
01:51:08nerdguy1138 (nerdguy1138) joins
01:52:51<@arkiver>OrIdow6: sounds good!
01:53:18<@arkiver>i just created a storyfire-grab repo on github, code will be there some time tomorrow
01:55:15<@arkiver>https://github.com/ArchiveTeam/storyfire-grab
01:55:22<OrIdow6>Ok
01:55:45<OrIdow6>I think I hears somewhere that they had to approve people to upload - suggests that the video will be small enough to get all of it
01:55:48godane1 quits [Remote host closed the connection]
01:56:03<OrIdow6>As that would prevent the usual long tail of videos with 1-5 views, etc.
01:56:09godane1 joins
01:56:10<OrIdow6>*heard
01:56:36<@arkiver>would be good
01:58:02<OrIdow6>Yeah
03:03:52josephwdye quits [Ping timeout: 244 seconds]
03:16:13igloo222251 joins
03:16:45ave5 joins
03:17:05ave quits [Read error: Connection reset by peer]
03:17:05igloo22225 quits [Read error: Connection reset by peer]
03:17:05ave5 is now known as ave
03:17:05igloo222251 is now known as igloo22225
03:28:41rsn quits [Remote host closed the connection]
03:29:02rsn joins
03:29:11astri quits [Ping timeout: 244 seconds]
03:29:25Arcorann quits [Ping timeout: 252 seconds]
03:39:44Billy549 quits [Ping timeout: 240 seconds]
03:42:16billy549 (Billy549) joins
03:49:23ave quits [Client Quit]
03:50:46ave (ave) joins
03:52:50DogsRNice quits [Read error: Connection reset by peer]
04:13:35qw3rty_ joins
04:17:24qw3rty__ quits [Ping timeout: 265 seconds]
04:22:27trinsic joins
04:26:34trinsic quits [Client Quit]
04:44:49Ajay quits [Remote host closed the connection]
04:45:05Ajay joins
04:59:57Merzius quits [Remote host closed the connection]
05:00:01treora quits [Quit: blub blub.]
05:01:18treora joins
05:02:01etnguyen03 quits [Client Quit]
05:52:11taka quits [Remote host closed the connection]
06:51:52astrid leaves [][]
06:53:44Atom-- quits [Read error: Connection reset by peer]
06:53:56Atom-- joins
06:57:00Atom__ joins
07:00:04Atom-- quits [Ping timeout: 240 seconds]
07:03:09Atom-- joins
07:06:04Atom__ quits [Ping timeout: 240 seconds]
07:12:07Atom__ joins
07:14:56BlueMaxima quits [Read error: Connection reset by peer]
07:14:58isthisthedarkweb quits [Ping timeout: 244 seconds]
07:15:28Atom-- quits [Ping timeout: 252 seconds]
07:19:04Atom__ quits [Ping timeout: 240 seconds]
07:20:21Atom joins
07:37:27hooway joins
07:54:18sliccricc quits [Remote host closed the connection]
07:54:49sliccricc (sliccricc) joins
08:05:21icedice quits [Client Quit]
08:34:24Jonboy3451 joins
08:36:30Arcorann (Arcorann) joins
08:37:04Jonboy345 quits [Ping timeout: 240 seconds]
08:37:24lukash7 quits [Ping timeout: 240 seconds]
08:37:35lukash7 joins
08:50:04daalekz quits [Ping timeout: 252 seconds]
08:56:09sliccricc quits [Remote host closed the connection]
08:57:22sliccricc (sliccricc) joins
09:21:13rsn_ joins
09:24:44rsn quits [Ping timeout: 240 seconds]
09:24:56shoghicp (shoghicp) joins
09:34:34Atom-- joins
09:35:43Atom quits [Ping timeout: 252 seconds]
09:41:27<@Fusl>EggplantN: there were around 300k ratings when i checked
10:51:37OrIdow6^2 (OrIdow6) joins
10:52:24OrIdow6 quits [Ping timeout: 240 seconds]
11:08:04britmob2 quits [Ping timeout: 240 seconds]
11:08:43britm0b quits [Ping timeout: 265 seconds]
11:09:19britmob joins
11:11:11britmob2 joins
11:16:04britmob2 quits [Ping timeout: 240 seconds]
11:16:24britmob quits [Ping timeout: 240 seconds]
11:16:25britm0b joins
11:20:44britm0b quits [Ping timeout: 240 seconds]
11:34:18britmob2 joins
12:09:34ThisAsYou quits [Client Quit]
12:55:36spirit joins
13:24:05britmob joins
13:32:05systwi (systwi) joins
13:33:43systwi_ quits [Ping timeout: 265 seconds]
13:54:01madmax quits [Ping timeout: 265 seconds]
14:01:24ColdIce quits [Ping timeout: 240 seconds]
14:22:22Arcorann_ joins
14:25:44Arcorann quits [Ping timeout: 240 seconds]
14:57:44Arcorann_ quits [Ping timeout: 240 seconds]
15:00:46godane1 quits [Ping timeout: 252 seconds]
15:06:40britm0b joins
15:09:25britmob quits [Ping timeout: 265 seconds]
15:15:57isthisthedarkweb (isthisthedarkweb) joins
15:18:22britm0b quits [Ping timeout: 252 seconds]
15:19:28britmob2 quits [Ping timeout: 252 seconds]
15:19:52britmob joins
15:21:40britmob quits [Read error: Connection reset by peer]
15:21:53britmob joins
15:27:04britmob quits [Ping timeout: 240 seconds]
15:31:34britmob joins
15:38:05britm0b joins
15:40:24britmob quits [Ping timeout: 240 seconds]
15:42:25britmob joins
15:45:04britm0b quits [Ping timeout: 240 seconds]
16:34:05nertzy (nertzy) joins
16:45:27LeGoupil joins
17:03:03Matthww0 joins
17:04:56Matthww quits [Ping timeout: 265 seconds]
17:04:56Matthww0 is now known as Matthww
17:14:45jeff-kao-propublica-org quits [Quit: Textual IRC Client: www.textualapp.com]
17:43:02Matthww1 joins
17:44:34Matthww quits [Ping timeout: 265 seconds]
17:44:34Matthww1 is now known as Matthww
17:53:36britmob quits [Read error: Connection reset by peer]
17:54:01britmob joins
17:54:09DogsRNice (Webuser299) joins
17:55:14Hackerpcs quits [Quit: Hackerpcs]
17:57:12Hackerpcs (Hackerpcs) joins
18:04:24<avoozl>JAA, Sanqui: got a temporary test up and running, which will go down in a bit, but it looks like we're getting sensible results from the indexer: http://hosted.viszie.com:9090/
18:05:32<Sanqui>that is hella cool!
18:06:09<avoozl>thanks... still tons of items in my todo list :P
18:06:47<Sanqui>there's a lot of german, which I didn't expect haha
18:07:18<avoozl>Oh I picked euh.. forums.euw.leagueoflegends.com-0000?.warc.gz
18:07:23<avoozl>so the EUW there may be a territory
18:07:36<@JAA>euw = EU West IIRC, so makes sense.
18:07:39<avoozl>I can run another if that makes sense
18:08:00<purplebot>Coronavirus edited by Wessel1512 (+178, /* Netherlands */) just now -- https://www.archiveteam.org/?diff=46249&oldid=46217
18:08:13<avoozl>I'm happy I was able to keep it all self-contained. you can basically just run warceater (or warceater.exe) ingest *.warc.gz and then run it with the serve command to get this
18:08:22<@JAA>Nice!
18:08:25<avoozl>indexes are purely on-disk, need no other service. will take up a ton of space though
18:08:36<avoozl>and the post-extraction logic is still very much hardcoded
18:09:38<avoozl>basically like this: https://imgur.com/a/RWZyKpz
18:09:54<avoozl>but this would be fairly easy to change into something that takes these selectors from some kind of configuration file
18:10:00<avoozl>that's low on my prio's though
18:10:28<@JAA>Yeah, that can always be changed later.
18:12:47<@JAA>'Searched for ".euw.", found 17790182 results in 7.770624154 seconds.' Well... lol
18:13:14<avoozl>haha yeah the url field is probably matching as well
18:13:21<avoozl>There's a syntax to do field based queries as well
18:13:41<avoozl>try msg:.iew.
18:14:03<avoozl>you can also use this to search for user names, thread ids, etcetera. but the field names are not all exposed yet
18:14:21<@JAA>Ah, very nice!
18:14:25<avoozl>https://blevesearch.com/docs/Query-String-Query/ is the current syntax, due to me using that search engine under water
18:14:45<avoozl>that may change, because I don't like the memory-mapped index structure bleve is using for this kind of slow-retrieval-huge-db situation
18:15:05<avoozl>I'd rather have something where the index can be in object/http storage and we retrieve bits of it on-demand... but that is another story
18:16:05<avoozl>it's currently running on my desktop, so it go down at some point.. I'll try and get some more info out over the next week
18:16:57<@JAA>Sounds great, and excellent work so far!
18:17:20<avoozl>pagination is also on my list. you currently only get the first 100 items
18:17:56<@JAA>Yeah, I noticed that. (Please no JS infinite pagination crap.) :-)
18:18:32<avoozl>haha no. I will just use normal html based stuff. there is no functional js in there right now. I didn't test but everything should work even with javascript off, though it'll look different I guess
18:19:32<@JAA>I didn't see any JS at all, and it's fully usable without it at the moment.
18:19:32Craigle quits [Remote host closed the connection]
18:19:36<@JAA>:-)
18:20:07<avoozl>The only thing where I'd like to put some JS in is when I would add something like faceting or timeline range selection.. because doing that in purely form post/retrieve would be a pain
18:21:12<thuban>progressive enhancement: it still works! who knew? :)
18:34:23Craigle (Craigle) joins
18:44:48yoyo113 joins
18:47:31<Sanqui>!a < https://etc.sanqui.net/archiveteam/webzdarma/webzdarma_mwlinkscrape_04 --no-offsite
18:47:36<Sanqui>...yeah not here
18:49:00<purplebot>Webzdarma edited by Sanqui (+89, webzdarma_mwlinkscrape_04 and 05) just now -- https://www.archiveteam.org/?diff=46250&oldid=46248
19:09:04Ruthalas quits [Ping timeout: 240 seconds]
19:09:20divi quits [Quit: Konversation terminated!]
19:09:37Ruthalas (Ruthalas) joins
19:11:38yoyo113 quits [Remote host closed the connection]
19:22:29JosephWdye joins
19:24:56ColdIce joins
19:47:14JosephWdye quits [Ping timeout: 244 seconds]
19:54:26VerifiedJ quits [Quit: The Lounge - https://thelounge.chat]
20:00:17VerifiedJ (VerifiedJ) joins
20:11:56<avoozl>ok running a new index version, so search frontend is offline, we'll see more later :)
20:12:43britmob quits [Read error: Connection reset by peer]
20:12:45britm0b joins
20:17:28britmob joins
20:20:04britm0b quits [Ping timeout: 240 seconds]
20:26:24nerdguy1138 quits [Ping timeout: 240 seconds]
20:41:17systwi_ (systwi) joins
20:41:50nerdguy1138 (nerdguy1138) joins
20:43:53systwi quits [Ping timeout: 265 seconds]
20:52:11nimaje quits [Client Quit]
21:44:31<mgrandi>Do we currently have something for watching twitter accounts for new posts?
21:45:17<mgrandi>Don't want to have to keep wasting AB resources to grab a person's entire twitter history repeatedly when there is probably only a few new urls
21:51:13daalekz joins
21:55:21daalekz quits [Read error: Connection reset by peer]
22:06:59BlueMaxima joins
22:22:50<atphoenix>human eyes are the only thing I know of right now. I think there was a 'since' or some other date-based option; not sure that still is an option.
22:24:07LeGoupil quits [Client Quit]
22:24:28<@JAA>Yes, 'since' exists as do some other operators. Twitter no longer has public documentation on them though.
22:27:34<thuban>JAA: accessible via the search, or only through the api (and therefore with a developer key)?
22:30:15<@JAA>Search. I'm not familiar with the API.
22:38:45<mgrandi>The API uses pagination since stuff changes so much, but the initial search query might have it
22:39:02<mgrandi>Also the api is limited to like 4k tweets per account but that might be OK for getting recent stuff
23:00:01Arcorann_ joins
23:29:49spirit quits [Remote host closed the connection]
23:41:04hooway quits [Client Quit]
23:53:29Merzius joins

HomeSearchPrevious dayNext day