| 00:00:44 | | dm4v quits [Read error: Connection reset by peer] |
| 00:01:12 | | dm4v joins |
| 00:01:14 | | dm4v is now authenticated as dm4v |
| 00:01:14 | | dm4v quits [Changing host] |
| 00:01:14 | | dm4v (dm4v) joins |
| 00:24:52 | | HP_Archivist (HP_Archivist) joins |
| 00:53:21 | | HP_Archivist quits [Client Quit] |
| 01:03:43 | | dm4v quits [Ping timeout: 252 seconds] |
| 01:05:58 | | dm4v joins |
| 01:06:01 | | dm4v is now authenticated as dm4v |
| 01:06:01 | | dm4v quits [Changing host] |
| 01:06:01 | | dm4v (dm4v) joins |
| 03:01:44 | | paul2520_traveling joins |
| 03:02:02 | <paul2520_traveling> | hey is there a way to pull all my Facebook messages with a user? |
| 03:02:18 | <paul2520_traveling> | I just learned a friend passed away :-( |
| 03:26:46 | <paul2520_traveling> | (I assume I can just snscrape her profile) |
| 03:38:05 | | qw3rty_ joins |
| 03:40:48 | | paul2520 (paul2520) joins |
| 03:41:48 | | qw3rty__ quits [Ping timeout: 244 seconds] |
| 03:41:50 | | paul2520_traveling quits [Remote host closed the connection] |
| 03:42:39 | <paul2520> | signed in with my main nick |
| 03:52:46 | <Jake> | paul2520: Facebook has been notoriously hard to scrape at all, snscrape _may_ work, but would definitely require cookies for anything Facebook. I imagine there's better tools if you just want a JSON dump or something? |
| 03:58:49 | <paul2520> | oof yeah, snscrape didn't do anything just now |
| 03:59:06 | <paul2520> | I don't know, I guess I just want anything/everything |
| 03:59:37 | <paul2520> | I could probably be happy with a few full-page screenshots and I'll see what I can do with just selecting all in the messenger window. |
| 04:04:29 | <paul2520> | I think I'll be ok -- I realize she was more active on Instagram, and snscrape should do well there, right? |
| 04:07:36 | <Jake> | snscrape I believe does work fine on Instagram, but may require cookies. |
| 04:10:33 | | paul2520 quits [Remote host closed the connection] |
| 05:46:12 | | Wayward quits [Remote host closed the connection] |
| 07:54:06 | | qwertyasdfuiopghjkl quits [Client Quit] |
| 10:01:24 | | qwertyasdfuiopghjkl joins |
| 11:07:49 | | jamesatjaminit (jamesatjaminit) joins |
| 11:10:47 | | qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds] |
| 11:24:04 | | systwi quits [Ping timeout: 250 seconds] |
| 11:27:42 | | systwi (systwi) joins |
| 11:31:59 | <nyany> | You can download a copy of your Facebook data |
| 11:32:06 | <nyany> | I have no idea if that also includes any messages |
| 11:38:21 | <AK> | I think when I tried before, it includes messages you have sent, but not the other half of the conversation, so it's lacking half of what we probably want |
| 11:52:35 | | BlueMaxima quits [Client Quit] |
| 11:56:48 | | Minkafighter2 is now known as Minkafighter |
| 12:09:08 | | @OrIdow6 quits [Ping timeout: 250 seconds] |
| 13:18:14 | | paul2520 (paul2520) joins |
| 13:18:41 | | paul2520 quits [Remote host closed the connection] |
| 14:07:57 | | ThreeHM quits [Remote host closed the connection] |
| 14:10:50 | | ThreeHM (ThreeHeadedMonkey) joins |
| 14:58:34 | | Arcorann quits [Ping timeout: 250 seconds] |
| 15:16:46 | | Hackerpcs quits [Client Quit] |
| 15:18:48 | | Hackerpcs (Hackerpcs) joins |
| 15:23:11 | | driib quits [Client Quit] |
| 15:23:23 | | driib (driib) joins |
| 15:32:50 | | spirit joins |
| 15:34:17 | | systwi_ (systwi) joins |
| 15:34:22 | | systwi quits [Ping timeout: 252 seconds] |
| 16:00:32 | | Krownest quits [Ping timeout: 250 seconds] |
| 16:36:11 | | systwi_ is now known as systwi |
| 17:28:48 | | useretail quits [Read error: Connection reset by peer] |
| 17:52:41 | | useretail joins |
| 18:03:28 | | Krownest (Krownest) joins |
| 18:13:08 | | jamesatjaminit quits [Ping timeout: 250 seconds] |
| 18:46:56 | | arkiver (arkiver) joins |
| 18:46:56 | | @ChanServ sets mode: +o arkiver |
| 21:16:01 | <@JAA> | Jake: snscrape does not support logging in or specifying cookies. |
| 21:45:38 | | bitbit (bitbit) joins |
| 21:50:19 | <bitbit> | Hi. I want to get a record of all IA identifiers, their descriptions, etc. I tried using the 'ia' cli (`./ia search "?" -f "description,title,..."`) it should have taken about 3 days to complete, but the connection was cutoff after about 3 hours. how can I get such a record? |
| 21:51:17 | | sec^nd quits [Ping timeout: 276 seconds] |
| 22:21:42 | <Jake> | oops. I thought it did! |
| 22:23:12 | <bitbit> | Jake are you 'jjjake'? |
| 22:23:22 | <Jake> | I don't think so? |
| 22:23:55 | <bitbit> | oh. I thought you were saying you're the person that created the ia cli. were you replying to my question? |
| 22:35:14 | <Jake> | sorry, I was replying to the message above. (You probably can't see it!) I'm not 'jjjake' the person who made the ia cli, but could you post the full command you tried to run? |
| 22:36:12 | <bitbit> | Jake: thanks. this is it `./ia search "?" -t 999999999 -f "audio_codec,addeddate,closed_captioning,collection,creator,date,description,external-identifier,identifier,isbn,issn,language,mediatype,notes,openlibrary_edition,publisher,runtime,subject,summary,title,related_collection"` |
| 22:38:55 | <Jake> | Ah, _everything_. That probably won't ever work. Maybe try https://github.com/jjjake/iamine from https://wiki.archiveteam.org/index.php/Internet_Archive_Census but I imagine this generates quite a bit of traffic that IA might not like. |
| 22:39:09 | <bitbit> | well, technically also ` > "ia.jsonl"` at the end. the cut off point of the resulting ia.jsonl is exactly 1,980,000 results. so probably its intentional |
| 22:43:12 | <bitbit> | Jake: thanks. I guess I'll try `./search search "?" -t 999999999` to get all the identifiers first |
| 22:44:23 | <Jake> | Ah, sorry, reading the docs more, that project takes a full list of identifiers from IA that I imagine only IA can generate. (second paragraph https://archive.org/details/ia-bak-census_20150304 ) Out of curiosity, are you looking for something specific? Maybe the last census would have it? |
| 22:49:39 | <bitbit> | Jake: I just want to have a listing of all IA items to maybe find interesting ones based on tags, etc. i.e. cook up an algorithm that gives recommendations |
| 22:51:34 | <bitbit> | the latest census is from 2016 it seems? there were only ~14m items back then, now there's 60 million, according to `./ia search "?" -n` |
| 22:54:57 | <bitbit> | woah... actually 81.9 million according to `./ia search "*" -n` |
| 22:58:18 | <Jake> | This script https://gitlab.com/bwn/cron-census-identifiers/-/blob/master/extract-identifiers.sh (linked on the wiki page) seems to have been used at some point to create similar lists, but there might be too many items now to search through them all? https://archive.org/details/archiveteam_census_2016 |
| 22:58:18 | <Jake> | https://archive.org/details/archiveteam_census_2017 https://archive.org/details/archiveteam_census_2018 |
| 23:11:20 | <bitbit> | Jake: thanks again. now I wait for it finish and hopefully not break :D |
| 23:25:44 | | chriscoffee quits [] |
| 23:56:01 | | Arcorann (Arcorann) joins |