| 00:42:42 | | HP_Archivist quits [Read error: Connection reset by peer] |
| 00:42:59 | | HP_Archivist (HP_Archivist) joins |
| 01:00:02 | | dm4v quits [Client Quit] |
| 01:04:42 | | dm4v joins |
| 01:04:44 | | dm4v is now authenticated as dm4v |
| 01:04:44 | | dm4v quits [Changing host] |
| 01:04:44 | | dm4v (dm4v) joins |
| 01:36:23 | | tzt quits [Ping timeout: 265 seconds] |
| 01:40:41 | | tzt (tzt) joins |
| 02:03:10 | | dm4v quits [Ping timeout: 258 seconds] |
| 02:05:01 | | dm4v joins |
| 02:05:04 | | dm4v is now authenticated as dm4v |
| 02:05:04 | | dm4v quits [Changing host] |
| 02:05:04 | | dm4v (dm4v) joins |
| 03:22:12 | | pabs quits [Quit: Don't rest until all the world is paved in moss and greenery.] |
| 03:24:41 | | pabs (pabs) joins |
| 04:21:48 | | qw3rty_ joins |
| 04:25:33 | | qw3rty__ quits [Ping timeout: 265 seconds] |
| 04:44:16 | | qwertyasdfuiopghjkl quits [Remote host closed the connection] |
| 04:47:09 | | qwertyasdfuiopghjkl joins |
| 05:41:42 | | NotWebuser joins |
| 07:12:00 | | qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds] |
| 08:38:36 | | BlueMaxima quits [Read error: Connection reset by peer] |
| 09:10:35 | | HackMii_ quits [Remote host closed the connection] |
| 09:11:11 | | HackMii_ (hacktheplanet) joins |
| 10:53:01 | | driib798943 (driib) joins |
| 10:54:09 | | driib79894 quits [Ping timeout: 265 seconds] |
| 10:54:09 | | driib798943 is now known as driib79894 |
| 13:23:58 | | Arcorann quits [Ping timeout: 258 seconds] |
| 13:49:13 | | sec^nd quits [Remote host closed the connection] |
| 13:49:50 | | sec^nd (second) joins |
| 13:54:38 | | Sluggs quits [Ping timeout: 258 seconds] |
| 14:28:26 | | qwertyasdfuiopghjkl joins |
| 14:29:20 | | Sluggs joins |
| 14:55:33 | | sec^nd quits [Remote host closed the connection] |
| 14:55:58 | | sec^nd (second) joins |
| 15:10:38 | | thelounge316 joins |
| 15:13:13 | | thelounge31 quits [Ping timeout: 265 seconds] |
| 15:13:13 | | thelounge316 is now known as thelounge31 |
| 15:17:37 | | sec^nd quits [Remote host closed the connection] |
| 15:17:57 | | sec^nd (second) joins |
| 15:39:39 | | Ruthalas quits [Quit: Ping timeout (120 seconds)] |
| 15:39:45 | | superkuh quits [Remote host closed the connection] |
| 15:40:09 | | Ruthalas (Ruthalas) joins |
| 15:40:38 | | superkuh joins |
| 16:46:05 | <Doranwen> | anyone here tried using this script before? https://github.com/radiolarian/AO3Scraper I had a friend ask me to grab them all the fics that fit a particular set of filters (would've run into the 1000s), but my attempts at using that script failed with some weird error messages (I only got as far as trying the work ids one) |
| 17:15:59 | | spirit quits [Client Quit] |
| 17:16:23 | <thuban> | Doranwen: no, but pastebin the error |
| 17:17:07 | <Doranwen> | I actually can't - it locked up the terminal and wouldn't let me c/p it - but I can probably produce a pic version of it, I did take a screenshot |
| 17:17:14 | <thuban> | hm, ok |
| 17:17:45 | <thuban> | (script author seems not to be aware of urllib.parse...) |
| 17:21:45 | <Doranwen> | thuban: https://paste.pics/02503fb5635397b87c0caab8cffd9531 |
| 17:22:13 | <Doranwen> | It kept adding .... on the terminal line, and I finally had to close the terminal to stop it |
| 17:22:33 | <Doranwen> | also, the work ids it produced didn't match the ones on the first page at all, so I figured it was a bit confused |
| 17:23:04 | <thuban> | you didn't wrap the argument you gave to the script in quotes, so it saw the ampersands in the url and got confused thinking you wanted to background it |
| 17:24:06 | <thuban> | (btw, transfer.archivete.am can host images https://transfer.archivete.am/inline/bG4mu/aatt.png) |
| 17:25:28 | <Doranwen> | ahhh, that's what it needed |
| 17:26:13 | <thuban> | well, strictly bash saw and ate the ampersands, but you know what i mean |
| 17:26:26 | | Doranwen nods |
| 17:26:35 | <Doranwen> | well, that gets rid of the first half of the errors |
| 17:26:51 | <Doranwen> | now it just starts with the syntaxwarning "is" with a literal stuff |
| 17:27:40 | <Doranwen> | and now it does the processing properly with the ... |
| 17:27:46 | <thuban> | those are warnings, not errors, so it will probably work anyway |
| 17:27:53 | <thuban> | yeah |
| 17:28:44 | <thuban> | (https://bugs.python.org/issue34850) |
| 17:28:54 | <Doranwen> | it's still pulling the wrong ids, though |
| 17:29:07 | <Doranwen> | the first id it pulls is for an entirely different fandom than the one in that list |
| 17:29:18 | <Doranwen> | *in the url I fed it |
| 17:29:39 | <thuban> | huh, lemme take a look |
| 17:29:44 | <thuban> | paste that url? |
| 17:30:08 | <Doranwen> | https://archiveofourown.org/works?utf8=%E2%9C%93&commit=Sort+and+Filter&work_search%5Bsort_column%5D=revised_at&work_search%5Bother_tag_names%5D=&exclude_work_search%5Barchive_warning_ids%5D%5B%5D=18&work_search%5Bexcluded_tag_names%5D=&work_search%5Bcrossover%5D=&work_search%5Bcomplete%5D=&work_search%5Bwords_from%5D=&work_search%5Bwords_to%5D=&work_search%5Bdate_from%5D=&work_search%5Bdate_to%5D=&work_search%5Bquery%5D=&work_search% |
| 17:30:10 | <Doranwen> | 5Blanguage_id%5D=&tag_id=Stranger+Things+%28TV+2016%29 |
| 17:30:16 | <Doranwen> | apparently it's too long for IRC, lol |
| 17:30:47 | <Doranwen> | the second id it produced was from an entirely *different* fandom - still the wrong one |
| 17:30:49 | <Doranwen> | very weird |
| 17:31:47 | <Doranwen> | oh wait |
| 17:32:04 | <Doranwen> | maybe not |
| 17:32:15 | <Doranwen> | I wonder if it just added onto the failed attempt I did earlier |
| 17:32:17 | <Doranwen> | instead of overwriting it |
| 17:32:47 | <Doranwen> | simple solution - remove the file and try again, lol |
| 17:32:50 | <thuban> | yes, it does (odd choice) |
| 17:32:58 | <thuban> | works for me |
| 17:33:00 | <Doranwen> | (I agree) |
| 17:33:02 | <Doranwen> | yay! |
| 17:33:36 | <Doranwen> | I wish it produced the *urls* of the works, but I can take the ids and try them in a different scraper that'll actually pull the epubs and see if that will do for my friend |
| 17:33:38 | <Doranwen> | much thanks |
| 17:33:50 | <Doranwen> | (it's not all that hard to turn a work id into a url, anyway) |
| 17:34:02 | <thuban> | (yeah) |
| 17:34:09 | <thuban> | np |
| 17:52:15 | <Doranwen> | Huh. There should've been 20k some ids in the list when done. I got less than 3k. |
| 18:20:07 | <Doranwen> | *But* if I specify 3k, I get all 3k. I guess I |
| 18:20:15 | <Doranwen> | *I'll just specify all 20k some and see what happens |
| 18:21:30 | <thuban> | Doranwen: i got almost 5k. it's clearly stochastic and the culprit seems to be the line "if (len(works) is 0)", so i suspect that "is" is indeed the problem here |
| 18:22:43 | <thuban> | try fixing the syntax warnings (by using "==" or "!=" as the message suggests) and trying again; if i'm correct specifying the number is not relevant |
| 18:24:22 | <thuban> | (also wtf this is O(n^2) for NO REASON) |
| 18:29:20 | <Doranwen> | will do :) |
| 18:44:02 | | spirit joins |
| 19:15:17 | <Doranwen> | thuban: I fixed all the "is" and "is not" and tried it again with no number specified… got only 2660 ids this time |
| 19:15:48 | <Doranwen> | I wonder whether I need to just do a block, go to the first page after it stopped pulling, and put *that* URL in for the next attempt? |
| 19:17:57 | <Doranwen> | it might at least have a chance of getting them all eventually |
| 20:01:31 | <thuban> | Doranwen: i expect that would work, but i'm going to go looking for the bug as i'm curious now |
| 20:01:58 | <Doranwen> | Good luck! |
| 20:02:19 | <Doranwen> | I am getting it to work with that, but figuring out how to properly fix the code so it actually pulled all of the ids would be better… |
| 20:10:52 | <thuban> | ah, apparently AO3 sometimes responds with a "Retry later" message instead of a page |
| 20:12:31 | <Doranwen> | yeah, I saw some discussion of someone trying to fix an issue with that |
| 20:12:38 | <thuban> | (this script is at the maximum permitted request frequency, so sometimes latency etc can cause issues) |
| 20:12:46 | <thuban> | i'll just fix it lol |
| 20:12:50 | <thuban> | and some other stuff |
| 20:12:59 | | Doranwen grins |
| 20:37:57 | | NotWebuser quits [Remote host closed the connection] |
| 20:39:33 | <Jake> | Seems to be some 429 handling code in the PRs? |
| 20:39:56 | | wessel1512 is now authenticated as wessel1512 |
| 20:42:11 | <Jake> | (specifically https://github.com/radiolarian/AO3Scraper/pull/27 ) |
| 20:43:04 | <thuban> | idk why people still write manual retry logic when requests has transport adapters now |
| 20:43:45 | <@JAA> | I like the `status = 429` line. |
| 21:06:30 | | DogsRNice (Webuser299) joins |
| 21:07:57 | | sec^nd quits [Remote host closed the connection] |
| 21:08:24 | | sec^nd (second) joins |
| 21:23:58 | | Matthww8 joins |
| 21:25:23 | | Matthww quits [Ping timeout: 265 seconds] |
| 21:25:23 | | Matthww8 is now known as Matthww |
| 22:12:59 | | BlueMaxima joins |
| 22:53:06 | | Hackerpcs quits [Client Quit] |
| 22:57:03 | | Myself quits [Ping timeout: 258 seconds] |
| 22:58:19 | | Hackerpcs (Hackerpcs) joins |
| 23:36:05 | | Myself (myself) joins |
| 23:42:34 | | Arcorann (Arcorann) joins |
| 23:45:14 | <pabs> | https://jeffhuang.com/designed_to_last/ https://news.ycombinator.com/item?id=29661935 |
| 23:45:21 | <pabs> | "A Manifesto for Preserving Content on the Web" |