| 00:12:00 | <purplebot> | URLTeam edited by VADemon (+59, /* "Official" shorteners */ add …) just now -- https://www.archiveteam.org/?diff=45979&oldid=45701 |
| 00:16:10 | | HP_Archivist (HP_Archivist) joins |
| 00:23:00 | <purplebot> | Voat edited by Wickedplayer494 (+28, And it is dead, also SearchVoat …) just now -- https://www.archiveteam.org/?diff=45980&oldid=45968 |
| 00:27:04 | | Arcorann quits [Ping timeout: 240 seconds] |
| 00:33:07 | <@arkiver> | OrIdow6: please ping me when you have the code online! |
| 00:33:15 | <@arkiver> | later tomorrow I'll be more available again |
| 00:33:18 | <OrIdow6> | Ok |
| 00:34:25 | <OrIdow6> | Actually may be finishing within the next 20 minutes or so |
| 00:35:07 | <@arkiver> | 300k items right? |
| 00:35:27 | <@arkiver> | we wont be starting it yet in the next 20 minutes, so take your time |
| 00:35:41 | <OrIdow6> | Yes, but only ~1/10 of those aren't empty |
| 00:35:43 | <OrIdow6> | Ok |
| 00:35:47 | <@arkiver> | nice |
| 00:35:49 | <@arkiver> | sounds good |
| 00:35:53 | <@arkiver> | should be fine then |
| 00:35:53 | <OrIdow6> | And I haven't written it for users yet, this is just for comics |
| 00:36:01 | <@arkiver> | yeah thats fine |
| 00:36:13 | <@arkiver> | most important stuff first |
| 00:36:17 | <OrIdow6> | Yep |
| 01:14:15 | <OrIdow6> | arkiver: https://github.com/OrIdow6/smackjeeves-grab |
| 01:37:43 | | chip (chip) joins |
| 02:00:02 | | Dallas quits [Client Quit] |
| 02:03:45 | | Dallas (Dallas) joins |
| 02:15:58 | | tzt quits [Ping timeout: 252 seconds] |
| 02:21:11 | | tzt joins |
| 02:37:13 | | Arcorann (Arcorann) joins |
| 03:10:35 | <thuban> | https://www.reddit.com/r/Marathon/comments/c73yx6/i_will_possibly_be_interviewing_greg_kirkpatrick/etk4lxf/ greg kirkpatrick has all the files from _duality_ lying around somewhere |
| 03:13:21 | <thuban> | "GK: I wrote a letter to Wegman's because they discontinued my favorite peanut butter and it was a very long, elaborate argument as to why they should bring it back and it worked. My peanut butter was restocked." |
| 03:22:26 | | Ajay quits [Remote host closed the connection] |
| 04:41:46 | | qw3rty_ joins |
| 04:45:04 | | qw3rty__ quits [Ping timeout: 240 seconds] |
| 05:15:04 | | balrog quits [Ping timeout: 240 seconds] |
| 05:37:47 | | balrog joins |
| 05:37:48 | | balrog is now authenticated as balrog |
| 08:25:24 | <jodizzle> | Is there a atdr.meo.ws Docker image for urlteam? |
| 09:11:02 | | ragu quits [Ping timeout: 252 seconds] |
| 11:56:48 | <@HCross> | I've got 700G of CNET forums so far |
| 12:32:34 | | grawity quits [Remote host closed the connection] |
| 13:14:44 | <@arkiver> | OrIdow6: looks good |
| 13:15:36 | <@arkiver> | OrIdow6: one thing though - I see a lot of matching on specific URLs to extract from the HTML |
| 13:15:52 | <@arkiver> | are you sure there will be no exceptions when it comes to these image URLs and all will be archived? |
| 13:16:37 | <@arkiver> | I usually extract as many URLs as possible and filter out those we do not want |
| 13:17:06 | <@arkiver> | in this code I see the opposite approach, extract only URLs we want (instead of dropping those we do not want) |
| 13:17:36 | <@arkiver> | which is probably fine, as long as there so special cases or exceptions when it comes to the URLs you're extracting |
| 13:20:38 | <@arkiver> | HCross: nice! |
| 13:20:57 | <@arkiver> | thats quite a bit of forum posts |
| 13:25:12 | <@arkiver> | OrIdow6: but, fine with me, just something to keep in mind for other possible projects |
| 13:25:51 | <@arkiver> | OrIdow6: I assume this is not needed? https://github.com/OrIdow6/smackjeeves-grab/blob/master/pipeline.py#L198-L210 |
| 13:52:04 | | Arcorann quits [Ping timeout: 240 seconds] |
| 15:04:43 | <atphoenix> | HCross, FYI, AB also visited the CNET Forums, although that pass skipped outlinks. Some of the JS stuff doesn't work in the version in WBM, although it does appear to have much of the content. |
| 15:07:03 | <@HCross> | ah, I'm grabbing it all |
| 15:09:20 | <atphoenix> | jodizzle, I don't see anything at https://atdrone.meo.ws/ArchiveTeam/urlteam-grab |
| 17:42:52 | <@Kaz> | one moment |
| 17:43:18 | <@Kaz> | use atdr.meo.ws/archiveteam/terroroftinytown-client-grab |
| 18:26:00 | <purplebot> | URLTeam edited by Arcorann (-153, /* Dead or Broken */ nazr.in died, …) just now -- https://www.archiveteam.org/?diff=45981&oldid=45979 |
| 18:26:00 | <purplebot> | Deathwatch edited by Themadprogramer (+143, Added CDBaby store closing in March …) just now -- https://www.archiveteam.org/?diff=45982&oldid=45972 |
| 18:26:45 | <thuban> | what's the recommended software for browsing/extracting warcs these days (that i could recommend to a non-technical user, so CLI maybe okay, library-only not)? |
| 18:27:00 | <purplebot> | ISP Hosting edited by Wessel1512 (+35) just now -- https://www.archiveteam.org/?diff=45983&oldid=45910 |
| 19:21:15 | <jodizzle> | Thanks Kaz |
| 19:44:28 | | Brad_Knowles joins |
| 20:19:52 | | Mateon2 joins |
| 20:20:35 | | Mateon1 quits [Ping timeout: 252 seconds] |
| 20:20:35 | | Mateon2 is now known as Mateon1 |
| 20:43:00 | | Brad_Knowles quits [Remote host closed the connection] |
| 20:53:02 | | Ajay joins |
| 20:53:40 | | Brad_Knowles joins |
| 21:13:02 | <OrIdow6> | thuban: Browsing or extracting? I think the webrecorder player (/ its "successor") might work for the former; programs for the latter are mostly/all CLI Python scripts |
| 21:13:38 | <thuban> | either |
| 21:14:23 | <thuban> | looks like the webrecorder player has been superceded [sic] by replayweb.page |
| 21:15:24 | <thuban> | unar claims (cli) support for extraction, tho i haven't tested it; not sure whether there are any/superior others (ecosystem seems a bit woolly). |
| 21:18:48 | <@JAA> | I just discovered unar's support the other day, and it's incomplete. |
| 21:19:22 | <@JAA> | In particular, no support for Transfer-Encoding or WARC/1.1. |
| 21:20:01 | <@JAA> | There's `warcat extract`, but I remember having issues with that in the past. |
| 21:22:23 | <@JAA> | In general, extraction into a directory structure is always messy because URLs and HTTP simply don't map to a dir structure. For example, what do you do with redirects (maybe symlinks, but what about the redirect body?), repeated retrievals of the same URL, or collisions on URLs containing consecutive slashes? |
| 21:25:21 | | Brad_Knowles is now known as Brad_Knowles_web |
| 22:19:18 | <OrIdow6> | arkiver: Changes made |
| 22:27:00 | <purplebot> | Drunken Archiving edited by Hook54321 (+1) just now -- https://www.archiveteam.org/?diff=45984&oldid=45595 |
| 22:29:57 | | Brad_Knowles_web quits [Ping timeout: 244 seconds] |
| 22:36:43 | | brad joins |
| 22:39:38 | | Arcorann (Arcorann) joins |
| 22:39:42 | <brad> | Okay, I’m back. Now using IRCCloud, so that I don’t have to set up my own bouncer. |
| 22:40:07 | <themadpro> | brad: welcome to the club! |
| 22:40:25 | <brad> | Please let me know if/when there is anything I can do to help! |
| 22:40:41 | <themadpro> | I also just made an IRCCloud account just this week and am suffering the consequences of having emerged out of the blue all of a sudden 😂 |
| 22:43:05 | <themadpro> | Half of the people recognize me but not from IRC and the other half have no idea when I showed up |
| 22:53:35 | <@kiska> | OOF? |
| 22:53:42 | <@kiska> | Also -ot this |
| 23:11:05 | | Arcorann quits [Ping timeout: 252 seconds] |
| 23:23:00 | <purplebot> | HQ Trivia edited by Wickedplayer494 (+35, /* Revival */ There's been a Words …) just now -- https://www.archiveteam.org/?diff=45985&oldid=44213 |
| 23:59:53 | <tech234a> | Just a reminder that Endomondo shuts down on December 31st; hopefully we can start it soon #findelmundo |