| 00:15:34 | | HackMii quits [Remote host closed the connection] |
| 00:16:41 | | HackMii (hacktheplanet) joins |
| 00:26:51 | | Doranwen is now authenticated as Doranwen |
| 00:37:41 | | HackMii quits [Remote host closed the connection] |
| 00:38:31 | | HackMii (hacktheplanet) joins |
| 01:15:37 | | AeonG joins |
| 01:54:48 | | britmob joins |
| 01:57:17 | | britm0b quits [Ping timeout: 252 seconds] |
| 01:58:52 | | britm0b joins |
| 02:00:01 | | Dallas quits [Client Quit] |
| 02:00:02 | | britmob quits [Ping timeout: 252 seconds] |
| 02:01:01 | | britmob joins |
| 02:03:04 | | britm0b quits [Ping timeout: 240 seconds] |
| 02:03:44 | | Dallas (Dallas) joins |
| 02:03:58 | | britm0b joins |
| 02:07:44 | | britmob quits [Ping timeout: 240 seconds] |
| 02:08:50 | | britm0b quits [Ping timeout: 252 seconds] |
| 04:10:46 | | lukash quits [Quit: The Lounge - https://thelounge.chat] |
| 04:13:26 | | lukash (lukash) joins |
| 04:54:00 | | qw3rty_ joins |
| 04:57:24 | | qw3rty__ quits [Ping timeout: 240 seconds] |
| 05:12:20 | | wickedplayer494 quits [Ping timeout: 252 seconds] |
| 06:19:28 | | godane2 joins |
| 06:21:44 | | godane1 quits [Ping timeout: 240 seconds] |
| 06:41:17 | | wickedplayer494 joins |
| 06:41:20 | | wickedplayer494 is now authenticated as wickedplayer494 |
| 06:52:59 | | pie_ joins |
| 06:53:25 | <pie_> | hah. didnt notice the move and now i cant check my logs |
| 07:23:02 | | LeGoupil joins |
| 08:37:11 | | hooway joins |
| 08:50:09 | | godane2 quits [Client Quit] |
| 08:58:40 | | c00k13 joins |
| 09:22:34 | | AeonG_ joins |
| 09:24:36 | | AeonG quits [Ping timeout: 252 seconds] |
| 10:06:03 | | Arcorann (Arcorann) joins |
| 10:53:32 | | @Fusl quits [Ping timeout: 252 seconds] |
| 10:59:07 | | AeonG__ joins |
| 11:01:04 | | AeonG_ quits [Ping timeout: 240 seconds] |
| 11:03:40 | | AeonG__ quits [Read error: Connection reset by peer] |
| 11:05:03 | | AeonG__ joins |
| 11:06:50 | | Fusl (Fusl) joins |
| 11:06:50 | | @ChanServ sets mode: +o Fusl |
| 11:08:38 | | qw3rty_ quits [Read error: Connection reset by peer] |
| 11:09:15 | | qw3rty_ joins |
| 11:23:47 | | @Fusl quits [Ping timeout: 252 seconds] |
| 11:42:03 | | Arcorann_ joins |
| 11:45:14 | | Arcorann quits [Ping timeout: 252 seconds] |
| 12:01:51 | | Fusl (Fusl) joins |
| 12:01:51 | | @ChanServ sets mode: +o Fusl |
| 12:11:40 | | @Fusl quits [Client Quit] |
| 12:21:39 | | Fusl (Fusl) joins |
| 12:21:39 | | @ChanServ sets mode: +o Fusl |
| 13:09:58 | | dd joins |
| 13:10:46 | | dd leaves |
| 14:21:24 | | Arcorann_ quits [Ping timeout: 240 seconds] |
| 15:02:17 | | LeGoupil quits [Client Quit] |
| 16:03:24 | | xit quits [Ping timeout: 240 seconds] |
| 16:03:24 | | lukash quits [Read error: Connection reset by peer] |
| 16:03:30 | | lukash7 joins |
| 16:04:06 | | xit joins |
| 18:47:22 | | VerifiedJ quits [Quit: The Lounge - https://thelounge.chat] |
| 18:50:41 | | VerifiedJ (VerifiedJ) joins |
| 19:11:08 | | honno joins |
| 19:15:36 | <honno> | Heya folks, I was wondering if anybody has ideas on improving the performance of accessing pages of an ol WARC when run locally. The collection specifically is at archive.org/details/archiveteam_gamemaker, scattered into multiple chunks (product of the megawarc factory I believe). Right now it takes ~3 mins to load a page both locally using pywb and on IA's wayback. My use case is web scraping like ~100,000 of the |
| 19:15:37 | <honno> | pages, so this makes things rather impractical heh. |
| 19:18:17 | <honno> | I'm not sure if this is normal behaviour for an archive of this size (like ~750gb), or if the provided index files are screwed up some how. Either way I've never done scraping on WARCs before and wondering if there are specific tools or techniques Im missing |
| 19:19:22 | <Sanqui> | have you tried profiling pywb to see where the bottlenecks lie? |
| 19:19:49 | <honno> | oo should of checked for a feature like that, will do |
| 19:20:08 | <Sanqui> | sorry I can't help much more, I haven't worked with warcs either (but I probably will have to sooner or later) |
| 19:20:49 | <Sanqui> | there are powerful profilers like https://pypi.org/project/pyinstrument/ |
| 19:24:08 | <honno> | yea it seems pywb has its own profile flag but gah dependency hell |
| 19:36:10 | <honno> | is it normal for GETing images to take like 0.1 seconds? The python code itself doesnt show any bottlenecks but it could just be the pages have so many small images in them |
| 19:37:28 | <honno> | if its not normal then hmm I guess I should look for common patterns in whats taking the longest per get request |
| 20:13:54 | <Sanqui> | 0.1 seconds sounds much more reasonable than 3 minutes |
| 20:29:17 | <honno> | ahh but I mean, GETing the page GETs a few dozen images that take 0.1 each (maybe concurrently? cant work that out) |
| 21:36:56 | | Arcorann_ joins |
| 22:01:55 | | honno quits [Remote host closed the connection] |
| 22:49:33 | | Stilett0 joins |
| 22:51:24 | | Stiletto quits [Ping timeout: 240 seconds] |
| 23:03:20 | | godane (godane) joins |
| 23:12:30 | | godane1 joins |
| 23:15:24 | | godane quits [Ping timeout: 240 seconds] |
| 23:51:00 | | lennier2 joins |
| 23:53:59 | | lennier1 quits [Ping timeout: 252 seconds] |
| 23:54:01 | | lennier2 is now known as lennier1 |
| 23:57:12 | | lennier2 joins |
| 23:59:28 | | lennier1 quits [Ping timeout: 252 seconds] |
| 23:59:32 | | lennier2 is now known as lennier1 |