| 00:09:25 | | hooway quits [Client Quit] |
| 01:01:00 | <purplebot> | Deathwatch edited by Westburn86 (+261, /* 2020 */ Add radio site) just now -- https://www.archiveteam.org/?diff=45969&oldid=45964 |
| 01:04:00 | <@JAA> | So I poked images.nga.gov a bit. Firstly, lots of JS, and some hilarious engineering. The full-size downloads make a GET request. The information on what to download is put into a JS object, serialised to JSON, percent-encoded, base64'd, and then put in a query parameter... |
| 01:04:27 | | thuban crosses self |
| 01:06:34 | <@JAA> | Now, it looks like all images are also available on nga.gov and only images.nga.gov shuts down. However, it also looks like the versions on nga.gov may be somewhat more compressed. At least in the few cases I tested manually, I got significantly larger files from images.nga.gov at the same resolution. |
| 01:08:06 | <@JAA> | Unfortunately, images.nga.gov is horribly slow, so we'll see how far that gets before the shutdown. |
| 01:15:31 | <@arkiver> | JAA: when is the shutdown? |
| 01:16:35 | <@JAA> | 1 Jan |
| 01:17:14 | <@arkiver> | i dont see it on the deathwatch page |
| 01:17:19 | <@arkiver> | well if you need help, ping me :) |
| 01:17:32 | <@JAA> | Just added it a minute ago. |
| 01:17:49 | <@JAA> | It's running now in AB. |
| 01:17:57 | <@JAA> | But the site is so slow that it might not finish in time. |
| 01:20:49 | <@arkiver> | JAA: does is speed when downloading concurrently? |
| 01:22:26 | <@arkiver> | it* |
| 01:24:21 | <@JAA> | Took about 8.5 minutes in AB to retrieve the first 1 per mille of IDs. So that'd put it at 5.9 days, which would be fast enough. |
| 01:24:43 | <@JAA> | But none of these IDs exist, not sure if it'll stay at that speed once it gets to the actual images. |
| 01:29:44 | <@JAA> | I'll keep an eye on it and figure something out. I can easily throw qwarc at it if needed. |
| 01:38:00 | <purplebot> | Deathwatch edited by JustAnotherArchivist (+128, /* 2021 */ Add National Gallery …) 21 minutes ago -- https://www.archiveteam.org/?diff=45970&oldid=45969 |
| 01:48:15 | <phuzion> | Has anyone snagged this yet? https://file.wikileaks.org/file/ |
| 01:48:47 | <phuzion> | Disregard, it's apparently quite old |
| 01:54:22 | <@JAA> | It's running through AB anyway. |
| 01:54:29 | <@JAA> | (Since last week) |
| 02:00:01 | | Dallas quits [Client Quit] |
| 02:03:56 | | Dallas (Dallas) joins |
| 02:31:03 | <phuzion> | Oh ok |
| 03:22:36 | | Stiletto joins |
| 03:23:24 | | Stilett0 quits [Ping timeout: 240 seconds] |
| 04:05:57 | | sec^nd quits [Ping timeout: 240 seconds] |
| 04:10:49 | | sec^nd (second) joins |
| 04:44:49 | | qw3rty__ joins |
| 04:48:04 | | qw3rty_ quits [Ping timeout: 240 seconds] |
| 05:34:03 | | sec^nd quits [Remote host closed the connection] |
| 05:34:21 | | sec^nd (second) joins |
| 05:52:36 | | wickedplayer494 quits [Ping timeout: 252 seconds] |
| 07:36:52 | | wickedplayer494 (wickedplayer494) joins |
| 07:44:04 | | wessel1512 quits [Ping timeout: 252 seconds] |
| 07:52:03 | | wessel1512 joins |
| 08:06:55 | | VADemon joins |
| 08:09:22 | | wessel1512 quits [Ping timeout: 252 seconds] |
| 08:14:14 | | wessel1512 joins |
| 08:17:48 | | wessel1512 quits [Read error: Connection reset by peer] |
| 08:17:55 | | wessel1512 joins |
| 08:28:50 | | fieugh (fieugh) joins |
| 08:29:14 | <fieugh> | obligatory drunken hello |
| 09:00:22 | | britmob2 quits [Quit: britmob2] |
| 09:20:10 | <HP_Archivist> | Question - I have an IA item that has photos and has a video that's been derived to an mp4. I want to have a photo be the item preview and the front file in the preview box. How do I specific to have an photo over a video? |
| 09:20:34 | <HP_Archivist> | I know _itemimage.jpg would usually chose a specific photo over a different one, but how to designate the same instead of a video? |
| 09:21:13 | <HP_Archivist> | specify to have a photo over a video?* |
| 09:50:36 | | hooway joins |
| 11:04:58 | | yauz joins |
| 11:07:44 | | ragu__ joins |
| 11:10:30 | | ragu_ quits [Ping timeout: 252 seconds] |
| 12:13:38 | | ragu__ is now authenticated as ragu |
| 13:09:08 | | wessel1512 is now authenticated as wessel1512 |
| 13:50:30 | | britmob2 joins |
| 14:23:08 | | VADemon quits [Client Quit] |
| 14:24:21 | <@arkiver> | fieugh: hi |
| 14:24:50 | <@arkiver> | HP_Archivist: upload the photo as _itemimage as well? |
| 14:24:53 | <@arkiver> | ah the preview box |
| 14:24:57 | <@arkiver> | err not sure |
| 14:25:03 | <@arkiver> | you can email info@archive.org |
| 14:51:53 | | hooway quits [Client Quit] |
| 14:52:32 | | Arcorann_ quits [Ping timeout: 252 seconds] |
| 14:59:42 | | hooway joins |
| 15:49:50 | | VADemon joins |
| 17:20:19 | | kallsyms (kallsyms) joins |
| 17:26:00 | <purplebot> | Software edited by Nemo bis (+100, /* Format Specific */ +ZIM) just now -- https://www.archiveteam.org/?diff=45971&oldid=42301 |
| 17:26:00 | <purplebot> | Deathwatch edited by Nemo bis (+61, /* 2020 */ there was an official …) just now -- https://www.archiveteam.org/?diff=45972&oldid=45970 |
| 17:28:00 | <@arkiver> | JAA: was anything already done for smackjeeves? |
| 17:46:51 | | Ajay quits [Remote host closed the connection] |
| 17:56:58 | <@JAA> | arkiver: OrIdow6 poked it a little some weeks ago and wanted to look into it more I think. |
| 18:23:42 | <thuban> | anyone have any idea about how tumblr does its awful photoset image-rewriting? |
| 18:27:04 | <thuban> | if you take an image from a photoset (eg https://architectureofdoom.tumblr.com/post/638312928993509376/ventilation-building-of-the-maastunnel-rotterdam) and try to visit the image url in a browser, tumblr will instead serve you a page about the image... the first time. subsequent loads seem to be the image directly. changing the referer (ie clicking link vs copy-url-and-open) |
| 18:27:05 | <thuban> | doesn't seem to alter this, and wget seems to get the image directly every time even if you set its user-agent to the browser's. |
| 18:33:04 | | VADemon quits [Ping timeout: 252 seconds] |
| 18:35:31 | <thuban> | submitting tumblr image urls to ia's "save page now" (via waybackpy) chokes on excessive redirects... unless you set its user-agent to wget's. in which case it saves the page, not the image, and in such a way that the image itself isn't actually saved. (setting wget's user-agent to waybackpy's doesn't change anything; it still gets the image successfully.) |
| 18:44:01 | <atphoenix> | if smackjeeves needs a channel...maybe sackedjeeves. (Sacked...fired). Or comicgone / comigone / comics gone / webcomicgone (word play on Comic Con). |
| 18:44:18 | <thuban> | i lied: you don't need to set the user-agent; "save page now" apparently just _sometimes_ 'works' and sometimes doesn't. (curl also seems to always get the image.) |
| 19:35:38 | <thuban> | curl reproducibly gets html if the user-agent header is set to my browser's or waybackpy's *and* the accept header is set to 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', and img if the user-agent is set to curl's *or* the accept header is set to 'image/webp,image/png,image/*;q=0.8,*/*;q=0.5' or '*/*' or ''. |
| 19:36:45 | <thuban> | which makes it seem like just setting waybackpy's user-agent to curl's should get the image, but it doesn't. why? wtf r u doing, brozzler? :( |
| 19:40:08 | <thuban> | i guess i should email info@ |
| 19:42:54 | <@arkiver> | atphoenix: smackedjeeves? |
| 19:42:57 | <@arkiver> | JAA: thanks! |
| 19:53:51 | | Ajay joins |
| 20:06:19 | <atphoenix> | arkiver, or that, yes :) |
| 20:15:41 | | Mateon1 joins |
| 20:32:10 | | Hackerpcs quits [Client Quit] |
| 20:33:57 | | Hackerpcs (Hackerpcs) joins |
| 21:26:47 | | yauz quits [Remote host closed the connection] |
| 21:30:47 | <@kiska> | Is there a channel I should be in? |
| 21:36:22 | <mgrandi> | So apparently twitter is highly considering making the new POTUS account start over with followers, should we like scrape the following list for the presidential-related twitter accounts? |
| 21:55:00 | <purplebot> | Running Archive Team Projects with Docker edited by Tech234a (+1440, Add instructions for running Docker …) just now -- https://www.archiveteam.org/?diff=45973&oldid=45927 |
| 21:56:41 | <tech234a> | ^ updates based on discussion in #warrior |
| 21:59:11 | <@Kaz> | uh |
| 21:59:16 | <@Kaz> | but that's not true |
| 21:59:28 | <@Kaz> | "Changes made within Docker containers, such as downloaded files, are reset when containers are restarted, so each time you run the container, it is configured to be just like it was when you ran it the first time." |
| 21:59:46 | <@Kaz> | the rest, yeah looks okay |
| 22:02:43 | <@Kaz> | https://archiveteam.org/index.php?title=Running_Archive_Team_Projects_with_Docker&diff=45974&oldid=45973 |
| 22:03:31 | <@Kaz> | I've snipped the bit about asking for requeues too - project pilots should be setting an acceptable TTL on jobs which would mean this is no longer an issue. In some cases, we requeue once the todo queue is empty |
| 22:07:10 | <tech234a> | Thanks! |
| 22:11:09 | <atphoenix> | Kaz, which part of this is untrue? Or did I misread? I thought Craigle said the containers are pretty much ephemeral? "Changes made within Docker containers, such as downloaded files, are reset when containers are restarted, so each time you run the container, it is configured to be just like it was when you ran it the first time." |
| 22:11:34 | <@Kaz> | on the whole, contents are not reset when a container is restarted |
| 22:11:46 | <@Kaz> | deleted / recreated? absolutely, but not on a restart |
| 22:12:04 | <@Kaz> | and if utilising a volume, they're not even reset on a delete of the container |
| 22:12:17 | <atphoenix> | so they work much like VM instance |
| 22:12:30 | <@Kaz> | well yeah, I guess you could say that |
| 22:12:36 | <@Kaz> | while the contasiner exists, its data remains intact |
| 22:12:39 | <tech234a> | Craigle: FYI ^ |
| 22:25:00 | <purplebot> | Running Archive Team Projects with Docker edited by Kaz (-389, snip some incorrect info, also …) 22 minutes ago -- https://www.archiveteam.org/?diff=45974&oldid=45973 |
| 22:26:53 | <Craigle> | Yeah, that's my mistake. Running them for these projects, I don't keep them as much as delete and recreate as needed. I wasn't thinking big picture. |
| 22:39:38 | <@JAA> | The images.nga.gov job has slowed down a bit. ETA is still nearly 6 days. |
| 23:04:30 | | godane (godane) joins |
| 23:12:27 | | Arcorann (Arcorann) joins |
| 23:12:36 | | Matthww8 joins |
| 23:12:37 | | Matthww quits [Read error: Connection reset by peer] |
| 23:12:37 | | Matthww8 is now known as Matthww |
| 23:59:47 | | Peetz0r quits [Ping timeout: 252 seconds] |