| 00:00:03 | <@arkiver> | hey it oads for me |
| 00:00:30 | <pabs> | and it seems to use replayweb, the WBM snapshots I saw for archives just have: ReplayWeb.page could not be loaded due to the following error: SecurityError: The operation is insecure. |
| 00:00:51 | <nicolas17> | https://ghostarchive.org/archive/1FbIB?wr=false seems to work without JS |
| 00:00:53 | <@arkiver> | aha they load an actual WARC file for playback |
| 00:00:58 | <pabs> | oh me too |
| 00:01:01 | <@arkiver> | with same ID |
| 00:01:02 | <@arkiver> | nic |
| 00:01:05 | | cascode quits [Ping timeout: 256 seconds] |
| 00:01:32 | <@arkiver> | like https://ghostarchive.org/chimurai4/pAcDN.warc for https://ghostarchive.org/archive/pAcDN |
| 00:01:45 | <nicolas17> | interesting |
| 00:01:48 | <nicolas17> | so we could archive the warcs |
| 00:02:00 | <nicolas17> | or we could dig into the warcs to get *all* valid URLs |
| 00:02:12 | <@arkiver> | yeah, archive both those and the WARCs themselves |
| 00:02:27 | <@arkiver> | we'd store their WARCs in WARCs though, they'd not be directly indexed into the Wayback Machine |
| 00:02:32 | <yfdyh000> | I thought of archive.is, but archiving it seems unlikely. |
| 00:02:52 | <pabs> | would be nice to save .is too, would need DPoS though |
| 00:03:00 | <nicolas17> | is "chimurai4" fixed or where does it come from? |
| 00:03:17 | <@arkiver> | nicolas17: no idea, i can't look into it more for now, need to focus on some other stuff |
| 00:04:21 | <pabs> | looking in Firefox devtools, chimurai4 comes from the archive page itself |
| 00:04:51 | <pabs> | source="https://ghostarchive.org/chimurai4/1FbIB.warc" |
| 00:05:07 | <nicolas17> | well that's the same for 1FbIB and pAcDN |
| 00:05:14 | <nicolas17> | so I guess it's fixed |
| 00:05:35 | <IDK> | arkiver: any update on sogou wenwen? |
| 00:06:36 | <IDK> | my AB job felt pretty smooth, just had some major discoverbility issues due to need for post requests |
| 00:08:51 | | satoru346 quits [Read error: Connection reset by peer] |
| 00:10:18 | <nicolas17> | I tried bruteforcing IDs on ghostarchive (single thread, no parallelism), that did 3 req/s most |
| 00:10:26 | <nicolas17> | it's now not loading anymore and I don't know if I got banned |
| 00:10:43 | <@arkiver> | nicolas17: not loading for me either right now |
| 00:10:45 | <nicolas17> | or if the site fell over again |
| 00:10:48 | <IDK> | me neither |
| 00:11:00 | <nicolas17> | they might have a CDN |
| 00:11:14 | <nicolas17> | because I restarted my command |
| 00:11:22 | <nicolas17> | and 0FbIB took forever to return a 404 |
| 00:11:31 | <IDK> | nicolas17: they have buttflare but https://globalping.io/?measurement=2pmzedP8fGVNnplAu00021Nu6 |
| 00:11:32 | <nicolas17> | then 1FbIB downloaded in 90ms |
| 00:12:08 | <nicolas17> | "cf-cache-status: HIT" yep... |
| 00:13:09 | <nicolas17> | the 404s are now suspiciously taking 1 minute exactly |
| 00:14:10 | <nicolas17> | I suspect they're returning 404 on timeout, maybe at the CF level |
| 00:16:15 | <@arkiver> | we may have to get on archiving Serbian sites |
| 00:16:39 | <IDK> | whats happening in serbia? |
| 00:17:03 | <yfdyh000> | IDK: yzqzss at stwp previously said they had already started on sogou wenwen, privately. |
| 00:17:04 | <pokechu22> | arkiver: I think c3manu was working on Serbian elections stuff last night, not sure what else is going on there though |
| 00:17:25 | <Umbire> | I recall something to that effect |
| 00:17:26 | <pokechu22> | ah, .rs for "Republic of Serbia" |
| 00:17:47 | <IDK> | yfdyh000: ah, good to know, I thought it was the metadata group that said they want to do it |
| 00:17:54 | <Umbire> | nothing in my logs though |
| 00:17:56 | <@arkiver> | yfdyh000: IDK: right, i just checked with them again on wenwen. it would be good to have a project at Archive Team and get the data in the Wayback Machine. |
| 00:18:04 | <@arkiver> | IDK: blegh on that metadata group, don't trust them |
| 00:18:11 | <Umbire> | (at least not from a casual ctrl + F "Serb") |
| 00:18:26 | <IDK> | i dont either thats why im requesting an AT project :-) |
| 00:19:25 | <@arkiver> | yeah, last i heard is that the site was very unstable. my thinking is that neither copy will be complete if we run at full speed now too. |
| 00:19:37 | <@arkiver> | which is not ideal |
| 00:19:39 | <klea> | Yeah. |
| 00:19:53 | <klea> | Divide and conqueor, they say? |
| 00:20:20 | <@arkiver> | i just checked on how the site is doing - i'm ready to get a project up within a day |
| 00:22:37 | <nicolas17> | ghostarchive seems to have 1% of keyspace |
| 00:22:38 | <@arkiver> | pokechu22: more that the winning party wants to make major changes to the public broadcasting, news, government sites, etc. |
| 00:22:53 | <pokechu22> | Ah :| |
| 00:23:03 | <@arkiver> | yeah :/ |
| 00:23:18 | <@arkiver> | we may do a general .rs up to two steps deep or so as well |
| 00:23:19 | <pokechu22> | Like what happened with Hungary, I guess? |
| 00:23:23 | <@arkiver> | kind of yeah |
| 00:24:09 | <@arkiver> | IDK: i'm going to get back more on wenwen, as i said if the site is stable and running well, i'm ready for a project |
| 00:30:44 | <nicolas17> | ghostarchive IDs AAA[0-9a-zA-Z][0-9a-zA-Z] found 36 hits out of 3844 |
| 00:39:37 | | cascode_ quits [Ping timeout: 256 seconds] |
| 00:42:41 | | cascode joins |
| 00:59:13 | | snvy quits [Client Quit] |
| 01:01:13 | <pabs> | for anyone going to FOSDEM, I just got spam about a devroom that seems relevant to folks here "Archives, Libraries, and Digital Memory" https://aldm-devroom.github.io/cfp/ |
| 01:09:15 | | Jake quits [Quit: Leaving for a bit!] |
| 01:09:49 | | Ikata quits [Quit: The Lounge - https://thelounge.chat] |
| 01:10:10 | | Ikata (Ikata) joins |
| 01:10:12 | | Jake (Jake) joins |
| 01:15:47 | | cascode quits [Read error: Connection reset by peer] |
| 01:19:02 | | haiseiko quits [Quit: Connection closed for inactivity] |
| 01:22:02 | <nicolas17> | trying AAB?? now |
| 01:22:15 | <nicolas17> | can someone figure out if there's symbols in IDs too? |
| 01:23:44 | <nicolas17> | I'm only seeing letters and numbers, guess it's not base64 |
| 01:26:29 | <nicolas17> | pabs: does it seem feasible to make 900M requests to find the 9M archived pages? |
| 01:27:18 | <nicolas17> | AAB: 27/3884 |
| 01:27:21 | <pabs> | that seems like a lot, I don't know the stats for previous DPoS brute-forcing though |
| 01:27:54 | <nicolas17> | for a total of 63/7768 I discovered |
| 01:28:32 | <pabs> | which URL pattern are you brute-forcing? |
| 01:28:40 | <@JAA> | Depends entirely on their server capacity. |
| 01:28:55 | <@JAA> | 900M is completely feasible on our side. |
| 01:29:13 | <nicolas17> | for a in {0..9} {A..Z} {a..z}; do for b in {0..9} {A..Z} {a..z}; do echo https://ghostarchive.org/chimurai4/AAA$a$b.warc => this found 36 |
| 01:31:04 | | haiseiko (Haiseiko) joins |
| 01:32:46 | <pabs> | hmm lwp-request HEAD gets a cloudflare 403 but curl gets 200 even without a UA header |
| 01:32:57 | <pabs> | ah, entirely UA based. |
| 01:44:00 | <nicolas17> | if I'm right in my suspicion that a 404 after exactly 60 seconds could be fake... wget --timeout=50 should avoid it :P |
| 01:44:03 | | Umbire quits [Ping timeout: 265 seconds] |
| 01:49:16 | <nicolas17> | chimurai4 might be a storage server name |
| 01:49:25 | <nicolas17> | https://ghostarchive.org/chimurai4/3rI3n.warc redirects to https://ghostarchive.org/chimurai2/3rI3n.warc |
| 01:58:14 | | cascode joins |
| 01:59:13 | | cascode quits [Read error: Connection reset by peer] |
| 02:59:13 | | beastbg8 quits [Read error: Connection reset by peer] |
| 03:00:14 | | scotrod24 quits [Quit: The Lounge - https://thelounge.chat] |
| 03:01:23 | | scotrod24 joins |
| 03:19:24 | | entrox quits [Quit: The Lounge - https://thelounge.chat] |
| 03:20:53 | | entrox joins |
| 03:39:02 | | haiseiko quits [Client Quit] |
| 03:52:38 | | Island quits [Read error: Connection reset by peer] |
| 04:03:27 | | skankhunt42 quits [Quit: Ping timeout (120 seconds)] |
| 04:03:51 | | skankhunt42 (skankhunt42) joins |
| 04:04:06 | | satoru346 joins |
| 04:04:22 | | Wuzado quits [Quit: ZNC - https://znc.in] |
| 04:07:06 | | Wuzado joins |
| 04:13:25 | | DogsRNice joins |
| 04:48:41 | | wickedplayer494 quits [Ping timeout: 265 seconds] |
| 04:49:23 | | wickedplayer494 (wickedplayer494) joins |
| 05:06:45 | <ap> | IDK: how high of a rate did you hit wenwen with? it returns bad data, Forbidden msgs in chinese (sometimes), and fake "question not found" or empty/nulled/0-value JSONs that fit the correct struct still for me upon being rate limited with no rate limit headers and it sends HTTP 200 OK as the response. it also sends 302 + 200 OK on not found |
| 05:06:45 | <ap> | question pages. |
| 05:09:28 | | nexussfan quits [Remote host closed the connection] |
| 05:09:31 | <pokechu22> | Is wenwen the site that seemed to be proxying WBM data of an older version of the same site? |
| 05:21:38 | | Rdav1986 quits [Quit: The Lounge - https://thelounge.chat] |
| 05:21:47 | | Rdav19867 (Rdav1986) joins |
| 05:25:12 | <pokechu22> | (Tried to check but I couldn't find anything in my #archivebot logs about it. I think it was a different site though.) |
| 05:39:00 | <yfdyh000> | wenwen.sogou |
| 05:45:57 | | DogsRNice quits [Read error: Connection reset by peer] |
| 05:53:10 | | d02 joins |
| 06:34:48 | | haiseiko (Haiseiko) joins |
| 06:46:32 | | iigau1109 quits [Quit: Leaving] |
| 06:52:34 | | Webuser518704 joins |
| 06:53:09 | | Webuser518704 quits [Client Quit] |
| 06:55:27 | <c3manu> | IDK: Bulgaria and Serbia were holding elections yesteday, they just have a lot of parties. if you ever wondered where the term "Balkanization" has its roots.. |
| 06:56:32 | <c3manu> | IDK: additionally in Serbia you have this: https://en.wikipedia.org/wiki/2024%E2%80%932026_Serbian_anti-corruption_protests - i wasn't specifically archiving that, but they put up a list for election and from what i read in the headlines they kicked the incumbent right-wing's ass |
| 06:57:23 | <ericgallager> | Brazil had elections too |
| 07:00:31 | | do2 joins |
| 07:03:49 | | d02 quits [Ping timeout: 256 seconds] |
| 07:09:49 | | pixel quits [Ping timeout: 265 seconds] |
| 07:24:16 | | do2 quits [Client Quit] |
| 07:34:34 | | jmcdowan17384|m joins |
| 07:38:07 | <pabs> | arkiver: https://nitter.click/ has a click-through thing |
| 07:38:31 | <pabs> | https://nitter.jaydenha.uk/ has some JS thing |
| 08:13:08 | | satoru346 quits [Ping timeout: 265 seconds] |
| 08:13:19 | | satoru346 joins |
| 08:19:09 | | cascode joins |
| 08:25:22 | | cascode quits [Read error: Connection reset by peer] |
| 08:32:13 | | zsrtrgh quits [Quit: WeeChat 4.1.1] |
| 08:33:45 | | zsrtrgh (zsrtrgh) joins |