| 00:25:30 | | etnguyen03 quits [Client Quit] |
| 00:40:03 | | Guest58 joins |
| 00:55:00 | | Guest58 quits [Client Quit] |
| 00:55:19 | | Guest58 joins |
| 01:01:44 | | DogsRNice_ joins |
| 01:03:14 | | DogsRNice quits [Ping timeout: 260 seconds] |
| 01:03:41 | | DogsRNice joins |
| 01:06:44 | | DogsRNice_ quits [Ping timeout: 260 seconds] |
| 01:07:59 | | etnguyen03 (etnguyen03) joins |
| 01:17:26 | | egallager joins |
| 01:18:59 | | ericgallager quits [Ping timeout: 260 seconds] |
| 01:19:55 | | Guest58 quits [Client Quit] |
| 01:20:46 | | Guest58 joins |
| 01:36:29 | | Guest58 quits [Ping timeout: 260 seconds] |
| 01:39:00 | | Guest58 joins |
| 01:49:45 | | lennier2 quits [Ping timeout: 258 seconds] |
| 01:53:18 | | lennier2 joins |
| 01:55:53 | | Guest58 quits [Client Quit] |
| 02:16:18 | | Guest58 joins |
| 02:30:22 | | Guest58 quits [Client Quit] |
| 02:31:44 | | Guest58 joins |
| 02:35:51 | | croissant_ joins |
| 02:38:03 | | croissant quits [Ping timeout: 258 seconds] |
| 02:59:34 | | Guest58 quits [Client Quit] |
| 03:07:40 | | etnguyen03 quits [Remote host closed the connection] |
| 03:09:55 | | Guest58 joins |
| 03:11:10 | | ericgallager joins |
| 03:11:34 | | egallager quits [Ping timeout: 260 seconds] |
| 03:16:46 | | Guest58_ joins |
| 03:16:49 | | Guest58 quits [Ping timeout: 260 seconds] |
| 03:17:55 | | Webuser728583 joins |
| 03:18:11 | | Webuser728583 quits [Client Quit] |
| 03:18:35 | | Guest58_ quits [Client Quit] |
| 03:21:47 | | Guest58 joins |
| 03:24:09 | | DogsRNice quits [Read error: Connection reset by peer] |
| 03:26:41 | | Guest58 quits [Client Quit] |
| 03:32:52 | | Shjosan quits [Quit: Am sleepy (-, – )…zzzZZZ] |
| 03:33:24 | | Shjosan (Shjosan) joins |
| 03:34:58 | | Island quits [Read error: Connection reset by peer] |
| 04:01:46 | | Radzig quits [Quit: ZNC 1.10.1 - https://znc.in] |
| 04:02:15 | | Radzig joins |
| 04:24:35 | | Webuser593583 joins |
| 04:28:34 | | ericgallager quits [Ping timeout: 260 seconds] |
| 04:29:39 | | Webuser593583 quits [Client Quit] |
| 04:29:41 | | ericgallager joins |
| 04:44:55 | | Guest58 joins |
| 06:08:15 | | SootBector quits [Remote host closed the connection] |
| 06:09:23 | | SootBector (SootBector) joins |
| 06:13:52 | | rohvani quits [Ping timeout: 258 seconds] |
| 06:29:27 | | HackMii quits [Remote host closed the connection] |
| 06:33:41 | | HackMii (hacktheplanet) joins |
| 06:36:46 | | ConstantK joins |
| 07:05:26 | | lemuria quits [Remote host closed the connection] |
| 07:16:36 | | dendory quits [Read error: Connection reset by peer] |
| 07:16:58 | | dendory (dendory) joins |
| 07:21:49 | | Cornelius quits [Ping timeout: 260 seconds] |
| 07:23:33 | | Wohlstand (Wohlstand) joins |
| 07:42:25 | | chunkynutz60 quits [Read error: Connection reset by peer] |
| 07:42:42 | | chunkynutz60 joins |
| 08:13:33 | | Cornelius (Cornelius) joins |
| 08:26:07 | | tertu2 quits [Ping timeout: 258 seconds] |
| 08:34:04 | | tertu (tertu) joins |
| 11:00:02 | | Bleo182600722719623455222 quits [Quit: The Lounge - https://thelounge.chat] |
| 11:02:47 | | Bleo182600722719623455222 joins |
| 11:45:31 | | Dada joins |
| 12:18:25 | | tek_dmn quits [Ping timeout: 258 seconds] |
| 12:19:19 | | Snivy quits [Ping timeout: 260 seconds] |
| 12:22:38 | | benjins3_ quits [Ping timeout: 258 seconds] |
| 12:38:29 | | benjins3 joins |
| 12:40:06 | | tek_dmn (tek_dmn) joins |
| 12:56:25 | | beastbg8_ quits [Read error: Connection reset by peer] |
| 12:57:07 | | beastbg8_ joins |
| 12:59:12 | <twiswist> | Just tried ludios-wpull because someone mentioned it above, but it's missing --rejected-log so I'll stick with wget. This omission is not documented in the "Differences between Wpull and Wget" page |
| 13:13:21 | | egallager joins |
| 13:15:19 | | ericgallager quits [Ping timeout: 260 seconds] |
| 13:22:26 | | FiTheArchiver joins |
| 13:22:57 | | FiTheArchiver quits [Read error: Connection reset by peer] |
| 13:25:01 | | FiTheArchiver joins |
| 13:26:24 | | egallager quits [Ping timeout: 260 seconds] |
| 13:28:22 | | ericgallager joins |
| 13:31:30 | | FiTheArchiver1 joins |
| 13:35:09 | | FiTheArchiver quits [Ping timeout: 260 seconds] |
| 13:40:13 | | FiTheArchiver joins |
| 13:40:19 | | Dada quits [Remote host closed the connection] |
| 13:41:28 | | FiTheArchiver3 joins |
| 13:42:46 | | Dada joins |
| 13:43:54 | | FiTheArchiver1 quits [Ping timeout: 258 seconds] |
| 13:44:40 | | FiTheArchiver quits [Ping timeout: 258 seconds] |
| 13:45:03 | | FiTheArchiver1 joins |
| 13:48:15 | | FiTheArchiver joins |
| 13:48:34 | | FiTheArchiver3 quits [Ping timeout: 260 seconds] |
| 13:48:55 | | FiTheArchiver quits [Read error: Connection reset by peer] |
| 13:51:04 | | FiTheArchiver joins |
| 13:51:11 | | FiTheArchiver1 quits [Ping timeout: 258 seconds] |
| 13:51:27 | | Island joins |
| 13:53:28 | | FiTheArchiver1 joins |
| 13:55:23 | | FiTheArchiver3 joins |
| 13:55:23 | | FiTheArchiver3 quits [Remote host closed the connection] |
| 13:56:05 | | cipherrot (petrichor) joins |
| 13:56:56 | | FiTheArchiver quits [Ping timeout: 258 seconds] |
| 13:57:19 | | petrichor quits [Ping timeout: 258 seconds] |
| 13:59:04 | | FiTheArchiver1 quits [Ping timeout: 260 seconds] |
| 14:04:16 | | Webuser694459 joins |
| 14:25:19 | | pabs quits [Ping timeout: 260 seconds] |
| 14:26:05 | | pabs (pabs) joins |
| 14:34:30 | | ducky quits [Ping timeout: 258 seconds] |
| 14:35:14 | <@JAA> | twiswist: It is mentioned there. It falls under 'Features greater than Wget 1.15'. |
| 14:39:21 | <twiswist> | Oh, alright |
| 14:48:26 | | ducky (ducky) joins |
| 15:05:56 | | wickedplayer494 quits [Ping timeout: 258 seconds] |
| 15:13:23 | <pabs> | https://www.iconfinder.com/ "9,375,000+ free and premium vector icons..." shuts down November 15, 2025 according to https://support.freepik.com/s/article/Iconfinder-Closure-FAQs?language=en_US https://en.wikipedia.org/wiki/Iconfinder |
| 15:13:35 | | Dada quits [Remote host closed the connection] |
| 15:15:18 | | Dada joins |
| 15:28:17 | | Webuser694459 quits [Client Quit] |
| 15:30:04 | | wickedplayer494 joins |
| 15:45:14 | | wickedplayer494 quits [Ping timeout: 260 seconds] |
| 15:45:33 | | wickedplayer494 joins |
| 15:46:40 | | wickedplayer494 is now authenticated as wickedplayer494 |
| 16:37:09 | | AK quits [Ping timeout: 260 seconds] |
| 16:42:01 | | AK (AK) joins |
| 16:53:29 | | BornOn420 quits [Ping timeout: 260 seconds] |
| 17:05:46 | <b3nzo> | the sitemap likely has all the urls https://www.iconfinder.com/iconfinder-googlesitemap-index.xml |
| 17:13:41 | <@arkiver> | ouch |
| 17:13:50 | <@arkiver> | nearly 10 million icons sounds amazing though |
| 17:14:01 | <@arkiver> | b3nzo: make sure you archive that! will come in handy if there's a project for it |
| 17:14:05 | <@arkiver> | (unless AB can handle it) |
| 17:14:09 | <@arkiver> | (can it?) |
| 17:19:44 | <b3nzo> | arkiver: i can run my grab-site instance for it, but if its added to AB or warrior then ig its better to just go through that. im pretty sure, my IPs would get blocked pretty soon if i have have to archive 10mil |
| 17:25:16 | <@JAA> | AB will not be happy with this one. Each icon has over a dozen downloads, so that alone brings it to well over 100M URLs. |
| 17:25:21 | <dendory> | The site seems to already have tons of crawls on the wayback machine, and perhaps more useful, there's a zip file of all their icons here: https://archive.org/details/perma_cc_4L5X-3ZBR |
| 17:26:41 | <@JAA> | Wrong link? That's just one specific icon. |
| 17:27:37 | | Snivy (Snivy) joins |
| 17:27:45 | <nicolas17> | discover the urls and put them into #// |
| 17:27:47 | <nicolas17> | hashtag yolo |
| 17:27:55 | <nicolas17> | (don't do that we'd kill the site) |
| 17:28:01 | <dendory> | My bad.. seems like there's multiple zip files of single icons.. hmm.. Maybe it'd be worth it to compile an archive of all the icons. |
| 17:30:09 | <@JAA> | Interestingly, the homepage mentions '6 million icons' (the 9M+ figure includes illustrations and whatnot), but then https://www.iconfinder.com/icons says <title>18,861+ high-quality icons - Iconfinder</title>. |
| 17:31:02 | <dendory> | Well I easily found that you can go from https://www.iconfinder.com/icons/1/download/png/128 to https://www.iconfinder.com/icons/100000/download/png/128 and download those as 128x128 png files.. might get a script up and zip them up. |
| 17:31:57 | <@JAA> | Yes, the IDs are sequential, and you don't need to know the slug. |
| 17:32:07 | <@JAA> | IDs go to over 13 million: https://www.iconfinder.com/search/icons?sort=timestamp_published |
| 17:32:10 | <@arkiver> | JAA: alright, i'll look into getting a Warrior project started for this one |
| 17:32:14 | <@arkiver> | nice that the URLs are sequential |
| 17:32:29 | <@arkiver> | let's make sure it's on deathwatch so it's not forgotten |
| 17:32:38 | <nicolas17> | JAA: might need to know the slug for WBM replay right? |
| 17:32:53 | <@JAA> | Looks like 'only' 216k are free: https://www.iconfinder.com/search/icons?sort=timestamp_published&price=free |
| 17:32:59 | <@JAA> | nicolas17: It redirects. |
| 17:33:07 | <@arkiver> | JAA: does that mean only web pages accessible for those 216k? |
| 17:33:16 | <@JAA> | No, web pages are accessible for all. |
| 17:33:20 | <nicolas17> | oh perfect |
| 17:33:30 | <@JAA> | 'Download with Pro': https://www.iconfinder.com/icons/13085800/advertising_and_promotions_business_management_finance_planning_analysis_technology_icon |
| 17:33:35 | <@arkiver> | just checked premium is available too as web pages |
| 17:35:01 | <@arkiver> | i was just going to take some off time |
| 17:35:31 | <@arkiver> | but let's add it to deathwatch and i'll look later into a project for it |
| 17:37:59 | <@JAA> | Deathwatch++ |
| 17:37:59 | <eggdrop> | [karma] 'Deathwatch' now has 1 karma! |
| 17:38:07 | <dendory> | Will need a lot of throttling because it blocks with error 429 Too Many Requests after just a handful of requests. |
| 17:38:30 | <@JAA> | Ah, I was going to say I could try with qwarc. Nevermind then. |
| 17:38:51 | <@arkiver> | i'm sure we can come up with a solution :) |
| 17:38:53 | <dendory> | It blocks at around 50 for a couple of seconds.. |
| 17:38:54 | <@arkiver> | yay for many IPs |
| 17:39:05 | <@arkiver> | dendory: what does "around 50" mean? |
| 17:40:03 | <dendory> | I did around 50 wget requests manually just to test various IDs and got blocked |
| 17:40:39 | <b3nzo> | the sitemaps have a total of 7,203,306 urls |
| 17:40:58 | <@arkiver> | ah |
| 17:41:00 | <@arkiver> | that is not too bad |
| 17:41:07 | | @arkiver will be afk for some time |
| 17:41:31 | <dendory> | I'm just trying to get the highest resolution free PNG icons to zip them up, I think that will be far more useful than all 7M pages |
| 17:42:05 | <@JAA> | Metadata is important, especially licence and author. |
| 17:42:12 | <@JAA> | URL preservation is also important. |
| 17:42:48 | <@arkiver> | yeah |
| 17:42:57 | | Lord_Nightmare quits [Quit: ZNC - http://znc.in] |
| 17:43:29 | <@JAA> | (It looks like a lot of the free icons there are under CC BY, so attribution is even required.) |
| 17:44:04 | <dendory> | Yeah every icon I found so far is CC BY 4.0 |
| 17:44:40 | <dendory> | I just think a zip file of all the free icons would be useful to have |
| 17:45:02 | <b3nzo> | are any specific reasons for archiving the highest quality image rather than archiving the preview page? |
| 17:45:21 | <nicolas17> | can confirm, "wget https://www.iconfinder.com/icons/1000{00..99}/download/png/128" got me 429'd after 57 requests |
| 17:46:05 | <dendory> | the block lasts 60 secs btw. |
| 17:47:09 | <@arkiver> | that rate limiting is really not too bad |
| 17:48:15 | | Lord_Nightmare (Lord_Nightmare) joins |
| 17:55:50 | <@JAA> | So almost 1 req/s, yeah, should be fine then with DPoS. |
| 17:58:00 | <nicolas17> | sleep 1 between requests, tell people to use concurrency 1 per IP? |
| 18:00:52 | <nicolas17> | wait I found another issue |
| 18:01:24 | <nicolas17> | my wget started getting redirects to login |
| 18:06:24 | | ducky quits [Ping timeout: 260 seconds] |
| 18:07:05 | <nicolas17> | can't reproduce... |
| 18:07:58 | <nicolas17> | oh there it is |
| 18:08:18 | <@JAA> | We might want a channel for this one. |
| 18:08:27 | <@JAA> | Is #iconloser too harsh? |
| 18:09:45 | <nicolas17> | https://www.iconfinder.com/icons/100530/download/png/128 -> 302 Found https://www.iconfinder.com/user/login?redirect_to=https%3A//www.iconfinder.com/icons/100530/download/png/128 |
| 18:09:54 | <nicolas17> | went away when I restarted wget, is this per TCP connection? |
| 18:17:18 | <dendory> | I've downloaded over a thousand so far using wget and doing fine so far. There are lots of missing IDs btw, doesnt seem to be an issue to just skip them. |
| 18:17:46 | <nicolas17> | are you using separate wget calls for each URL? |
| 18:19:35 | <dendory> | Just did a really quick shell script, more of a test than anything: https://sharetext.io/45b80631 |
| 18:20:28 | <nicolas17> | yeah one wget per URL |
| 18:20:42 | <nicolas17> | if you pass multiple URLs at once, wget reuses the connection for multiple requests |
| 18:20:49 | <@JAA> | You can use https://transfer.archivete.am/ to share things without JS nonsense sites. |
| 18:21:17 | <nicolas17> | and it turns out once you send 100 requests in the same connection, the website starts redirecting to login (?!) |
| 18:22:02 | <nicolas17> | (I'm using -w 0.9 to make wget wait between requests, to avoid the 429) |
| 18:25:08 | <h2ibot> | Nicolas17v2 edited Deathwatch (+193, /* 2025 */ Add iconfinder): https://wiki.archiveteam.org/?diff=57592&oldid=57574 |
| 18:25:50 | <dendory> | Not sure, I just left the script running and I'm at 1300... other than the 60sec waits and random 404s it's going fine. |
| 18:26:19 | <dendory> | A better way to do this would be to fetch metadata to at least get the name of the icons instead of just IDs.. might look into that |
| 18:26:45 | <nicolas17> | there's a name in the HTTP headers |
| 18:27:08 | <dendory> | sweet |
| 18:27:10 | <nicolas17> | content-disposition: attachment; filename=99400_dribbble_icon.png |
| 18:32:30 | | ducky (ducky) joins |
| 18:33:47 | | SootBector quits [Remote host closed the connection] |
| 18:34:54 | | SootBector (SootBector) joins |
| 18:39:37 | | DogsRNice joins |
| 18:41:59 | <dendory> | Updated the script to use the right filenames: https://transfer.archivete.am/lxpaJ/iconfinder.sh |
| 18:42:01 | <eggdrop> | inline (for browser viewing): https://transfer.archivete.am/inline/lxpaJ/iconfinder.sh |
| 19:01:42 | | Ointment8862 (Ointment8862) joins |
| 19:03:30 | | Guest58 quits [Quit: My Mac has gone to sleep. ZZZzzz…] |
| 20:10:04 | | NF885 (NF885) joins |
| 20:18:53 | | NF885 quits [Client Quit] |
| 20:33:17 | | ThreeHM quits [Quit: WeeChat 4.7.1] |
| 20:35:15 | | ThreeHM (ThreeHeadedMonkey) joins |
| 20:39:19 | | Webuser537497 joins |
| 20:39:47 | | Webuser537497 quits [Client Quit] |
| 20:55:34 | | etnguyen03 (etnguyen03) joins |
| 21:02:35 | | etnguyen03 quits [Client Quit] |
| 21:03:04 | | etnguyen03 (etnguyen03) joins |
| 21:20:45 | | cyanbox quits [Read error: Connection reset by peer] |
| 21:22:47 | | cyanbox joins |
| 21:24:16 | | twiswist quits [Quit: twiswist] |
| 21:25:21 | | twiswist (twiswist) joins |
| 21:31:25 | | etnguyen03 quits [Client Quit] |
| 21:39:47 | | etnguyen03 (etnguyen03) joins |
| 21:41:37 | | Dada quits [Remote host closed the connection] |
| 21:51:58 | | Wohlstand quits [Remote host closed the connection] |
| 22:21:12 | | BearFortress joins |
| 22:31:02 | | abirkill (abirkill) joins |
| 22:32:22 | | Webuser301963 joins |
| 22:32:57 | <Webuser301963> | Hey could you archive this site please? https://crba.dedyn.io |
| 22:41:25 | <pokechu22> | Archivebot job started |
| 22:44:41 | | Webuser301963 quits [Client Quit] |
| 22:48:55 | | matoro quits [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.] |
| 22:49:14 | | matoro joins |
| 23:13:15 | | etnguyen03 quits [Client Quit] |
| 23:26:37 | | rohvani joins |
| 23:40:37 | | Shard7 quits [Quit: Im doing something rq. Il brb] |
| 23:45:43 | | Shard7 (Shard) joins |
| 23:53:07 | | etnguyen03 (etnguyen03) joins |