| 00:30:28 | | sonick (sonick) joins |
| 01:00:02 | | dm4v quits [Client Quit] |
| 01:01:34 | | dm4v joins |
| 01:01:36 | | dm4v is now authenticated as dm4v |
| 01:01:36 | | dm4v quits [Changing host] |
| 01:01:36 | | dm4v (dm4v) joins |
| 01:40:44 | | driib791 (driib) joins |
| 01:44:34 | | driib79 quits [Ping timeout: 265 seconds] |
| 01:44:34 | | driib791 is now known as driib79 |
| 02:03:54 | | dm4v quits [Ping timeout: 265 seconds] |
| 02:04:14 | | dm4v joins |
| 02:04:16 | | dm4v is now authenticated as dm4v |
| 02:04:16 | | dm4v quits [Changing host] |
| 02:04:16 | | dm4v (dm4v) joins |
| 02:18:39 | | qwertyasdfuiopghjkl joins |
| 02:25:10 | | AnotherIki quits [Ping timeout: 265 seconds] |
| 03:29:22 | | Iki joins |
| 03:36:40 | | Meli quits [Ping timeout: 252 seconds] |
| 03:38:12 | | Meli (Meli) joins |
| 04:01:57 | | qw3rty_ joins |
| 04:05:43 | | qw3rty__ quits [Ping timeout: 258 seconds] |
| 04:59:29 | | qw3rty__ joins |
| 05:03:13 | | qw3rty_ quits [Ping timeout: 258 seconds] |
| 06:55:48 | | pabs quits [Client Quit] |
| 07:02:51 | | pabs (pabs) joins |
| 07:03:35 | | tzt quits [Ping timeout: 258 seconds] |
| 07:09:15 | | BlueMaxima quits [Client Quit] |
| 07:18:49 | | pabs quits [Client Quit] |
| 07:58:11 | | HP_Archivist (HP_Archivist) joins |
| 08:02:27 | | useretail joins |
| 09:21:56 | | pabs (pabs) joins |
| 09:29:46 | | pabs quits [Ping timeout: 252 seconds] |
| 09:38:21 | | qwertyasdfuiopghjkl35 joins |
| 09:40:20 | | qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds] |
| 10:07:01 | | godane1 joins |
| 10:08:41 | | Terbium quits [Quit: http://quassel-irc.org - Chat comfortably. Anywhere.] |
| 10:09:17 | | Terbium joins |
| 10:09:22 | | godane quits [Ping timeout: 252 seconds] |
| 11:14:29 | | spirit joins |
| 11:25:57 | | Minkafighter quits [Quit: The Lounge - https://thelounge.chat] |
| 11:26:41 | | Minkafighter joins |
| 13:32:13 | | Minkafighter quits [Client Quit] |
| 13:32:56 | | Minkafighter joins |
| 13:41:52 | | Arcorann_ quits [Ping timeout: 258 seconds] |
| 13:52:28 | | Barto quits [Ping timeout: 265 seconds] |
| 13:52:42 | | Barto (Barto) joins |
| 14:01:29 | | qwertyasdfuiopghjkl35 is now known as qwertyasdfuiopghjkl |
| 14:04:50 | | onetruth quits [Read error: Connection reset by peer] |
| 14:57:46 | | yawkat quits [Ping timeout: 258 seconds] |
| 15:00:30 | | AK quits [Quit: AK] |
| 15:00:53 | | AK (AK) joins |
| 15:06:07 | | qwertyasdfuiopghjkl57 joins |
| 15:08:25 | | qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds] |
| 15:08:40 | | spirit quits [Client Quit] |
| 16:05:04 | | qwertyasdfuiopghjkl57 is now known as qwertyasdfuiopghjkl |
| 16:08:42 | | qwertyasdfuiopghjkl quits [Remote host closed the connection] |
| 16:11:46 | | qwertyasdfuiopghjkl joins |
| 16:20:11 | | pabs (pabs) joins |
| 16:42:22 | | programmerq quits [Read error: Connection reset by peer] |
| 17:00:09 | | @Fusl_ quits [Remote host closed the connection] |
| 17:00:09 | | pnJay___ quits [Remote host closed the connection] |
| 17:00:09 | | ghuntley quits [Remote host closed the connection] |
| 17:00:48 | <TheTechRobo> | Aight IDK_ |
| 17:00:58 | <IDK_> | Alright |
| 17:01:04 | <IDK_> | so I found this from a roblox discord server |
| 17:01:09 | <TheTechRobo> | can you write in Python? |
| 17:01:23 | <IDK_> | No sorry |
| 17:01:29 | <TheTechRobo> | Ok |
| 17:01:41 | <TheTechRobo> | I'll see if I can write a script for you, but 1sec i have to improve mine |
| 17:01:54 | <IDK_> | k |
| 17:07:20 | <TheTechRobo> | IDK_ : Looking over your scrape it shouldn't be _too_ difficult |
| 17:07:24 | | Ctrl-S quits [Remote host closed the connection] |
| 17:07:24 | | justcool393 quits [Remote host closed the connection] |
| 17:07:24 | | @hook54321 quits [Remote host closed the connection] |
| 17:08:11 | <IDK_> | I mean its kinda long and discord chat exporter wrote it in a really inconvinent way |
| 17:08:24 | <TheTechRobo> | Yes, but that's where re.split comes in |
| 17:08:49 | <TheTechRobo> | actually no we can just use .split |
| 17:09:08 | <TheTechRobo> | nvm yes well need re.split |
| 17:10:06 | <TheTechRobo> | Basically, we just split by both spaces and newlines, then see if it starts with https:// or http:// |
| 17:10:14 | <@JAA> | `grep -o 'http\S*'` is a decent basic approach. |
| 17:10:14 | <TheTechRobo> | by iterating over each item |
| 17:10:31 | <TheTechRobo> | JAA: Now I feel stupid :D |
| 17:11:18 | <@JAA> | It'll fail when you have parentheses around links and stuff like that, obviously. |
| 17:11:27 | <TheTechRobo> | But so would mine |
| 17:11:50 | <@JAA> | You'd need a much more complex regex for that. |
| 17:11:50 | | aarchi quits [Remote host closed the connection] |
| 17:11:50 | | Dragnog quits [Remote host closed the connection] |
| 17:11:50 | | Dallas quits [Remote host closed the connection] |
| 17:11:50 | | @HCross quits [Remote host closed the connection] |
| 17:11:51 | | themadpro quits [Remote host closed the connection] |
| 17:12:21 | <IDK> | uxbridge is down I see |
| 17:12:30 | <IDK> | uxbridge.irccloud.com |
| 17:14:07 | <@JAA> | This handles stuff wrapped in brackets or angle brackets in addition to things surrounded by whitespace: `grep -Po '\(\Khttp\S*(?=\))|<\Khttp\S*(?=>)|(^|\s)http\S*'` |
| 17:15:22 | <@JAA> | Er |
| 17:15:27 | <@JAA> | grep -Po '\(\Khttp\S*(?=\))|<\Khttp\S*(?=>)|(^|\s)\Khttp\S*' |
| 17:16:01 | <TheTechRobo> | JAA: I need some regex help. I'm going to assume that the only valid letters in a URL would be soemthing like [a-ZA-Z/%], so I want to match everything OTHER than that for re.split. a) would that be a good idea and b) what's the regex for that? |
| 17:16:19 | <TheTechRobo> | Oh right and also 0-9 |
| 17:16:28 | <@JAA> | And ? and : and plenty others. |
| 17:16:42 | <TheTechRobo> | Yep, I'm incredible stupid |
| 17:16:47 | <@JAA> | URL normalisation is also a mess. |
| 17:17:12 | <@JAA> | Non-punycoded IDNs etc. |
| 17:18:18 | <@JAA> | But the syntax to match any character not in some set is preceding the list with a caret. [a-z] matches a lower-case Latin alphabet letter, [^a-z] matches anything else. |
| 17:18:55 | | tech234a quits [Ping timeout: 252 seconds] |
| 17:19:43 | <@JAA> | Another fun one is if your URLs come from HTML without handling of HTML entities, so you need to fix & etc. |
| 17:22:26 | <TheTechRobo> | I decided against it and just to take my 50k urls by splitting every space, newline, and parentheses |
| 17:22:42 | <TheTechRobo> | 50k is good enough, anyway |
| 17:23:05 | <TheTechRobo> | And the URLs that don't fit those reqs will be so niche that I'd only gain a few hundred |
| 17:23:20 | <@JAA> | TheTechRobo: Re #//, tip, when including a URL in a sentence, always put a space after it. Because otherwise you have to handle commas and full stops at the end of the URL as well (and it's impossible to know in general whether they're part of the URL or not). |
| 17:23:26 | | @jrwr quits [Ping timeout: 622 seconds] |
| 17:23:56 | <TheTechRobo> | Ah, OK |
| 17:24:32 | <@JAA> | Or wrap it in angle brackets, which is the other convention I know. |
| 17:24:42 | <TheTechRobo> | Nice thing about my script is that I can always run it again without having to filter out the old ones myself |
| 17:25:30 | <TheTechRobo> | since I added an auto-ignore-list |
| 17:25:44 | | sonick quits [Ping timeout: 622 seconds] |
| 17:28:25 | | mgrandi quits [Ping timeout: 622 seconds] |
| 17:28:25 | | IDK quits [Ping timeout: 622 seconds] |
| 17:29:11 | | NotEggplant quits [Ping timeout: 622 seconds] |
| 17:30:03 | <TheTechRobo> | If anyone needs the script (they are using the desktop app of Discordhistorytracker (.dht)) here is my script: https://github.com/TheTechRobo/discordhistorytracker-urls-extractor |
| 17:30:13 | <TheTechRobo> | Feel free to send pull requests + bug reports + fix my buggy code |
| 17:30:43 | | revi quits [Ping timeout: 622 seconds] |
| 17:35:00 | | benjinsmith joins |
| 17:38:00 | | benjins quits [Ping timeout: 258 seconds] |
| 17:49:39 | | benjinsmith is now known as benjins |
| 17:49:41 | | benjins is now authenticated as benjins |
| 17:51:48 | | HP_Archivist quits [Ping timeout: 258 seconds] |
| 17:56:03 | | spirit joins |
| 18:05:01 | | yawkat (yawkat) joins |
| 18:14:05 | | HP_Archivist (HP_Archivist) joins |
| 18:27:44 | <IDK_> | thetechrobo: thx |
| 18:27:52 | <TheTechRobo> | IDK_: for what? |
| 18:27:57 | <TheTechRobo> | *IDK_ |
| 18:28:01 | <TheTechRobo> | ** IDK_ |
| 18:28:17 | <IDK_> | The discord url extractor |
| 18:52:22 | | HCross (HCross) joins |
| 18:52:22 | | @ChanServ sets mode: +o HCross |
| 18:55:03 | | HP_Archivist quits [Ping timeout: 258 seconds] |
| 18:55:07 | | Dallas (Dallas) joins |
| 18:55:10 | | aarchi (aarchi) joins |
| 18:55:44 | | Ctrl-S joins |
| 18:56:58 | <IDK_> | TheTechRobo: Should I start my own discord tracking archive |
| 18:57:09 | <TheTechRobo> | IDK_ : wdym |
| 18:57:30 | <IDK_> | or is there a archive I can contribute in |
| 18:57:44 | | hook54321 (hook54321) joins |
| 18:57:44 | | @ChanServ sets mode: +o hook54321 |
| 18:57:46 | | Dragnog joins |
| 18:57:52 | <IDK_> | https://server7.kiska.pw/uploads/2f7d89fa27fe718d/Discord%20History%20Tracker%2011_21_2021%207_57_34%20PM.png |
| 18:58:27 | | justcool393 (justcool393) joins |
| 18:59:35 | | themadpro (themadpro) joins |
| 18:59:47 | | jrwr (jrwr) joins |
| 18:59:47 | | @ChanServ sets mode: +o jrwr |
| 19:03:22 | <TheTechRobo> | Press CTRL+SHIFT+I in the discord client and copy the tracking script |
| 19:03:38 | <IDK_> | Yep its scraping here |
| 19:03:43 | <TheTechRobo> | AFAIK there's not currently a way to send it to an outside serve,r although you could modify the script |
| 19:03:50 | <TheTechRobo> | Nice |
| 19:03:56 | <IDK_> | But should I send the log here |
| 19:04:11 | <IDK_> | I dont think you could get banned for this or could you |
| 19:04:30 | <@JAA> | Not aware of a coordinated project like that. There's #discard, but it uses custom software and hasn't been very active lately. |
| 19:04:35 | | revi (revi) joins |
| 19:08:02 | <IDK_> | im kinda wondering has anyone ever gotten account disabled for this |
| 19:08:06 | <IDK_> | *aka banned |
| 19:08:07 | | tech234a (tech234a) joins |
| 19:09:10 | | NotEggplant joins |
| 19:09:32 | <TheTechRobo> | IDK_ it is against tos |
| 19:09:36 | | mgrandi (mgrandi) joins |
| 19:09:38 | <TheTechRobo> | so maybe |
| 19:10:02 | | Fusl_ (Fusl) joins |
| 19:10:02 | | @ChanServ sets mode: +o Fusl_ |
| 19:10:24 | | sonick (sonick) joins |
| 19:10:33 | | pnJay___ joins |
| 19:11:35 | | ghuntley joins |
| 19:11:55 | <IDK_> | TheTechRobo: How long have you been running this |
| 19:12:00 | <IDK_> | And not getting banned |
| 19:16:00 | | IDK (IDK) joins |
| 19:21:29 | <IDK> | back on main |
| 19:23:39 | | qwertyasdfuiopghjkl quits [Ping timeout: 244 seconds] |
| 19:40:57 | <IDK_> | Thetechrobo: is using someone else's tracking code unsafe or impossible |
| 19:42:59 | <TheTechRobo> | IDK_ : Not impossible but youll have to modify it to point to their server |
| 19:43:04 | <TheTechRobo> | IDK_ : Idk |
| 19:43:12 | <TheTechRobo> | Something like 6mo I started this? |
| 19:43:26 | <TheTechRobo> | For that reason, you're not getting my Discord username :P |
| 19:43:40 | <TheTechRobo> | Although since you have my GitHub, you'll find it pretty easy to get |
| 19:43:56 | <IDK_> | I mean- not getting discord username |
| 19:45:04 | <TheTechRobo> | Normally the tracking script points to localhost |
| 19:45:52 | <TheTechRobo> | When I get my own Vultr VPS, I'll probably open it up for connections, so we all can connect to a centralised DB |
| 19:45:57 | <TheTechRobo> | Of course, it'll have downloads too |
| 19:46:29 | <TheTechRobo> | I don't want ArchiveTeam to have to archive my archive of Discord :P |
| 20:12:06 | | spirit quits [Client Quit] |
| 20:24:44 | | LeGoupil joins |
| 20:28:55 | <IDK> | yes |
| 20:50:04 | | systwi quits [Ping timeout: 265 seconds] |
| 20:56:52 | | HP_Archivist (HP_Archivist) joins |
| 21:34:45 | | systwi (systwi) joins |
| 21:37:15 | | thelounge31 quits [Quit: Ping timeout (120 seconds)] |
| 21:45:12 | | hexa- quits [Quit: WeeChat 3.1] |
| 21:46:36 | | hexa- (hexa-) joins |
| 22:09:16 | | Arcorann_ joins |
| 22:14:02 | | LeGoupil quits [Client Quit] |
| 22:32:45 | <duce1337> | how do i open/view .warc files? |
| 22:37:48 | <@JAA> | Depends on what you're trying to do exactly. |
| 22:38:25 | <duce1337> | downloaded one of the files from archivebot saves, extracted it and got 1 .warc file |
| 22:38:42 | <duce1337> | how do i view it and see what's inside? |
| 22:38:44 | <duce1337> | JAA: |
| 22:42:37 | <@JAA> | duce1337: pywb for a local Wayback Machine is probably what you're looking for. |
| 22:42:56 | <@JAA> | Also, no need to decompress the file. All WARC software can (or should be able to) handle .warc.gz files. |
| 23:03:07 | | BlueMaxima joins |