HomeSearchPrevious dayNext day

00:04:37Eighty quits [Quit: leaving]
00:04:50<godane2>does anyone know of a way to bypass cloudflare?
00:05:10<godane2>i'm stuck with my manualzz grab cause i can't do it command line anymore
00:05:48<godane2>right now i use linkgopher firefox plugin to grab a list of ids
00:06:17<godane2>then take that list and make a real html page that downloadthemall will be able to use
00:06:30Eighty (Eighty) joins
00:06:51Matthww0 joins
00:07:15daalekz joins
00:07:22Matthww quits [Read error: Connection reset by peer]
00:07:22Matthww0 is now known as Matthww
00:07:33<@JAA>I suppose it should be possible to use a headless browser, but it won't be easy. Otherwise, pass the barrier in a browser, then use the same user agent and cookies in whatever you want to download with, I suppose.
00:10:46Arcorann_ joins
00:11:25<godane2>i tried headless and keeps coming back with cloudflare page
00:11:53<godane2>the only way i can see it working is with a timeout for gui browser was used
00:12:11<@JAA>Yeah, it'd need a delay to wait the 5 seconds for the challenge etc.
00:12:36<godane2>i want the browser to do everything it normally does but timeout then dump the page
00:13:02<godane2>i try doing a delay in headless and it still gives me the cloudflare page
00:14:30<@JAA>Just to be clear, it's about Cloudflare's I'm Under Attack mode aka 'just checking your browser', right?
00:15:14<@JAA>I'm pretty sure that used to work on crocoite (aka chromebot), which uses headless Chromium or Chrome. It's been a while since I checked though.
00:16:02<@JAA>But in principle, I don't see any reason why it shouldn't work. Headless browsers are the real deal minus graphical display. Stuff's still getting partially rendered internally as if it was displayed, just not actually drawn.
00:22:43ragu_ quits [Remote host closed the connection]
00:23:14ragu_ joins
00:25:13Matthww4 joins
00:25:24Matthww quits [Ping timeout: 240 seconds]
00:25:24Matthww4 is now known as Matthww
00:46:23crispyalice2 (crispyalice2) joins
00:47:24<godane2>JAA: yes i think thats what it is
01:18:44janpaul123 (janpaul123) joins
01:52:25ragu_ quits [Remote host closed the connection]
01:53:08ragu_ joins
02:02:47janpaul123 quits [Client Quit]
02:13:26HackMii quits [Remote host closed the connection]
02:40:41ThreeHeadedMonkey quits [Client Quit]
02:43:17<@JAA>I intend to ignore a bunch of videos by default on socialbot's ArchiveBot jobs. Many of them were grabbed hundreds or thousands of times before, and so archiving them yet again is a waste of both our and IA's resources. They also cause disk space crashes due to their size. If anyone disagrees, yell at me in #archivebot by midnight. (More details in the discussion just now over there.)
02:43:42<@JAA>(Just for visibility, discussion should stay there.)
02:53:21ThreeHeadedMonkey (ThreeHeadedMonkey) joins
02:54:54Josephwdye joins
03:15:08Wayward (wayward) joins
03:16:24user777 joins
03:42:00Josephwdye quits [Ping timeout: 244 seconds]
03:51:32ragu_ quits [Remote host closed the connection]
03:52:04ragu_ joins
04:03:49ragu_ quits [Remote host closed the connection]
04:04:27ragu_ joins
04:11:13qw3rty_ joins
04:14:41qw3rty__ quits [Ping timeout: 265 seconds]
04:55:01etnguyen03 quits [Client Quit]
05:06:15DogsRNice quits [Read error: Connection reset by peer]
05:11:03schjits joins
05:25:35xit quits [Quit: Ping timeout (120 seconds)]
05:25:53xit joins
05:36:39sliccricc quits [Remote host closed the connection]
05:40:50schjits quits [Ping timeout: 244 seconds]
05:41:55sliccricc (sliccricc) joins
05:45:01sliccricc quits [Remote host closed the connection]
05:45:11sliccricc (sliccricc) joins
06:36:46jut quits [Ping timeout: 252 seconds]
06:38:28jut (jut) joins
07:06:21godane1 joins
07:06:24ragu_ quits [Read error: Connection reset by peer]
07:08:44godane2 quits [Ping timeout: 240 seconds]
07:30:55Wayward quits [Ping timeout: 265 seconds]
07:34:29sliccricc_ (sliccricc) joins
07:39:06sliccricc quits [Remote host closed the connection]
08:28:46BlueMaxima quits [Client Quit]
08:29:24ThreeHeadedMonkey quits [Ping timeout: 265 seconds]
08:31:03ThreeHeadedMonkey (ThreeHeadedMonkey) joins
08:40:54Wayward (wayward) joins
08:45:03hooway joins
08:58:20godane2 joins
09:00:52godane1 quits [Ping timeout: 252 seconds]
09:21:18godane1 joins
09:23:44godane2 quits [Ping timeout: 240 seconds]
09:39:00<purplebot>Webzdarma edited by Sanqui (+76, /* ArchiveBot jobs */ webzdarma_mwlinkscrape_06, …) just now -- https://www.archiveteam.org/?diff=46261&oldid=46250
09:48:24nerdguy1138 quits [Ping timeout: 240 seconds]
09:51:52godane2 joins
09:54:02godane1 quits [Read error: Connection reset by peer]
10:03:36nerdguy1138 (nerdguy1138) joins
10:21:00rogueCephus18 joins
10:21:56Mateon2 joins
10:22:59rogueCephus1 quits [Ping timeout: 265 seconds]
10:23:55Mateon1 quits [Ping timeout: 252 seconds]
10:23:55Mateon2 is now known as Mateon1
10:32:49daalekz1 joins
10:36:31daalekz quits [Ping timeout: 265 seconds]
10:45:43<Arcorann_>This could be a while, I'm still trying to work out exactly what happened and whether there's even anything to salvage
10:46:04<Arcorann_>Meantime, I'll drop some relevant threads here
10:46:05<Arcorann_>https://forum.novelupdates.com/threads/centinni-behind-the-scenes.109517/
10:48:16HackMii (hacktheplanet) joins
11:36:30HackMii quits [Remote host closed the connection]
11:36:50HackMii (hacktheplanet) joins
13:02:19Mateon1 quits [Ping timeout: 252 seconds]
13:24:52Mateon1 joins
14:28:40Arcorann_ quits [Ping timeout: 252 seconds]
14:34:48daalekz1 quits [Ping timeout: 265 seconds]
15:21:20ragu joins
15:24:06rsn_ quits [Ping timeout: 265 seconds]
15:27:30LeGoupil joins
15:27:36Hackerpcs quits [Client Quit]
15:29:34Hackerpcs (Hackerpcs) joins
15:37:08rsn joins
16:16:04rsn quits [Ping timeout: 240 seconds]
16:24:08Merzius joins
17:11:27rsn joins
17:14:46Merzius quits [Remote host closed the connection]
17:29:37ThreeHeadedMonkey quits [Ping timeout: 252 seconds]
17:31:04Barto quits [Ping timeout: 240 seconds]
17:31:34ThreeHeadedMonkey (ThreeHeadedMonkey) joins
17:35:19Barto (Barto) joins
17:52:04Barto quits [Ping timeout: 240 seconds]
17:58:21Barto (Barto) joins
18:50:34Josephwdye joins
18:54:57isthisthedarkweb quits [Ping timeout: 244 seconds]
19:08:54onetruth joins
19:52:55Ryz quits [Remote host closed the connection]
19:53:00<purplebot>Google edited by Atphoenix (+3977, /* Google Docs Editors */ added …) just now -- https://www.archiveteam.org/?diff=46262&oldid=46259
20:08:24lennier1 quits [Ping timeout: 240 seconds]
20:09:06<avoozl>JAA: if you can think of another (can be larger) forum than the LoL ones, I would love to kick of a download and see how much effort is required to retarget it to a different forum.
20:11:05lennier2 joins
20:14:05lennier2 is now known as lennier1
20:17:10LeGoupil quits [Client Quit]
20:20:05<atphoenix>avoozl: CNET?
20:21:23<avoozl>atphoenix: is that all packed up in warc someplace?
20:21:49<atphoenix>we ran it through AB in Dec
20:22:13<atphoenix>it's not a standard phpBB forum though. What forum software does your code work on?
20:23:10<avoozl>https://archive.org/details/www.cnet.com_forums_202101 is correct?
20:23:43<avoozl>My forum stuff currently needs to be adapted, so I'm looking at different forums to make sure it's adaptable enough
20:26:59<avoozl>Ok download running, thanks for the tip, I'll see if I can adapt they ingest some time this week
20:27:23<atphoenix>there were ~2. One started just at the forums folder, but we learned that some stuff was outside that folder, so another attempt was made at the root, with exclusions to skip obviously non-forum stuff
20:27:24<atphoenix>https://archive.fart.website/archivebot/viewer/job/blaam
20:28:16<avoozl>That looks larger, I'll add it to the queue :)
20:29:21<atphoenix>https://archive.fart.website/archivebot/viewer/job/2xjxk
20:29:26<atphoenix>that too
20:30:15<atphoenix>IIRC, one had outlinks, the other did not
20:30:17<avoozl>That looks seriously big, interesting. Is there a lot of media/images there?
20:30:24<avoozl>Ahh ok
20:30:37<spirit>general forum archiving? i love this! https://www.bokt.nl/forums/ https://www.gaiaonline.com/forum/ https://arstechnica.com/civis/ https://www.dslreports.com/forums/all https://www.eevblog.com/forum/ might be a good set for hating yoursel :D
20:30:55<thuban>viel erfolg, avoozl! this is a really cool project.
20:31:21<avoozl>Thanks
20:31:25<atphoenix>the smaller one started at root, but skipped outlinks and unrelated sections of the site via ignores.
20:32:02<atphoenix>the large one started at the forum folder, had outlinks, but could miss forum stuff that was not in the same folder tree
20:32:51Ryz (Ryz) joins
20:36:51<atphoenix>avoozl looks like you could find other examples here: https://archive.fart.website/archivebot/viewer/?q=forum
20:37:02<atphoenix>not sure how many of those are large
20:37:50<atphoenix>some forums run on Drupal too. One of those is running in AB now, might be in viewer in a few days.
20:38:13<avoozl>Wow that is quite the list, will have a look
21:05:11<@JAA>avoozl: Yes, the one you linked is my complete crawl of the CNET forums that contains only the relevant HTML pages. Probably best for your purposes.
21:05:40crispyalice2 quits [Ping timeout: 244 seconds]
21:06:14<@JAA>The AB jobs always have images and often outlinks, so they naturally blow up massively due to that.
21:21:57janpaul123 (janpaul123) joins
21:29:05janpaul123 quits [Client Quit]
21:42:14janpaul123 (janpaul123) joins
21:55:42sliccricc_ quits [Remote host closed the connection]
21:55:58sliccricc_ (sliccricc) joins
22:02:29sliccricc_ quits [Remote host closed the connection]
22:02:39sliccricc_ (sliccricc) joins
22:05:17crispyalice2 (crispyalice2) joins
22:09:44sliccricc_ quits [Ping timeout: 240 seconds]
22:32:24nertzy leaves [Leaving]
22:33:08nertzy (nertzy) joins
22:34:19Barto quits [Ping timeout: 252 seconds]
22:38:08DogsRNice (Webuser299) joins
22:44:43daalekz joins
22:50:33Josephwdye quits [Ping timeout: 244 seconds]
22:58:53Barto (Barto) joins
23:31:49hooway quits [Client Quit]
23:54:15BlueMaxima joins
23:56:11Arcorann_ joins

HomeSearchPrevious dayNext day