Having trouble downloading a forum thread with wget and httrack

submitted by
edited

I’m a big Civ IV fan, and I’d like to download all of the Sample Game threads on this Civfanatics site: https://forums.civfanatics.com/threads/sample-game-directory.258697/

I’d like to have all of the posts and images from each sample game, each of which is a forum thread with multiple pages, posts, and images hosted from elsewhere.

Here’s the wget function I tried (which I mostly stole from here https://letswp.justifiedgrid.com/download-discourse-forum-wget/) :
wget –mirror –page-requisites –convert-links –adjust-extension –compression=auto –reject-regex “/search” –no-if-modified-since –no-check-certificate –execute robots=off –random-wait –wait=1 –user-agent="Googlebot/2.1 (+http://www.google.com/bot.html)” –no-cookies -r https://forums.civfanatics.com/threads/sample-game-directory.258697/

It gave me an index file that was way too small, and while it did take me to the index page, seemingly locally on my system, clicking on any of the forum threads took me to the actual website instead of a local mirror. Also I don’t think any of the images (which are linked from elsewhere) downloaded.

I’m mostly just interested in a subset of the sample games, so I’d be fine downloading each sample game forum post separately, but I haven’t been able to do that either.

Any help would be appreciated!

Edit: I also just tried httrack with no results either:
httrack “https://forums.civfanatics.com/threads/all-leaders-challenge-game-strategy-session-montezuma.168755/” “+*https://forums.civfanatics.com/threads/all-leaders-challenge-game-strategy-session-montezuma.168755/*” -v

6
14

Log in to comment

Hot Top New Old
edited

You’re trying to run a scraper, and unfortunately a lot of AI companies are doing the same thing on such a big scale that it creates a DOS attack. So lots of sites now have anti-scraping measures. I clicked the forum link and saw a brief interstitial that looked like bot protection.

I’m unfamiliar with httrack but wget fails pretty often by the site just rejecting the user agent. Try “wget -Dfoo [url]” to save the response headers in the file “foo”, if I remember it right. That will let you check if there is an error code. You could also examine the too-small html index that you got, to see if it has error messages inside.

Sometimes curl works when wget fails. For both of them, there are CLI options to set the user agent to something different.

Getting images from forums often requires you to have a login cookie in your client. It’s simplest to log into the site and then paste the cookie into your scraping program or script.

The next thing after wget/curl would be to write a scraping script (say with python urrlib) that can analyze the html a little as it goes. The thing after that would be script an actual browser, with puppeteer or selenium etc.

Regarding the links pointing to your local disk, that’s probably because they are relative links, like “href=./foobar.html”. You can make them point to the remote server by inserting an HTML BASE tag into the top of the file, like <BASE HREF="https://whateveritwas.com/forum"> or whatever. If it’s just for one or two pages you can do that manually. Otherwise, modify your scraping script to insert it, before or after saving the output.

 reply
11
OP

Thanks for the detailed response! I don’t understand a lot of this, but I’ll try to work my way through it, and I’ll try curl instead of wget!

 reply
2
edited

I tried wget and got a small index file that looks like a bot challenge, maybe Cloudflare Turnstile. I don’t know of a simple automated workaround. But, Turnstile gives you a rewritten url that then goes back to the forum page and sets a cookie, iirc. So if you can visit the page with a real browser, then save the cookie and transfer it to wget (there’s some option to set an arbitrary header) that’s one thing to try.

There’s another hack used by fanfiction.net readers, where if you want to save a multi-page story, you can manually visit each page with a browser, i.e. click “next” again and again to load all the pages. Up to a few dozen such clicks isn’t so bad. Fanficfare (fanfic downloading program, sometimes abbreviated FFF, https://github.com/JimmXinu/FanFicFare ) then has an option to retrieve the pages from your on-disk browser cache instead of trying to get them from the remote server. That’s another approach you can try, either with fanficfare or your own scripts.

The site you’re looking at uses xenforo which is a very popular forum server program. The actual layout of xenforo sites varies, but fanficfare probably already recognizes something similar, so try using one of those interfaces. I think spacebattles.net (another fic site) uses xenforo and FFF supports it, so it might be a good start. You will have to modify FFF to recognize civfanatics instead of spacebattles. It will help to know or pick up some Python, but you shouldn’t have to become an expert.

Added: if you really want to automate your scrape, you will have to orchestrate a browser as mentioned earlier. IDK if there is code around to already do it. If you can program, it’s not terribly hard to use Puppeteer or Selenium, but it will take some farting around to deal with the site layout and anti-bot stuff. It’s not guaranteed to work right off the bat, but with enough determination you can do it, especially if your scrape volume is low and you can run it slowly. I’m not deeply involved in this stuff (like you, I just occasionally want to download something for personal use) but there are tons of webpages and articles by people (who I’d mostly consider evil) who do it at scale.

 reply
6
OP

Thanks so much - FFF looks like it might be exactly what I’m looking for, so I’ll see if I can get it to work tomorrow!

 reply
4
edited

FFF won’t work on that site right out of the box, because of the bot challenge. You’ll need a workaround. FFF has the same problem with fanfiction.net (FFN) which is one of the biggest fanfic sites. So there are a number of FFF github issues and doc entries related to FFN and looking at those might help. I do know that FFN is scrapable using browser orchestration. https://github.com/FicHub/fichub.net may have some code for that, but getting your own instance running will be quite a bit more headache than just running fanficfare.

 reply
4

Many websites are detecting tools like curl, wget and httrack and either rate limit or block them. Sometimes one can get around it when using more detailed configurations that mimick like a human using a real browser, but that is basically tricking them. The reason why they are blocking such automated tools is multifolded (if that is even a word): a) downloading pages and data in short amount of time creates huge stress for the servers, meaning its like an attack that halts their systems, b) that in turn also leads to higher server costs for them, c) they don’t get visitors for the advertisements and engagement, as the tools download autonomously, d) is also sometimes seen as theft from Ai companies, so they do try to block automated mass downloads.

Besides these tools there are other ways, more sophisticated and requires more work to mimick real browsers. In some cases even using real browser engines, but changed the browser in a way to make automated work possible, such as https://www.selenium.dev/ . Especially needed when JavaScript and other dynamic content is involved. Maybe even programming Python with custom solutions. There is also a way to directly program in Typescript using https://deno.com/ . But these solutions are a bit advanced and only recommended if you want go deeper.

Soo, having all said that, this might not help directly solving your issue, but I hope its still helpful in why its difficult. Especially nowadays, because companies have better detection and blocking, because of Ai…

 reply
2
Insert image