BNA Up / Down ... This is what's going on?

[AUbicycles]

My heart sinks every time I see the downtime and the aim is 100% uptime but want to share the two things affecting it.

Firstly.. bots / crawlers as has been mentioned. BNA has hundreds of thousands of pages and when bots come, they should slowly read the pages however there are a lot of bots and a lot of them load as much content as fast as possible. The result is that the server overloads and eventually goes offline and can only be manually restarted.


Secondly, with the relaunch of the main site - there are a few technical changes and although it perfectly accommodates genuine visitors, when overloaded by the bots, it can impact the site and sometimes with temporary or permanent downtime.



Again, thanks for your patience.
Christopher

[MichaelB]

All good. Been pretty damn good as far as I am concerned, so major thumbs up from me

[10speedsemiracer]

Also a thumbs up from me, we love your work Chris.

[g-boaf]

Looks like the forums are showing blank occasionally today as well, but randomly.

[AUbicycles]

Right now - there is an ongoing Bot attack and all of the usual methods to block them are not working.

A big technical change is on the way (I have been preparing it for the last 3 weeks) and I will try and fast-track this which will then allow me to add on another layer of protection against the bot attacks.

[DavidS]

Good Luck AUBikes, we all appreciate the forums.

DS

[AUbicycles]

Thanks... a frustrating approach, everytime I block an attach, the IP changes and I have to then find the new pattern and block it. Unfortunately they are automating and I have to react manually... while the forum and site responsiveness remains wobbly.

[g-boaf]

Thanks... a frustrating approach, everytime I block an attach, the IP changes and I have to then find the new pattern and block it. Unfortunately they are automating and I have to react manually... while the forum and site responsiveness remains wobbly.

Frustrating! I figured it must have been bots.

[10speedsemiracer]

I'm very much out of date in this regard, but was wondering if instead of blocking the bots, the Server/Site could mandate a slower crawl speed for any and all Bots? So like only allowing a certain number of hits per second? Although I'm fairly sure GoogleBot would probably just ignore this somehow..
I can't remember the specifics but was something I'd come across a while ago.

Please excuse my suggestion if you've already done this or have better tools at your disposal. Was just thinking..

[AUbicycles]

Thanks for this.
Apparent 'good' bots like microsoft/bing ignore crawl speed.
And the malicious ones ramp it up and push a server to its limits and easily beyond. When the server falls over, they use a new IP address. The current one is pretending to be both Googlebot and Microsoftbot (with the same IP).

[AUbicycles]

A bit of a marathon session - following about weeks (actually months) of testing there have been a series of updates. All of which will hopefully be invisible and there will be no effect for visitors on the main site and and guests and members in the forum.

For the techies -
There was a php upgrade. Far from pressing a button - a complete hosting environment needed to be built from scratch and involves load-balancing and various automation. Testing this was the biggest part and then all of the compatibility for the software and making fixes.

For the forum - while I had done a a few test-runs of the forum software upgrade but was hoping to move this back. Unfortunately compatibility made it necessary to upgrade. A massive challenge is the 1.5GB database... which also needed an update and turned out to be pretty tricky. Backups are there incase there are any serious issues, but some spot checks suggest that everything is as it should be.

An finally, in the last one tackles the malicious bot attacks that ramped up over the last few days and kept on pushing the server offline and giving me massive headaches. A service provider called Digital Ocean in the US is hosting someone performing this malicious activity over numerous IPs so it has been very tricky to stop. Beyond catching individual IPs... a new limiting function has been set to prevent misuse. Over the coming week I will optimise but have already an immediate drop.


If you spot anything, please comment here... or comment on facebook. The chances are that the wonderful moderators on this forum will also drop me a quick message on my phone and I will get onto any issues as soon as I am able to.

[MichaelB]

Sounds like a LOT of hard bloody work !!

Thought computers were meant to make it easy ….

[Tim]

I think those rotten Bots are at it again. BNA forum either down or very slow at my end.
This site's as near as I get to social media. I really miss it when it's down and very much appreciate the effort Chris puts in to maintaining it. Thanks Chris.

[brumby33]

Yes something's going on however after a bit of persistant clicking of the refresh button...the forum will show up.

About 5 times today I kept on getting an error message when I clicked on the forum tab, it also happened on my mobile.

SQL ERROR [ mysqli ]

Too many connections [1040]

An sql error occurred while fetching this page. Please contact an administrator if this problem persists.

Cheers

brumby33

[antigee]

as above

[DavidS]

I seem t be mainly getting the SQL error when I try and post. Got an example on the environment thread a few minutes ago but my post did go up. Seems to work better when posting using the full editor.

Good Luck AUBikes, something is going on.

DS

Looks like my avatar is misbehaving again too

[AUbicycles]

Testing, travelling so a bit hard to identify what is going on.

Feel free to post any error messages or email them. If it persists, allow another day until I am able to investigate.

[Bunged Knee]

Had the same problem yesterday from my phone and took a screenshot of it?

[AUbicycles]

Thankyou, if the problem appears to persist (and did not resolve itself) let me know. I will check it regardless to see if I can see what happened.

[brumby33]

It's been ok today AU Bicycles......so cross fingers and hope for the best that those wascally wodents don't come back!!

[Tim]

Working well today thanks Chris.
Been on a forum binge this arvo.
Making up for lost time.
Keep up the good work.

[AUbicycles]

Thanks for the updates, will see I can identify the fault. At this stage the typical bots and brute force attacks should be able to be ruled out because of the current measures in place.

[AUbicycles]

This time the culprit is Huawei. I have put in more blocks on them and also reset the cookies (requires all members to log in again).

[foo on patrol]

You have to wonder why, they target forums like this? Well done.

Foo