Forum Security Incident & Server Migration — Please Read

[ted]

Hey everyone,

I'm the server guy, who runs the hardware/software here. I only seem to pop in when there are outages or problems, as there was this week.

I wanted to give you all a heads-up about some recent events affecting the forum.

What happened:
The forum server was compromised. Starting Tuesday morning, and we noticed Wednesday. Attackers had gained unauthorized access and had planted malicious files on the system. While we have no evidence that member data was accessed or exfiltrated, we took the situation very seriously.

What we did:
I spent Thursday trying to recover from it, but the files were too buggered. So the compromise actually accelerated a migration we had already been planning on moving the forum to new hardware on a faster, isolated network. Here's a summary of what was done:

• Removed all malicious files and cleaned the server, eventually giving up
• Migrated the forum to a new server with stronger hardware
• Network should be higher speeds; unsure if that will be noticeable.
• Updated software installed for this major version of VBulletin
• Implemented a dual-layer firewall (both at the network VM level and at the operating system level) with strict outbound traffic controls
• Routed all public traffic through Cloudflare, which now provides DDoS protection and bot filtering
• Removed all remote access paths from the internet. Only through the Cloudflare tunnel now.
• Added a Cloudflare security challenge on the admin panel to block automated attacks
• Locked down server-level script execution to prevent future exploitation
• Configured secure email delivery through Mailgun

This should greatly reduce the effectiveness of the inevitable future exploit found in this forum software.

The good news:
The forum is back online, faster, and significantly more secure than before. Most things should be working as expected.

We need your help:
If you notice anything that isn't working correctly or broken links, missing images, login issues, or anything else that seems off, please reply to this thread and let me know. Your reports will help us catch anything we may have missed during the transition.

I do see the broken emojis on this message. I'll work on that a bit today. Not sure how long they've been broken images.

Thank you for your patience and understanding!

— Ted

[Hank Jarrett]

You're the best! Didn't understand a lot of your explanations, but the site looks great and I am CERTAINLY not a computer guy. Without guys like you I am afraid our planes would have been really rare by this time. The links here and on FB are what keeps most of us flying (and FB is NOT my favorite platform!)

Hank J

[ted]

ha! Thanks!

We'll just say I decided to take a different approach to keeping it up and running. Not just trying to patch it whenever I have a rare minute. But a handful of changes that will make it harder for them to do damage if there is a way in.

As sure as there's software, there's someone on the internet who will eventually find some way to break into it.

I hope next time I learn of a break-in that it just means I have to update the software rather than rebuild the whole server.

[skip]

Ted,
This is the world we live in now. Thanks very much for your efforts. we do appreciate them!
Skip Egdorf

[Jim Herpst]

TED, Your efforts and work is as Hank stated very much appreciated! I have always enjoyed this site and would shudder thinking it ever went away. Thank you!!!

[Robert Lees]

Thanks Ted. Bob & you are the unsung stars.

[Jim Herpst]

thanks ted. Bob & you are the unsung stars.

indeed!

[waltermrich]

Agree with everyone , we are thankful for keeping this Forum going , for Taylorcraft owners it is invaluable.

[ted]

thank you, everyone.

I don't have to put much time into helping out here very often. But hearing and knowing you all cherish the site certainly motivates me to get it back online ASAP.

Bob & Robert put in the vast majority of the work around here!

[TCUDustoff]

Ted,

Food for thought. I recently moved my self hosted stack to Cloudflare tunnels versus DDNS due to an ever increasing amount of hits registered by my CrowdSec instance. I ended up disabling access for folks in China, Russia, and proxy / allied countries and nation states; as unfortunate as this may be, the amount of malicious hits I was getting on my network went to almost zero. I hate to gate-keep our community, but I can't imagine there are many Taylorcrafts flying in China (or Russia). With the ease of which you can spin up a nefarious AI agent and put it to work, it may be something that ends up being required from a security POV.

I'm also not super familiar with how this vBulletin software is structured, but if you run behind a reverse proxy I recommend the use of something like Authellia or Authentik for 2FA passkey protection. You can do some fun stuff with wildcard domains and paths to target the admin users only, which is something I've done with anything that has a forward facing admin panel for added security. Assuming the forum software supports OpenID Connect 1.0, or OAuth 2.0, its pretty quick and easy to spin something up within Docker or LXC.

As others have said, again thanks for what you do.

[ted]

TCUDustoff It is through cloudflare tunnel currently.

No geographic filtering currently. I had done that in the past and seemed to always have a few stragglers asking how to get access. (If they were from an untrusted IP space, it would give them a page asking them to send me their network range in an email, and I could whitelist it.) I could add it at a later date.

The problem is that eventually a software bug is always found. And someone somewhere (even on trusted networks) will eventually try that exploit. Usually the attacker, if successful, comes in as the web server UID. And then wget/curl/etc. an exploit for the underlying OS. Among many controls, the web server user/UID is not able to make remote calls to external endpoints. It's not impossible, but it will be significantly more difficult for the attacker to make the jump from finding an exploit in vbulletin to gaining a priv account and taking over the system and doing damage.

And most attackers are not targeting the victim/server, just looking for the low-hanging fruit. So I'm hoping that makes the bad guy(s) give up. (Until the patch for vbulletin is out and applied.)

I don't think 2FA would help since I'm fairly certain they aren't logging in through the exploit-of-the-day.

Computers/tech/security is exhausting. I should have become a tradesman instead. I still think about making the switch.

I appreciate the input. And you're welcome!