PHP, TLS, etc… We appear to be facing a tsunami of vulnerabilities being discovered in record time. Further, the window to act on these vulnerabilities has basically closed to zero; hackers’ tools now automatically fold the very latest vulnerabilities into their botnets within minutes/hours of discovery. I have always been extremely reluctant to expose any of my services - even with the latest and greatest security tools. My choice to stay totally within the confines of tailscale seems vindicated given this week’s news. How are you all taking this?
My server was recently compromised via the latest vulnerability. Fortunately, the damage wasn’t too severe. Since my server must remain publicly accessible, it can’t be hidden behind a VPN—I can only minimize its exposure as much as possible while ensuring my backup and recovery processes are rock-solid. I’ve configured the firewall to allow only essential ports, restricted SSH access to key-based authentication, enabled automatic installation of security updates, and set up daily backups, one of which is synchronized to an off-site location. Some have suggested treating public-facing machines as if they’ve already been compromised by default. I agree with this approach, and it’s exactly what I plan to do next.
It certainly isn’t good to depend on it but security by obscurity is your first and probably most important line of defense. When these vulnerabilities come out they need to know your service exists to exploit it.
The best way to do that is to setup a reverse proxy to reject invalid hostnames. A single port is all you need exposed for multiple web services, and the IP scans won’t find the script kiddies anything valuable. Be careful with lets encrypt, there is a public journal of all issued certs, so make sure to get yourself a wildcard cert so that you aren’t advertising your valid hostnames to the world.
Use the reverse proxy to put basic auth in front of your services (if they can handle that). Basic auth is very simple, no web forms to attack, just a single header. If your php site is vulnerable they won’t ever be able to even attempt to reach it without a valid username/password.
I’ve got a mix of containers and VMs behind that reverse proxy. The containers have network security groups around them limiting network access. The VMs are in a Harvester hypervisor with KubeOVN for software defined networking. That means each VM is in its own subnet with network security groups around the subnet preventing any unnecessary network access.
For machines/containers that need Internet access I have a Squid proxy with an explicit whitelist of allowed URLs so that the machines can’t reach out to unauthorized sites to download their malware or contact command and control nodes.
Unattended updates for the VMs, and FluxCD keeping helm charts up to date hourly for the containers. For my own containers, weekly scheduled container rebuilds.
I do have one public website that I host for a friend (indexed by search engines so the sites name is known to the world). My friend had originally deployed that as a WordPress site, but WordPress/PHP don’t have the best security track record so I was concerned about that. The site was pretty simple so I recreated it as 100% static content, so now the attack surface is just a nginx container with a read-only filesystem.
Ipv6 only hosting still seems as good as an airgap😅
Recently removed the A record from a few semi-public services that got hammered by AI crawlers, works like a charm
I patch as soon as possible and If I were to get pwned id restore a backup and patch again. All my public facing services are sitting in a own, isolated network so an attacker wouldnt get far. If my Hypervisor got compromised id do the same. Id be very fucked if something got into my Proxmox Backup Server tho.
I went the other way this week. Nothing public anymore, and the only thing open on the WAN is the WireGuard port. I lost the easy Jellyfin share links for family and gained sleep. The PHP mess made the call easy.
I only have a VPN endpoint exposed. If I need to access any of my web services I simply VPN in and away I go.
Tailscale is the super easy version of this. Otherwise make sure your containers don’t have root etc. block inter container networking as much as you can. Lots of relatively simple things work unless you’re actually an interesting target.
This
I moved all my files to one share. At the root of that share is a robot_readme.md file that lists all the contents of my share with a note that says read everything you want, you will anyway.
They left a note that said “nah, bruh, you’re good”.
I guess I’m okay.
I still do, but I don’t host anything on 80 or 443. I use a random large port number, and I’ve seen zero illegitimate requests in 20 months.
IF there’s an active exploit AND it affects a container AND you have a container escape route AND you pick my domain you’d need to statistically try thousands of times to even hit the single open NAT port.
I also practice domain/IP obfuscation in addition to high port:
- TLS certs are wildcard certs, so bots can’t get my subdomains from transparency logs.
- My DDNS IP is on a subdomain (random string) not on the base domain.
- My public services are also on random subdomains, CNAME’d individually to the IP subdomain.
- There is no wildcard CNAME that the bots can query to get the IP.
This eliminates scanning almost completely. They can’t find an IP to go with my base domain, first of all. And even if a bot is scanning the IPv4 address space and all the ports and it finds the reverse proxy it can’t get past it without knowing a FQDN. I still get the occasional hit from such bots but it’s very rare.
Also, all my public services are secured with mTLS (preferably) or at the very least a long key in a custom HTTP header (both enforced at the proxy).
You should also use a default self-signed cert for something generic like “localhost” on your reverse proxy so random IP scan don’t see any domain when they hit your IP, but legitimate queries coming through the domain name still see the right cert.
In my case I have defined a fallback in Caddy so any request that doesn’t match one of the explicitly handled domains will be an error.
I use a
route {}block withhandle {}blocks inside it, and the last handle has anerrordirective in it.https://*.example.com { route { @foo host "foo.example.com" handle @foo { reverse_proxy http://whatever:12334/ } ... more proxy hosts ... handle { error 400 } } }The
routeblock is essential here because handlers inside a route block are processed strictly in the order they are defined. Outside route blocks that is not true, caddy will try to optimize and can execute directives in any order it sees fit.
Thank you for sharing this, vey insightful
How do u use a port other than 443 for https. AFAICT all web browsers are pinned to it (for ssl checks I think) and I could never get an actual https connection through an alt port. Maybe I didn’t understand it properly.
Edit: I think actually it’s the auto cert provisioning workflow. The CA I’m signing against hardcodes port 443 :/
Edit, edit: this was the issue https://caddy.community/t/non-standard-ports-for-https/6916/4
The API based challenges are better since you don’t even need to make the site externally accessible to get certificates.
Let’s encrypt and similar services support those and for popular registrars (aws, porkbun, etc,) here are existing plugins for caddy, swag (nginx proxy with built in cert support) and a few others.
Yeah, if you use something like Nginx Proxy Manager you can’t use the automatic flow, you need to use the token based challenge.
Boo, no idea why acme forces these specific ports. Swapping them seems like a pretty basic obfuscation technique that could help a lot to secure any services.
Edit: actually seems like this might be possible with the dns challenge type. I’m gonna test this out later https://github.com/caddy-dns/porkbun
It is, that’s what I had meant by token based challenge lol. You need a DNS API key for your manager to do it directly, it flags a DNS record with a token to validate ownership. I was just calling it out from memory, sorry!
Much appreciated stranger. Hopefully now I can stop exposing the two most attacked ports in existence from my LAN XD.
This was the final push that convinced me to rebuild my server from scratch! I started doing so 3 weeks ago :D
I was running my containers on a single good old Debian, that I replaced with Proxmox. I set up several VM that don’t share anything between them :
- One for all the services I don’t really care about, like game servers and utilities with no personal data.
- A second one for services in need of a GPU passthrough (Immich and Jellyfin) and partial access to some of my data (photos and movies in this case)
- And a third one that gets access to all my personal data, only hosting a container for WebDAV access and a web file browser. I might hide these two services behind a VPN in the future if I feel like doing so, for now it is still to practical to have them freely available…
- A last VM is running network related app like my reverse proxy.
I should have done this years ago, but I’m still learning and learning costs time :p
Next I want to hide all of Internet facing services behind an identity provider/application proxy (I don’t really know if it is the right way to name it ?) like Authentic to totally block access to any content from unauthorized visitors.
I guess I’ll be mostly fine once this is done ^^
Your separation between VMs is exactly what I want to do. But given how I’m still rocking a single truenas machine running all my containers, I’ve got a quite a bit of setup to do… Like you said, rebuilding everything is a slog.
Hasn’t changed much. I have accepted the risk and, for now, the desire to have publicly accessible stuff outweights my lack of desire to deal with the realities of a VPN. I mean, I want to share stuff with others, and I want to be hosting it myself because I have grown to distrust someone else’s service… so what else can I do but risk it?
Besides: I expect tailscale or the underlying VPN services to already be a target. I would be surprised if they don’t find anything. What then?
If they find anything we fix it, update our services, and move on.
The chances of being hit by a zero-day as a random self-hoster are very small, but anyway if you keep backups and can restore a clean state (and patch your system) you should be good.
The goal is not to have perfect security, it’s to mitigate the risks and to handle fallout correctly.
I moved to closing everything down and wireguarding in to home 5 years ago and I’m glad I did. I only expose game server these days which have a different vuln profile.
If/when I open up for Jellyfin/etc, I’ll probably run them on a different box to my NAS in a DMZ with strong firewall rules. And use a VPS for anything else that doesn’t require NAS storage to reduce blast radius.
Can we go back to late 90s internet yet?
I completely folded all services to the “inside”.
exposure is unchanged. automated patching and upgrades are working. rollbacks, etc are working. minimal attack surface exposed and properly blocking traffic by geography. rotation of keys/passwords as well as short lived rotated TLS certs.
not one of those things alone is a solution, but together can make a solid dent in securing your infrastructure.
My threat model is not changed: I want to serve something and I want to not be busy with "oh suddenly there’s
atwo PHP vulnerabilityies. My enemy is the world, but that is not really unchanged from before in a world where there exists capitalism and rent seekers want to collect from us even for breathing. I’m not intended to be unpaid tech support. So I just serve the things with reasonable scopes (under wireguard, or in a borrowed VPS, etc) and let it go. If an automated system manages to find a hole, computers are so fast that it’s not going to make a difference if I’m sitting in front of the chair vs enjoying some modicum of social life visiting a friend three cities away.I have my ops agent constantly checking SBOMs against current CVEs and if something triggers, I get a report with severity and repair options researched. I can then decide how I want it to proceed. I’m much less concerned about internal only services. SBOMs are refreshed after updates for future checks.
