Hi, I was looking at private CAs since I don’t want to pay for a domain to use in my homelab.
What is everyone using for their private CA? I’ve been looking at plain OpenSSL with some automation scripts but would like more ideas. Also, if you have multiple reverse-proxy instances, how do you distribute domain-specific signed certificates to them? I’m not planning to use a wildcard, and would like to rotate certificates often.
Thanks!
Edit: thank you for everyone who commented! I would like to say that I recognise the technical difficulty in getting such a setup working compared to a simple certbot setup to Let’s Encrypt, but it’s a personal choice that I have made.
My experience is it’s really a lot of work and with the prevalence of letsencrypt, there is not a lot of automated setups for this use case (at least that I have been able to find). It is kind of a pain in the ass to run your own CA, especially if you plan to not use wildcard and to rotate certs often. If you use tailscale, they offer https certs with a subdomain given to you:
[server-name].[tailnet-name].ts.net
That’s honestly what I’m moving towards.
From where I sit, not a lot of reason to do it. It’s a bigger pain to deal with your own CA than to just use LE.
I realise that it is more work than using Let’s Encrypt, but this is a personal choice I’m making to not use a public domain for my internal network. I also do not like the idea of tailscale having access to my domain names, but then again, I’m likely not going to use Tailscale at all.
No judgement here. I think it’s a worthy goal just not one I am particularly interested in at this point. Maybe if the automation was a bit easier and the mobile device management was easier I might join you.
I’d just buy a single domain, it’s like £5 a year and use a letsencrypt wildcard and have it auto renew via DNS challenges. Very easy. You can do what you’re doing with letsencrypt, but you’ll have to set up HTTP challenges for each sub domain, or DNS challenges for each sub domain. Obviously doable, but more work.
Doing it without letsencrypt and just doing it privately? I dunno if I’d bother with that, firstly you’ll have to go through the hassle of making sure any browser and computer that connects to it has the root cert of the private CA, or you’ll get self signed errors, which is a faff. I’d honestly just pay the £5 or so a year, you’ll spend more time (and time is ultimately money) doing it without it.
As someone who has done both I couldn’t agree more.
I run Home Assistant and could just pay their subscription to get external access, which also supports development. But I’m a cheap Yorkshireman so I went with DuckDNS, Let’s Encrypt, Nginx and all that jazz for a long time. I’m just trying to hook my home up to Google so I can talk to it, I don’t wanna pay money!
I had so many bloody glitches! I’d have to sign things in on a weekly basis, Google lost access all the damn time and it was just a nuisance.
I have a rule for hobbies, I do it cheap for at least a few months to see if I lose interest. If I get a bit obsessive, it’s worth spending money on the hobby.
So after a year or two of fucking around with Nginx and DuckDNS I found the cloudflare plugin, which worked for free for a while. It was night and day, everything remained connected!
It ultimately bugged out one day and I decided to just bite the bullet and buy a domain. It’s a hobby I’ve invested enough time in that a few quid for a solution that just works is worth it.
I think I dropped £35 for 5 years (I forget, it might be a decade) of owning my own .com domain name, which cloudflare manages for me.
I now have to reconnect my Google home to my home assistant once every couple of months instead of every week.
I haven’t missed the money, and I certainly haven’t missed the fucking tinkering to get it to work.
Thanks, I recognise the challenge in getting this working rather that simple relying on Let’s Encrypt. A little bit of money is not the problem here though; I have made a personal choice to use a private, reserved domain for my internal network and simply do not want to use a public domain for it.
You don’t have to make the public domain, well, public. You can not hook up any DNS records for it, so externally it won’t resolve anywhere and just use internal DNS.
Of course. I just want to take the learning experience from it as well has using a reserved domain instead of a public domain I’d purchase
I’m using step-ca. Its running on dedicated SBC. ACME certs created for each service renewing automatically daily. Honestly this setup wouldn’t be worth it if it wasn’t for daily cert rotation. I’m not using wildcard certs with own CA as it’s bad practice and defeats the purpose. There are bunch of different ACME renewal scripts/services. K8s cert manager handling kubernetes services automatically. Opensense has ACME cert plugin, nginx proxy manager is using external cert managed by script. I’m validating certs with DNS using TSIG. Step-ca have several integrations with different DNS services. I chose TSIG because it’s universal. There is pi-hole integration if you using that. Buying valid domain is not needed as long as you have internal DNS. You need to Install root Ca on every machine that will be connecting to services. If you have many VM’s configuration management is the way to go.
Thank you, and yes, I agree. Frequent certificate cycling and revocations is one of best parts of having a private CA, along with particular certificates for particular domains. With that said: Could you tell me more about TSIG? This is the first time I have come across it, and 2 seconds of reading tells me that it is used to validate automated updates and zone transfers for the DNS server, and also to validate domains? I didn’t think of this avenue before, but it seems very interesting to implement.
And yes, I will have an internal DNS, and I will be inserting the Root CA certificate on all of my client devices.
If you want to run your own pki with self-signed certificate in your homelab I really encourage you to read through this tutorial. There is a lot to process and read and it will take you some time to set everything up and understand every terminology but after that:
- Own self-signed certificate with SAN wildcards (https://*.home.lab)
- Certificate chain of trust
- CSR with your own configuration
- CRL and certificate revocation
- X509 extensions
After everything is in place, you can write your own script that revoks, write and generates your certificate, but that is another story !
Put everything behind your reverse proxy of choice (traefik in my case) and serve all your docker services with your own self-signed wildcard certificates ! It’s complex but if you have spare time and are willing to learn something new, it’s worth the effort !
Keep in mind to never expose such certificates on the wild wild west ! Keep those certificate in a closed homelab you access through a secure tunnel on your LAN !
edit
Always take notes, to keep track of what you did and how you solved some issues and always make some visuals to have a better understanding on how things work !
never expose such certificates on the wild wild west ! Keep those certificate in a closed homelab you access through a secure tunnel on your LAN !
I’m curious, what’s the reason?
In many architectures in which certificates are used, a client with a valid certificate is a trusted client, so a certificate falling into the wrong hands is problematic.
Thank you. Could you explain a bit more about your setup and the aspects I should be looking at? Specifically:
- Certificate chain of trust: I assume you’re talking about PKI infrastructure and using root CAs + Derivative CAs? If yes, then I must note that I’m not planning to run derivative CAs because it’s just for my lab and I don’t need that much of infrastructure.
- I have yet to figure out the CRL part with OpenSSL, I’ll have to read more about it. Thanks.
- I do not know what X.509 extensions are and why I need them. Could you tell me more?
- I’m also considering client certificates as an alternative to SSO, am I right in considering them this way? I will also have to think about what I could do what clients without a client certificate or my root CA’s certificate in their certificate store (maybe run another instance which doesn’t need all of the encryption and setup I’m doing and somehow redirect such clients there whilst logging their traffic?).
Thanks for the mention, I was looking at a script to automate certificate generation and revocation too.
Since we’re talking about reverse-proxies, I’ll mention that I plan to run an instance of HAProxy per podman pod so that I terminate my encrypted traffic inside the pod and exclusively route unencrypted traffic through local host inside the pod. I’m doing this because I do not want to see any unencrypted traffic in my network. Of course, this is some more overhead but I think this is doable. I got this idea from another post I made a while back. Of course, that means that every pod on my network (hosting an HAProxy instance) will be given a distinct subdomain, and I will be producing certificates for specific subdomains, instead of using a wildcard.
Thanks, I’ll be sure to document my progress as I go.
Use something like no-ip, you can get a domain for free and renewing it every 30 days with a few clicks is much easier then managing a CA.
The only downside is the TLD but if you don’t care to much about how your domain name looks it really is the best option.
I use no-ip with letsencrypt, the LE bot does the certificate stuff for me, I use a single domain with different ports for each service and no-ip sends an email every 30 days to reconfirm the domain. Simple and easy.
I would not like to use a public domain for my internal traffic, even if the traffic is not routed outside. This is more of a personal choice than me looking for the easiest technical solution. I do own a domain, but I’ll keep that specifically for elements facing the internet.
Thanks!
Written in go, very small and portable: https://github.com/FiloSottile/mkcert. There’s also step-ca, bigger and uses ACME to deploy certificates, never used it tho.
Just be awake of the risks involved with running your own CA.
Thanks, could you tell me why one would run this over plain OpenSSL with automation? Also, what risks would I run running a private CA? I’d love to know!
could you tell me why one would run this over plain OpenSSL with automation?
Those projects essentially are the automation…
what risks would I run running a private CA? I’d love to know!
More or less you’re adding a root certificate to your systems that will effectively accept any certificate issues with your CA’s key. If your PK gets stolen somehow and you don’t notice it, someone might be issuing certificates that are valid for those machines. Also real CA’s also have ways to revoke certificates that are checked by browsers (OCSP and CRLs), they may employ other techniques such as cross signing and chains of trust. All those make it so a compromised certificate is revoked and not trusted by anyone after the fact.
I do realise the security problem in keeping the private key safe. I plan to use a VM with encrypted storage underneath. Do you think that’s OK for a homelab, or should I invest time into integrating HSM modules from Nitrokey?
Why are you pushing for your own CA in the first place?
I would not like to use a public domain for my internal network. By extension, I do not want any public CA to know the domains and subdomains I use in my lab and home network
I don’t at the moment, because I don’t have a need for it, but I did for a while run a PoC with Step CA, and that seems like the easiest way to get up and running, even if its features are overkill for a home lab.
Step CA is really nice if you want to learn more about how a real CA works. Had some fun playing with it but yeah it’s a bit overkill for home lab xD.
You can achieve the same result with openssl with less complexity !
Thanks, I have been recommended Step-CA a few times under this post, I’ll take a look!
I’m not using a private CA for my internal services, just plain self-signed certs. But if I had to, I would probably go as simple as possible in the first time: generate the CA cert using ansible, use ansible to automate signing of all my certs by the CA cert. The
openssl_*modules make this easy enough. This is not very different from my current self-signed setup, the benefit is that I’d only have to trust a single CA certificate/bypass a single certificate warning, instead of getting a warning for every single certificate/domain.If I wanted to rotate certificates frequently, I’d look into setting up an ACME server like [1], and point
mod_mdorcertbotto it, instead of the default letsencrypt endpoint.This still does not solve the problem of how to get your clients to trust your private CA. There are dozens of different mechanisms to get a certificate into the trust store. On Linux machines this is easy enough (add the CA cert to
/usr/local/share/ca-certificates/*.crt, runupdate-ca-certificates), but other operating systems use different methods (ever tried adding a custom CA cert on Android? it’s painful. Do other OS even allow it?). Then some apps (Web browsers for example) use their own CA cert store, which is different from the OS… What about clients you don’t have admin access to? etc.So for simplicity’s sake, if I really wanted valid certs for my internal services, I’d use subdomains of an actual, purchased (more like renting…) domain name (e.g.
service-name.internal.example.org), and get the certs from Let’s Encrypt (using DNS challenge, or HTTP challenge on a public-facing server and sync the certificates to the actual servers that needs them). It’s not ideal, but still better than the certificate racket system we had before Let’s Encrypt.Finally got around to replying to the comments I got haha!
Thanks for the explanation. I’m curious why you’re not running your own CA since that seems to be a more seamless process than having to deal with ugly SSL errors for every website, every time you rotate the certificate.
I’m wondering about different the process is between running an ACME server and another daemon/process like
certbotto pull certificates from it, vs writing an ansible playbook/simple shell script to automate the rotation of server certificates.About my clients: I am likely never going to purchase Apple products since I recognise how much they lock down their device. Unfortunately, there are not that many android devices in the US with custom ROM support. With that said, I do plan to root all of my Android devices when KernelSU matures (in about a year, I think) - I’m currently reading up on how to insert a root and client certificate into Android’s certificate store, but I think it’s definitely possible. Other than that, I might have a throwaway Windows VM sometimes, which is doable, alongside a Void linux box with a Debian chroot. All in all, fairly doable, just a bit of work to automate.
Thanks!
I’m curious why you’re not running your own CA since that seems to be a more seamless process than having to deal with ugly SSL errors for every website
It’s not, it’s another service to deploy, maintain, monitor, backup and troubleshoot. The ugly SSL warning only appears once, I check the certificate fingerprint and bypass the warning, from there it’s smooth sailing. The certificate is pinned, so if it ever changes I would get a new warning and would know something shady is going on.
every time you rotate the certificate.
I don’t really rotate these certs, they have a validity of several years.
I’m wondering about different the process is between running an ACME server and another daemon/process like certbot to pull certificates from it, vs writing an ansible playbook/simple shell script to automate the rotation of server certificates.
- Generating self-signed certs is ~40 lines of clean ansible [1], 2 lines of apache config, and one click to get through the self-signed cert warning, once.
- Obtaining Let’s Encrypt certs is 2 lines of apache config with
mod_mdand the HTTP-01 challenge. But it requires a domain name in the public DNS, and port forwarding. - Obtaining certs from a custom ACME CA is 3 lines of apache config (the extra line is to change the ACME endpoint) and a 100k LOC ACME server daemon running somewhere with its own bugs, documentation, deployment and upgrade management tooling, config quirks… and you still have to manage certs for this service. It may be worth it if you have a lot of clients who don’t want to see the self-signed cert warning and/or worry about their private keys being compromised and thus needing to rotate the certs frequently (you still need to protect the CA key…)
likely never going to purchase Apple products since I recognise how much they lock down their device
hear hear
there are not that many android devices in the US with custom ROM support. With that said, I do plan to root all of my Android devices when KernelSU mature
I bought a cheap refurbished Samsung, installed LineageOS on it (Europe, but I don’t see why it wouldn’t work in the US?), without root - I don’t really need root, it’s a security liability, and I think the last time I tried Magisk it didn’t work. The only downside is that I have to manually tap
Updatefor F-Droid updates to run (fully unattended requires root).I’m currently reading up on how to insert a root and client certificate into Android’s certificate store, but I think it’s definitely possible.
I did it on that LineageOS phone, using
adb push, can’t remember how exactly (did it require root? I don’t know). It works but you get a permanent warning in your notifications telling you that The network might be monitored or something. But some apps would still ignore it.Europe
Samsung devices in the US have their bootloaders locked, regardless of whether you bought it from a carrier or not. I’ll be looking at other devices, and even then, custom ROM support has all but stopped for everything but the pixel. Living in Europe is great for this, for you have the FP5 available.
But some apps would still ignore it.
Apps with their own certificate store like Firefox? Yeah, I’m thinking about how I can deal with that. Is there a FOSS Android MDM solution that I can use?
I run a crude automation on top of OpenSSL CA. It checks for certain labels attached to kubernetes services. Based on that it creates kubernetes secrets containing the generated certificates.
I would be very interested in your setup. Could you give me a high-level overview of how your setup works?
I’d also like a link to your automation/configuration if you would. Thanks!
I just uploaded to Github: https://github.com/akashrawal/nsd4k
I only made it for myself, so expect very rough edges in there.
Thank you!
I run easy-rsa on a linux box. Just manually generate CSR’s and sign them via SSH.
And simply trust the CA cert in windows, linux and whatever extra places (normally firefox cert store).
Post the crl.pem to /var/www/html/ and let NGINX use that.
For most things public like plex or whatever i just use letsencrypt. Easy-rsa is really just for internal stuff like my NAS, VPN etc.
Just manually generate CSR’s and sign them via SSH.
Could you tell me what you mean by “signing with SSH”?
Copy them to the box. Sign them. Copy the cert file off the box back to the requester.
Self-host your own ACME server. Then you can use certbot pointed there.
These instructions are old so not sure if newer/better ways, https://blog.sean-wright.com/self-host-acme-server/
Thank you, I was looking to host Step-CA, whilst OpenSSL is another option. I’m also planning to combine it with a vault for secrets like Conjur and encrypt the volumes underneath. I want to reach the best security posture possible in this kind of setup
I run a private CA for client SSL.
For traditional server SSL I just use let’s encrypt, although I already have the domain (less than $10 a year) for my public facing stuff, and just use a subdomain of that one for my homelab.I have a container with openssl for the private CA and generating user certs as well as renewing the let’s encrypt ones. I just use openssl without anything fancy.
The output folder is only mounted rw in that one container
I only ever mount the subfolders in read-only in other containers that need those certs.
All these containers are running on the same server so I don’t even have to copy anything around, the containers don’t even need connectivity between them, it’s just mounted where needed.I’d be very interested to hear your reasoning behind using a private CA for clients but using Let’s encrypt for servers. Thanks for the explanation on the OpenSSL setup!
I’m just doing mutual TLS to authenticate clients which I use the pricate CA for.
I could use the orivate CA for the server instead of lets encrypt and trust that on devices, but letsencrypt is easy enough and useful for other things that I open publicly. mTLS avoids needing a vpn for more sensitive servicesDid you design your lab infrastructure to operate in a zero trust security framework? I’m very interested; I never really grasped mTLS, but I’m going to try and read more about it. Thanks!
Edit: Alright, I understand the high-level concept of mTLS. Just need to check the implementation details with my reverse-proxy
Not really, although now that I have certs for those anyway, maybe I should.
More like I’m using some services on the go that I want to always work, whether I’m on the LAN or on the go.
Opening home automation or 3d printers to the Internet is unwise to say the least.
mTLS in the reverse proxy for those allows me to have more security without having to establish a VPN first.Oh, that’s a great idea! Indeed, using certificates to identify yourself would work quite well in such a scenario. Whilst I would always use a VPN server, this has given me something new to think about! Thanks!
What’s nice is it provides a similar level of protection to using a VPN with PKI, but just for that specific subdomain. While a VPN would be have to be connected manually before use (or all the time), this is built-in.
The odds of someone breaking through the mTLS and breaking through that application’s security at the same time are much smaller than either separately.
If you don’t have a valid cert, you’re dropped by the reverse proxy before anything even gets passed to the server behind it.I’m a big fan of it.
I’ll take a look, thanks!
I do have Vault PKI set up but I don’t use it that much. It’s only if I want to do mTLS with something.
For almost all of my actual services, I use a wildcard cert that covers something like *.int.example.com. I use acme.sh to create and renew the cert then have a python script that copies it to any vms or services that need it
Why bother with certificate rotation in a homelab environment? You could issue certificates for three years and just call it a day.
Personally, I have experience with Microsoft Certificate Services, which works pretty well out of the box and is also quite well supported. But you need a Windows Server for it.
It’s just a personal feeling really, I feel better having them being rotated. Of course, when it comes to rotating client certificates, I might have to think long and hard about the pain I’m about to inflict on myself.
I’ll take a look at MCS, although I’d prefer not having to trust Microsoft on things critical to my lab
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters CA (SSL) Certificate Authority DNS Domain Name Service/System HTTP Hypertext Transfer Protocol, the Web NAS Network-Attached Storage SAN Storage Area Network SBC Single-Board Computer SSH Secure Shell for remote terminal access SSL Secure Sockets Layer, for transparent encryption SSO Single Sign-On TLS Transport Layer Security, supersedes SSL VPN Virtual Private Network k8s Kubernetes container management package nginx Popular HTTP server
[Thread #324 for this sub, first seen 1st Dec 2023, 23:25] [FAQ] [Full list] [Contact] [Source code]
Bad bot. You don’t want to tell us what a CA is?
Obviously California or Canada.
Ah sorry this is a tech community so obviously it is computer animation.
