K3s + Nginx Proxy Manager: fixing the double TLS certificate conflict

The setup

You have a self-hosted K3s cluster. Nginx Proxy Manager (NPM) runs as a Docker container and handles all inbound traffic on ports 80 and 443. Your app lives inside K3s, exposed through Traefik as an ingress controller.

You add cert-manager with a ClusterIssuer pointing to Let's Encrypt, annotate your ingress, and run kubectl apply. The pod is healthy. But the site shows 404 Not Found — and the TLS certificate never gets issued.

Diagnosing: where does port 80 actually land?

The first reflex is to check what's listening:

ss -tlnp | grep ':80'

You'll see something like:

LISTEN 0  4096  0.0.0.0:80  0.0.0.0:*

Then check which process owns it:

ps aux | grep docker-proxy

root  1476001  docker-proxy -host-port 80 -container-ip 172.18.0.7 ...

NPM is a Docker container with ports: ["80:80", "443:443"]. It is the process that binds ports 80 and 443 on the host — not Traefik.

Traefik, in a standard K3s installation, is exposed as a NodePort service:

kubectl get svc -A | grep traefik

traefik  NodePort  10.43.147.1  9000:30900/TCP,8080:30080/TCP,8443:30443/TCP

Traefik only listens on 30080 (HTTP) and 30443 (HTTPS) on the host. It never receives traffic on port 80 from the outside world.

The root cause

cert-manager's HTTP-01 ACME challenge works like this:

1. cert-manager creates a temporary pod and a solver ingress inside K3s
2. It asks Let's Encrypt to validate: http://yourdomain.com/.well-known/acme-challenge/<token>
3. Let's Encrypt hits your server on port 80

The problem: that request lands on NPM, which has no proxy host configured for your domain yet. NPM returns 404. Let's Encrypt rejects the challenge. The certificate is never issued.

You can confirm this directly:

curl -v http://yourdomain.com/.well-known/acme-challenge/sometoken

The response headers will show Server: openresty — that's NPM's nginx, not Traefik.

Meanwhile, inside the cluster, cert-manager logs:

Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'

The double TLS problem

The ingress annotation traefik.ingress.kubernetes.io/router.tls: "true" combined with entrypoints: web,websecure makes Traefik try to serve HTTPS on its own. But:

• The TLS certificate isn't issued (challenge failed)
• Even if it were, HTTPS traffic hits NPM on 443 first — not Traefik's NodePort 30443

So you end up with two competing TLS systems:

• NPM: ready to handle SSL for your domain, but has no proxy host configured
• cert-manager: trying to issue a cert it can never get because port 80 is owned by NPM

Neither works. Hence the 404.

The fix

The architecture should be:

Internet → NPM (port 80/443, SSL handled by NPM)
               ↓
         Traefik NodePort 30080 (plain HTTP only)
               ↓
         Your app pod

NPM is the SSL termination point. K3s/Traefik only needs to handle plain HTTP routing internally.

Step 1 — Clean up the K8s ingress

Remove cert-manager annotations and the TLS section. Tell Traefik to use only the web entrypoint:

# k8s/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  namespace: myapp
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  ingressClassName: traefik
  rules:
    - host: yourdomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp-web
                port:
                  number: 80

Apply it and clean up the broken cert-manager resources:

kubectl apply -f k8s/ingress.yaml
kubectl delete certificate,certificaterequest,challenge --all -n myapp

Step 2 — Verify Traefik routes correctly

Test the routing internally before touching NPM:

curl -H "Host: yourdomain.com" http://127.0.0.1:30080/
# expect: 200 or 307 (redirect to /en or /home)

If you get 200, Traefik is routing correctly.

Step 3 — Configure NPM

In the NPM admin panel (port 81), add a new Proxy Host:

• Domain Names: yourdomain.com
• Scheme: http
• Forward Hostname / IP: 127.0.0.1
• Forward Port: 30080
• SSL tab: Request a new Let's Encrypt certificate, enable "Force SSL"

NPM owns port 80, so its ACME HTTP-01 challenge succeeds on the first try. The certificate is issued, HTTPS works, and traffic flows cleanly through the chain.

Why not keep cert-manager?

cert-manager is the right tool when Traefik is directly exposed on ports 80/443 — typically in a bare-metal K3s setup without a host-level reverse proxy. The moment you add NPM in front, cert-manager loses direct access to port 80 and becomes redundant: NPM already manages Let's Encrypt certificates natively.

Rule of thumb: only one system should own port 80 on a host. If that's NPM, then NPM handles TLS. If that's Traefik directly, then cert-manager handles TLS.

Key takeaways

• Server: openresty in a curl response means NPM answered, not Traefik
• Check what owns port 80 with ss -tlnp before debugging certificates
• cert-manager HTTP-01 requires direct access to port 80 — NPM blocks this silently
• The fix is architectural: remove TLS from K8s, let NPM terminate SSL, proxy to Traefik NodePort
• cert-manager and NPM are redundant — pick one TLS strategy per host