HAProxy Load Balancing Across Multiple Backends on Ubuntu

One app server is a single point of failure. It reboots for a kernel patch, or it falls over under a traffic spike, and your site is down. Two servers behind a load balancer means either one can die and users never notice.

HAProxy is the workhorse for this — a fast, battle-tested TCP/HTTP load balancer that fronts your backends, spreads requests across them, and quietly pulls a sick server out of rotation the moment its health check fails. It’s what sits in front of a huge slice of the internet’s traffic.

This guide sets it up on Ubuntu 24.04: one HAProxy node balancing across two or more app servers, with automatic health checks, a load-balancing algorithm that fits real traffic, a live stats dashboard, TLS termination, and the one trick that makes maintenance painless — draining a backend without dropping a request.

Prerequisites

Before you start, make sure you have:

• A load balancer server running Ubuntu 24.04 LTS — modest specs are fine; HAProxy is extremely light
• Two or more backend app servers already serving your app on a known port (this guide assumes HTTP on 8080)
• Root or sudo access on the load balancer
• Network reachability from the load balancer to each backend on the app port

HAProxy doesn’t run your app — it points at it. The backends keep serving exactly as they do now. HAProxy just becomes the single address the world connects to, and fans requests out behind the scenes.

Step 1: Install HAProxy

Ubuntu’s repository carries a recent, well-supported HAProxy. Install it:

sudo apt update
sudo apt install -y haproxy

Confirm the version — you want 2.8 or newer for the modern config keywords used below:

haproxy -v

The package ships a systemd service already enabled. You’ll configure it before starting for real.

Step 2: Understand Frontends and Backends

HAProxy’s config has two halves, and everything makes sense once they click:

• A frontend is what the world connects to — an IP and port HAProxy listens on, plus the rules for where traffic goes.
• A backend is a pool of real servers HAProxy forwards to, plus how it picks between them and how it health-checks them.

Traffic flows frontend → backend → one server in the pool. One frontend can route to many backends (by hostname or URL path); for now we keep it simple: one frontend, one backend, several servers.

Step 3: Write the Load Balancer Config

HAProxy reads /etc/haproxy/haproxy.cfg. The package installs a sensible global and defaults block — leave those, and append your frontend and backend. Open the file:

sudo nano /etc/haproxy/haproxy.cfg

Add this at the bottom, substituting your real backend IPs:

frontend web_front
    bind *:80
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /healthz
    http-check expect status 200
    server web01 10.0.0.11:8080 check
    server web02 10.0.0.12:8080 check
    server web03 10.0.0.13:8080 check

Every line earns its place:

• bind *:80 — HAProxy listens on port 80 on all interfaces. This is now your site’s front door.
• balance roundrobin — hand each new request to the next server in turn. More on algorithms in Step 5.
• option httpchk GET /healthz with http-check expect status 200 — HAProxy actively probes each backend by requesting /healthz and only keeps it in rotation while it answers 200. Expose a cheap health endpoint in your app; if you don’t have one yet, GET / works as a starting point.
• server … check — the check keyword is what turns health checking on for that server. Without it, HAProxy will happily forward traffic to a dead box.

Always validate the config before reloading — a typo takes the whole load balancer down:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

You want Configuration file is valid. Then start it:

sudo systemctl enable --now haproxy
sudo systemctl reload haproxy

Point a browser or curl at the load balancer’s IP on port 80 and you’re being balanced across all three backends.

Step 4: Turn On the Stats Dashboard

HAProxy has a built-in status page that shows every server’s health, live session counts, and bytes moved. It’s the fastest way to see the load balancer working. Add another frontend-like block:

listen stats
    bind 127.0.0.1:8404
    stats enable
    stats uri /
    stats refresh 5s
    stats admin if TRUE

Reload (sudo systemctl reload haproxy) and tunnel to it over SSH rather than exposing it — the page reveals your whole topology:

ssh -L 8404:localhost:8404 you@loadbalancer
# then open http://localhost:8404 locally

You’ll see each backend green (UP) or red (DOWN), request rates, and — because of stats admin — buttons to drain or disable a server by hand. Bind it to 127.0.0.1 as shown; never put the stats page on a public port without authentication.

Step 5: Pick the Right Balancing Algorithm

roundrobin is a fine default, but the algorithm should match your traffic:

Change it by editing the one balance line in the backend:

backend web_servers
    balance leastconn
    ...

For most stateless web apps, roundrobin or leastconn is the right answer. Reach for source only when a client must keep landing on the same backend.

Step 6: Sticky Sessions (Only If You Need Them)

If your app stores session state in memory on each server — rather than in a shared Redis or database — a user bounced to a different backend gets logged out. The clean fix is shared session storage, but when you can’t change the app, pin each user to a backend with a cookie:

backend web_servers
    balance roundrobin
    cookie SRVID insert indirect nocache
    option httpchk GET /healthz
    http-check expect status 200
    server web01 10.0.0.11:8080 check cookie web01
    server web02 10.0.0.12:8080 check cookie web02
    server web03 10.0.0.13:8080 check cookie web03

HAProxy sets a SRVID cookie naming the backend that served the first request, then routes that user back to the same server. indirect nocache keeps the cookie invisible to the app and uncached.

Sticky sessions are a crutch, not a design. They undo half the benefit of load balancing — if a server dies, every user pinned to it loses their session anyway. Move session state to Redis when you can, and delete this block.

Step 7: Terminate TLS at the Load Balancer

Terminating HTTPS on HAProxy means one place to manage certificates and backends that speak plain HTTP internally. HAProxy wants the certificate and private key concatenated into a single PEM file. If you use Certbot (see my Let’s Encrypt guide), assemble it:

sudo mkdir -p /etc/haproxy/certs
sudo bash -c 'cat /etc/letsencrypt/live/example.com/fullchain.pem \
  /etc/letsencrypt/live/example.com/privkey.pem \
  > /etc/haproxy/certs/example.com.pem'
sudo chmod 600 /etc/haproxy/certs/example.com.pem

Then bind port 443 with the certificate and redirect plain HTTP up to HTTPS:

frontend web_front
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/example.com.pem
    http-request redirect scheme https unless { ssl_fc }
    default_backend web_servers

{ ssl_fc } is true when the connection already arrived over TLS, so the redirect only fires for plain HTTP. Validate and reload, and your backends never have to think about certificates again.

Certbot renewal: the renewal hook must re-concatenate the PEM and reload HAProxy. Drop a script in /etc/letsencrypt/renewal-hooks/deploy/ that rebuilds example.com.pem and runs systemctl reload haproxy, or renewals will silently stop matching the served cert.

Step 8: Drain a Backend for Zero-Downtime Maintenance

This is the payoff. To patch or deploy to web01 without dropping a single in-flight request, put it in drain state first — HAProxy stops sending it new connections but lets existing ones finish. From the stats page (Step 4), click the server’s Drain button, or do it from the command line via the admin socket:

echo "set server web_servers/web01 state drain" | \
  sudo socat stdio /run/haproxy/admin.sock

Watch the stats page until web01’s session count hits zero, then take it down, patch it, bring it back, and return it to rotation:

echo "set server web_servers/web01 state ready" | \
  sudo socat stdio /run/haproxy/admin.sock

(sudo apt install -y socat if you don’t have it.) Repeat per server and you’ve done a rolling maintenance window with no visible downtime — the whole reason you built this.

Common Issues and Fixes

Every backend shows DOWN on the stats page

The health check is failing. HAProxy is requesting /healthz and expecting 200 — if your app returns 404 there, either add the endpoint or change option httpchk to a path that exists. Test what the backend actually returns: curl -i http://10.0.0.11:8080/healthz from the load balancer.

curl to the load balancer hangs or refuses

HAProxy isn’t listening or a firewall is blocking port 80/443. Check sudo systemctl status haproxy and sudo ss -tlnp | grep haproxy, then confirm UFW allows the web ports: sudo ufw allow 80,443/tcp.

Config won’t reload — “cannot bind socket”

Another service already owns port 80 (often a leftover Nginx or Apache). Find it with sudo ss -tlnp | grep :80 and stop it — only one process can bind the port.

Users randomly logged out

Session state lives in memory on the backends and you’re load balancing without stickiness. Add the cookie config from Step 6 as a stopgap, then move sessions to shared storage.

Changes to haproxy.cfg don’t take effect

You edited the file but ran restart on a bad config, so systemd kept the old process — or you never reloaded. Always sudo haproxy -c -f /etc/haproxy/haproxy.cfg first, then sudo systemctl reload haproxy (reload is graceful; restart drops connections).

Summary

Here’s what you built:

1. HAProxy installed on Ubuntu 24.04, fronting two or more app servers on a single public address
2. A frontend and backend with active health checks that pull a sick server out of rotation automatically
3. A live stats dashboard, bound to localhost, showing every server’s health and load
4. A balancing algorithm matched to your traffic, with sticky sessions available when the app forces it
5. TLS terminated at the load balancer, so backends speak plain HTTP internally
6. A zero-downtime drain workflow for rolling maintenance

The natural next step is removing the load balancer itself as a single point of failure — two HAProxy nodes with a floating IP via keepalived — plus wiring it into Prometheus so you graph backend health over time. Both are queued as future posts.

Running a single app server and nervous about the next reboot, or fighting a load balancer that keeps dropping sessions? Get in touch — high availability is one of the highest-leverage things to get right.