Administrator manual

Who this is for: the person who installs Lattice, designs the addressing plan, points relays at it, chooses a DNS mode, mints API tokens, and is on the hook when DHCP is quiet.

Companion: Console user.

1. What Lattice is

Lattice is software-only DDI:

1. IPAM — the addressing plan (prefixes, subnets, pools, reservations, excludes). Source of truth.
2. DHCP — identical lattice-dhcpd processes that allocate from Postgres. No primary DHCP server. No split pools for HA.
3. DNS — pluggable adapters that write A/PTR into AD, BIND, or PowerDNS, or an embedded authoritative server (lattice-dns) for Lattice-managed internal zones.

One Postgres database is the system of record for intent and leases. DHCP nodes do not keep authoritative leases in memory. The UI talks only to lattice-api. dhcpd talks SQL on the hot path, not HTTP.

Public site: latticeddi.com. That hostname is the product, not the customer’s API hostname. Customer deployments keep their own URL.

Design rules you will feel every day

• Correctness over cleverness. A duplicate lease is a product-killing bug. Allocation is a SQL transaction with FOR UPDATE SKIP LOCKED.
• Intent vs reality. A prefix is a plan. A reservation is intent DHCP must honour. A lease is what dhcpd actually handed out. There is no generic “used” flag.
• Ack never waits on DNS. Grant commits the lease, then the worker claims an outbox row. Missing A/PTR after Ack is an outbox problem, not a DHCP rollback.
• Lease times are wall-clock. NTP/chrony on every node is a hard dependency. UTC everywhere.
• API first. If you cannot do it with Authorization: Bearer lattice_live_… against /api/v1, it is not a product feature.

What Lattice is not

• Not Infoblox DNS security, RPZ, or threat intel.
• Not DCIM (racks, cables). NetBox import creates missing prefixes only.
• Not a recursive resolver, workstation cache, or public/external DNS.
• Not the AD forest (_msdcs, site SRV). Leave that on the DCs.
• Not DNSSEC (internal zones stay unsigned on purpose).
• Not ISC/Microsoft pairwise DHCP failover. Shared Postgres is HA for leases.
• Not a BGP anycast speaker. You may put a VIP or anycast in front; health-check HTTP /readyz, never UDP/67.

2. Pieces you install

Same binaries for community and paid. The signed entitlement is the SKU.

Process config vs network intent

Process settings are environment variables (optional YAML LATTICE_CONFIG). Network intent lives only in Postgres.

Do not run dhcpd and api on the same LATTICE_LISTEN_HTTP port on one host.

3. Editions and license

Lattice enforces a signed entitlement file, not a serial typed into a box. Ed25519, public key compiled into the binaries. No phone-home.

Managed IPs (the meter) = distinct addresses that are leased + reserved + quarantined. Prefix size, free inventory, offered, and excluded do not count. A /16 with 200 live addresses is 200, not 65,536.

Community includes DHCPv4, IPAM, API, BIND TSIG. AD GSS-TSIG (dns.ad_gsstsig) is starter+. Cluster UI, join, bundled Patroni, backups UI, and dhcp-only extra nodes need feature ha.

Enforcement (this is the contract)

1. Missing or invalid file (including a hand-edited max_managed_ips) → community, 1,000. The unsigned cap is never trusted.
2. Over the cap: DHCP does not crash. Existing leases still renew. New Offers and new reservations return 402 license_limit.
3. After expires_at + grace_days (default 14): same as over-limit (renew only).
4. A reservation that already counts still gets DHCP when over the cap.
5. No per-node license. Any number of dhcpd processes.
6. max_managed_ips: 0 in a valid signed file is a revoke (renew only). Deleting the file falls back to community.

Admins replace the file in the console (License) or PUT /api/v1/license. The body is the JSON file. It is stored in Postgres so every node sees it. Operator and readonly can look; they cannot replace.

Download from the customer portal. Detail: License.

Go binaries are not DRM. The control that matters in the field is: signed file, community fallback that does not trust a forged cap, DHCP that keeps renewing.

4. Topology

Pick one. You can rebuild later with the same binaries and the same entitlement.

Single node (community, or paid that can take a window)

• One Ubuntu 22.04/24.04 VM, or the laptop lab (make up)
• Relays and browsers point at that VM (:8080 UI/API, :67 DHCP, :53 DNS)
• One Postgres. Two dhcpd processes in the lab compose are for development (Discover on A, Request on B). The community SKU is still one node.

Three-node HA (paid, feature ha)

• Three Ubuntu VMs in the same site (low RTT, same L2)
• One unused IPv4 on that subnet — the VIP. Not in netplan. keepalived owns it.
• Relays and browsers point at the VIP (HTTPS :443, DHCP :67, DNS :53)
• Each node runs api, worker, dhcpd, lattice-dns, maint, Patroni, etcd, HAProxy, keepalived
• etcd quorum is 2 of 3. Two nodes cannot form a safe cluster. If two die, the survivor does not become a writable primary (fail closed)
• Extra dhcp-only / dns-only nodes after the control plane exists: relays at that site point at the leaf, not the VIP

Bring your own Postgres (paid)

• lattice-install --mode byo-postgres --pg-dsn '…'
• You own failover. Lattice still assumes one writer.

Public cloud: keepalived needs real L2 and a free address the platform will accept. Many VPCs do not. Prefer single-node, or BYO Postgres plus a cloud load balancer, unless you already know how to run VRRP there.

Host sizing, netplan, NTP, ports, firewall: Install prerequisites. HA internals: Paid HA.

5. Install

Laptop lab

Requires Docker, Go 1.24+ (images use 1.26), Make, curl.

make up
curl -sf http://127.0.0.1:8080/readyz    # api + Postgres
# console: http://127.0.0.1:8080/

Stop: make down (keeps the volume). TLS lab: make secure → https://127.0.0.1:8443/ (self-signed).

Production box

1. Ubuntu Server 22.04 or 24.04 only. Fresh VM. Do not share it with another DHCP server, BIND on :53, or a second Postgres on :5432.
2. Stable IPv4 in netplan before the installer. Lattice detects the address; it does not rewrite the NIC (that can drop SSH).
3. Chrony synchronized. Offset well under 500 ms. UTC.
4. Paid: entitlement on disk or ready to paste. Community: skip; missing file is 1,000 IPs.
5. Extract the tarball, then:

sudo ./scripts/lattice-install

No arguments = wizard. It asks license, host specs, topology, node IP vs VIP, firewall, then a summary. b goes back. Nothing is installed until you confirm. The installer installs Docker if needed.

Non-interactive examples (full flags in lattice-install --help and Paid HA):

lattice-install --mode community
lattice-install --mode ha-primary --vip 10.0.0.10 --peers 10.0.0.11,10.0.0.12,10.0.0.13
lattice-install --mode ha-join --peer 10.0.0.11 --token <printed-on-primary> --ca-hash <printed-on-primary>
lattice-install --mode dhcp-only --peer 10.0.0.11 --token … --ca-hash …
lattice-install --mode upgrade
lattice-install --mode save-bundle --out lattice-images.tar

Join token and CA hash are printed on the primary. They are not shown in the Cluster console. That is deliberate.

Air-gap: save-bundle on a connected host, copy the tar with the release, --bundle on the dark site.

6. First login and tokens

Open the console. If no tokens exist, the login screen mints the first admin. Copy lattice_live_… once. It is never shown again. Stored as SHA-256; the table keeps a display prefix only.

Until the first token exists: POST /api/v1/tokens works from anywhere; other writes are loopback-only; GETs from the network are readonly so the UI can load. After the first token, Bearer is required. Missing/invalid Bearer is 401. Wrong role is 403.

Hiding a button is not security. The API enforces RBAC.

The secret lives in memory + sessionStorage (this tab), not localStorage. A new tab does not inherit it. Closing the tab drops it. XSS on the Lattice origin can still steal it — treat the UI as a client, not a vault. Serve UI and API from the same origin. Do not set Access-Control-Allow-Origin: *. Do not put the token in query strings or logs.

Mint more under Tokens. Rate limit: 10 token creates / min / IP.

Automation: Authorization: Bearer lattice_live_…. Idempotency: header Idempotency-Key on creates. Same key + same body → original 201. Same key + different body → 409.

7. Addressing plan (the mental model)

Build top-down. DHCP does nothing until a pool exists.

tenant (default is seeded)
  └── prefix          addressing plan block (may never be a DHCP scope)
        └── subnet    L3 network Lattice may serve
              ├── pool(s)         dynamic range
              ├── reservation(s)  this client always gets this IP
              ├── exclude(s)      never offer (gateway, VIP, already in use)
              └── options         extra DHCP options

Children of a prefix must sit strictly inside the parent. Pool ranges must sit inside the subnet and must not overlap. IPv4 inventory expands at most 65,536 addresses per pool.

How to mark an address

A reservation is a row dhcpd cannot violate. Dynamic clients never receive that IP. That MAC/DUID always gets it (in that subnet). At least one of MAC, client-id, DUID is required.

Deleting a subnet with active leases is 409 unless you check Force (?force=true), which drops those leases.

Tenants: prefixes and classes hang off a tenant. Omit tenant_id to use default. v1 is a slug, not hard isolation. Operators cannot list tenants.

First subnet (production or lab)

1. Plan → add a prefix (e.g. 10.20.0.0/16). This is the plan, not the DHCP scope.
2. Add a subnet inside it (e.g. 10.20.20.0/24) with gateway / VLAN / site if you have them.
3. Exclude the gateway (and any VIP). Single IP: start = end.
4. Add a pool (e.g. .10–.200). DHCP will not Offer until this exists.
5. Point relays at this node (paid HA: the VIP). Lab compose: host UDP :6767 / :6768. Direct clients need an interface binding; most sites use relays.
6. Prove it: ./bin/lattice-dhcptest -server 127.0.0.1:6767 -server2 127.0.0.1:6768 -giaddr 10.20.20.1 -n 5 (lab). -giaddr must sit in the subnet CIDR.

8. How DHCP actually works

Each lattice-dhcpd is interchangeable. Discover may land on node A and Request on node B. That is normal. Request re-validates against Postgres. Never assume “I offered this.”

Packet path (v4)

1. Receive UDP (direct or relayed).
2. Identify subnet: option 82 link-selection (RFC 3527) if present; else giaddr; else ingress interface.
3. Identify client: client-identifier if present, else chaddr (MAC).
4. Load reservation. If this client is reserved in this subnet, that IP is the only candidate.
5. Discover — Offer existing valid lease, else allocate a candidate. Short-lived offered row (~30s). Does not commit a long lease.
6. Request — if the IP is still this client’s (reservation, existing lease, or offered-to-this-client): commit lease, enqueue DNS upsert, Ack. Else Nak.
7. Release / Decline — transactional state change + DNS delete event. Decline quarantines the IP for LATTICE_QUARANTINE_TTL.
8. Renew / Rebind — extend expiry if the lease still belongs to this client.

Never allocate: reserved, excluded, leased, quarantined.

Lease states: offered · leased · released · expired · declined · quarantined. A client has at most one leased address per subnet. Unique IP among offered/leased/quarantined.

Server identifier (option 54) is not PXE. PXE is next_server (siaddr / option 66) and boot_file (option 67) on the subnet. Mismatch of server-id on Request is ignored on purpose so a VIP does not NAK a valid Request.

Relays and trust

Point DHCP relays at the VIP (HA) or the node (single). Health-check HTTP /readyz, never UDP/67. /readyz on dhcpd is a Postgres ping plus a subnet-count query.

LATTICE_TRUSTED_RELAYS is a comma-separated allow list. Empty means accept every relay (lab; logs a warning). Production should list the relay IPs/CIDRs. Option 82 is accepted and logged; subnet selection still uses giaddr / link-selection. Trusted sources are env, not an IPAM screen. A stolen admin token must not open host firewall ports.

Built-in options vs JSON options

Mask, router, and lease-time come from the subnet row. Set Gateway so IPAM and DHCP agree. NTP servers on the subnet becomes DHCPv4 option 42 (this is for clients; host clocks are chrony on the Lattice VMs — two different problems).

Extra options: code + JSON payload + optional vendor class.

{"addresses":["8.8.8.8","1.1.1.1"]}
{"ip":"10.20.20.1"}
{"string":"office.example.com"}
{"uint32":86400}
{"hex":"0102"}

Priority (later wins): global, then subnet, then pool, then class, then reservation. A stored option 42 wins over the subnet NTP field if both exist.

There is no dhcpd.conf snippet to edit on each node. Config is Postgres, refreshed via LISTEN/NOTIFY plus a short poll. Hot-reload does not drop UDP sockets.

Client classes

Match vendor_class, user_class, or mac_prefix. Attach the class to a pool to steer allocation. Class-scoped options override subnet options. Admin writes; operator reads.

IPv6

Prefixes and subnets are first-class. Lattice does not explode a /64 into inventory rows. Pools still cap expansion at 65,536 where inventory exists.

• IA_NA — DUID identity. Leave PD len blank on the pool.
• IA_PD — set PD len (e.g. 64). The lease stores prefix_len. That pool is not an IA_NA/v4 host pool.

Reservations may use DUID. If the VLAN is SLAAC-only, do not create a DHCPv6 pool.

PXE

On the subnet: PXE next-server and PXE boot file. Do not encode these as JSON DHCP options.

9. DNS — pick one mode per zone

Ack never waits. The worker claims the outbox (lease.granted / renewed / released / expired / declined). Failures retry with backoff; after 8 attempts the row is failed (dead-letter). Duplicate upserts are expected (at-least-once). Adapters must be idempotent. Do not roll back the lease.

Do not point the same zone at embedded and an external adapter. That is split brain.

Secrets are refs (env:LATTICE_TSIG_SECRET, file:/etc/lattice/lattice.keytab), never shown again after create. Store the actual secret on the worker host, mode 0600.

BIND TSIG

RFC 2136 + HMAC-SHA256. allow-update { key lattice.; }; on the zones Lattice owns. Worker → BIND 53. dhcpd does not need that path. Lab profile: docker compose … --profile bind. Docs: BIND TSIG.

Active Directory (GSS-TSIG)

Does not talk LDAP. Subprocess nsupdate -g (in-process GSS is not implemented). Dedicated AD user, keytab 0600, clock skew vs DCs under 5 minutes. Feature dns.ad_gsstsig is starter+. Docs: AD DDNS.

PowerDNS HTTP

Worker PATCHes rrsets on the Authoritative HTTP API. Create the zone in PowerDNS first. Docs: PowerDNS.

lattice-dns (embedded)

Authoritative only (RecursionAvailable=0). Serves SOA/NS, A/AAAA/PTR from leased rows plus static records. Off-zone → REFUSED. Name in zone, wrong type → NODATA. Missing name → NXDOMAIN. Recursion, RPZ, public DNS, AD-forest hosting, DNSSEC: out of scope.

AXFR is TCP. allow_transfer empty = any (lab only). Optional transfer TSIG. Serial bump sends NOTIFY to also_notify. IXFR is served when the slave serial matches the previous compiled snapshot; otherwise AXFR. Process restart forgets history, so the next IXFR is AXFR.

Lab publishes host 5354; production is :53.

Create a dns_server with adapter embedded, then bind the subnet’s DDNS adapter to that name (or embedded). Static records live on DNS (/dns): A, AAAA, CNAME, MX, SRV, CAA.

If workstations need a cache, run Unbound/BIND in front and conditional-forward our zones; leave AD on the DCs.

Hostname on the lease: option 12 is sanitized to one DNS-safe label. FQDN is zone plus that label.

10. Console map (admin)

11. Webhooks, audit, NetBox

Audit is append-only IPAM mutation (create/update/delete, token create/revoke, force-release, NetBox import, …). Readonly can look.

Webhooks POST outbox JSON after the DNS adapter runs (or is skipped). Same retry as DNS. Never rolls back the lease or the IPAM write. URL must be https (plain http only for localhost). Header X-Lattice-Signature: sha256=… is HMAC-SHA256 of the raw body. Topics: lease.granted, lease.renewed, lease.released, lease.expired, lease.declined, ipam.audit, or *. Operator cannot read webhook secret_ref.

NetBox: Lattice is master for DHCP subnets, pools, reservations, and leases. Import only creates missing prefixes. It never deletes and never writes pools. There is no live pull with a stored NetBox token — dump from NetBox, POST the JSON here.

12. High availability and backups (paid)

Community GET /api/v1/cluster is 403 feature_required. After a Starter+ file is on License, deploy with lattice-install (wizard) on three Ubuntu VMs.

Health is HTTP /readyz, not UDP/67. UDP 67 and 53 on the VIP are keepalived IPVS (open-source HAProxy has no generic mode udp). DNS TCP/53 stays on HAProxy. UI/API is HTTPS :443.

Fail closed: synchronous_mode on, use_pg_rewind off. Uncertainty → no writer. Duplicate leases are worse than a short outage.

Backups (Cluster UI, admin): off-box dump to s3://, sftp://, or a local path. Restore is one-shot onto a new cluster, then cut the VIP:

lattice backup restore --from s3://bucket/file.dump

Do not restore a dump onto a live writer that dhcpd is still using. PITR (WAL archive) is the real RPO; logical dump is a second copy. See Postgres.

Alert destinations (PATCH /cluster notify): webhook URL, Alertmanager base URL (we POST /api/v2/alerts), PagerDuty Events API v2 routing key, SMTP. Optional.

Upgrade: lattice-install --mode upgrade on a replica first, then the leader (switchover runs). --force if switchover stalls.

Diag: lattice diag -o bundle.tgz — logs, redacted configs, docker ps, Patroni/etcd, /readyz.

etcd is Patroni’s DCS only. Lattice binaries never speak etcd; they speak SQL.

13. Observability

Lattice does not speak syslog, SNMP, or NTP itself.

• Logs: JSON slog on stdout. DHCP lines include node, xid, mac, ip, subnet_id, msg_type. Do not parse logs for lease truth — Postgres is the store.
• Metrics: Prometheus /metrics on api, dhcpd HTTP, worker. Unauthenticated — scrape from a private network. After the first token, non-loopback scrapes need Bearer (loopback stays open).
• Webhooks: SIEM/CMDB subscribe to ipam.audit and/or lease topics.
• Overview: derived alerts (pool free, stale dhcpd, outbox lag, license) from Postgres every 8s.

Alert on lattice_outbox_lag, lattice_pool_free, lattice_dns_updates_total{result="failed"}, process up. Clock offset > 500 ms on hosts. GSS-TSIG dies opaquely beyond ~5 minutes vs DCs.

Ship journald → Vector/rsyslog. Examples: Observability.

14. Day-2 operations

Runbooks: Runbooks.

Postgres

Single writer. dhcpd /readyz is 503 while the writer is gone; the VIP should take the node out. Application code does not retry a write against a replica.

Lab DSN from the host: postgres://lattice:[email protected]:5433/lattice?sslmode=disable.

15. Kubernetes

Kubernetes examples keep Postgres external. They are not a hosted Lattice cluster and not an operator. Prefer lattice-install on Ubuntu unless you already run k8s as a platform. See k8s.

16. Checklist: a competent admin’s afternoon

• Host(s) meet the spec table for licensed IPs; Ubuntu 22.04/24.04; NTP; static IPv4
• Entitlement in place (or accept community 1,000)
• lattice-install completed; /readyz is 200
• First admin token copied once; extra operator/readonly tokens minted
• Prefix → subnet → exclude gateway → pool
• Relays point at VIP/node; LATTICE_TRUSTED_RELAYS set in production
• One DNS mode per zone; worker can reach BIND/AD/PowerDNS or lattice-dns
• A test client (or lattice-dhcptest) gets a unique lease; Overview shows it
• Release/expire removes DNS (eventually)
• Metrics scraped; webhook or SIEM on ipam.audit if required
• Paid HA: three nodes, VIP unused in netplan, Cluster page shows members, backup destination set

If any of that fails, Console user for console clicks. Day-2 symptoms: Runbooks.