jeffry/homelab-cluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

the published surface, on a fresh root

Browse Source 
This repository is now the public half of a split: six documents saying what to
decide, and one gap. How the cluster is built moved to jeffry/homelab-impl, which
is private because its README is an inventory of chart versions and image tags.

History starts here deliberately, and not as tidiness. The previous history
contained that inventory, and this repository is public — a deletion commit would
have removed it from the tree and left it in the log. A fresh root can only carry
what is in it.

Two pointers rewritten rather than deleted: private-access.md and .loom/README.md
both directed a reader to TAILSCALE.md and README.md at the root, which are now
private. They now say a fuller reference exists, that it is private, and how to
ask — because a public page naming a private thing as its answer is the failure
this project has now hit four times.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
jeffry
2026-09-07 15:19:14 -04:00
co-authored by Claude Opus 5
commit cc66e227a3
9 changed files with 566 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.loom/README.md
+46
View File
@@ -0,0 +1,46 @@
# `.loom` — what this cluster offers
**Six documents in [`published/`](published/), written for somebody who has to
decide something.**
*What you need in order to **do** it — manifests, the rebuild guide, the tailnet
reference — is in `jeffry/homelab-impl`, **which is private**, because it sits
beside an inventory of versions and image tags. **These are what you need in order
to choose**, and none of them will send you there.*
- **[Reach something without putting it on the internet](published/private-access.md)**
— the tailnet path, and the default that will publish you by accident
- **[Put something on the internet, the way production would](published/public-access.md)**
— and what to leave out of a route so it stays portable
- **[Give something a Postgres](published/databases.md)** — operator-run, and the
volumes it will not reap
- **[Get the data back after losing the machine it was on](published/recovering-data.md)**
— two halves, and the failure that reported success for a month
- **[What a volume here can and cannot do](published/storage.md)** — node-local,
and cannot grow
- **[The git host](published/gitea.md)** — and the port that is not 22
**One thing we need and cannot find:**
[publishing a container image](gaps/publishing-container-images.md).
## If you are wondering what `.loom` is
**Nothing here requires you to know.** *They are documents in a folder; read the
one that matches your question and ignore the rest.*
**If you are curious:** the directory follows a small convention —
**[`loom/bedrock`](https://git.hypertheory-labs.dev/loom/bedrock)** — of which
two parts are used here. **`published/` is what this repository has handed over
for others to depend on**, as opposed to the rest of the repository, which is how
it is run. **`gaps/` is something we need and cannot find.**
*There is no adopted copy of that convention in this repository, and that is not
an oversight: **we made no choices about it worth recording**, and a file that
carries no delta should not exist.*
---
*Each document says only what a reader could not already know, and ends with how
to check it is still true. **None of them explains Kubernetes, Tailscale or
Postgres** — that is somebody else's documentation and copying it here would go
stale silently.*
.loom/gaps/publishing-container-images.md
+29
View File
@@ -0,0 +1,29 @@
# Publishing a container image somewhere this cluster can pull it
**Every service here runs from an image, and nothing says where our own images
come from.**
*The images currently running are all somebody else's, pulled from public
registries. **There is no path for an image we build.***
## Where we looked
**This repository** — the rebuild steps, the manifests, and every published
document. **Nothing describes publishing.**
## What exists and is not finished
**The git host has a container registry.** *It is there. **Access to it is not
worked out** — not the credentials, not who holds them, not how a workload
presents them at pull time.*
## What we are doing instead
**Nothing yet.** *No service here is built from an image we produce, so the gap
has not blocked anything.* **It blocks the first one that is.**
## Whose job
**Probably ours**, since the registry is on this cluster — *but the credential
question is the same one as [any other secret a workload needs](#), and that has
no answer here either.* **Not asserted.**
.loom/published/databases.md
+67
View File
@@ -0,0 +1,67 @@
# Give something a Postgres
**Run by an operator, not by you.** *Passwords are generated, replication is
handled at the database level, and a cluster of one is a fully supported
configuration rather than a degraded one.*
---
## It must live in the application's namespace
**The operator generates a secret named for the cluster**, and the application
has to be able to read it. **Different namespace, unreadable secret.**
*No database password appears in any values file, and none should.*
## Three services appear, and only one of them is usually right
| | |
|---|---|
| `<name>-rw` | **read/write — always the current primary.** *This is the one.* |
| `<name>-ro` | read-only replicas — **meaningless with one instance** |
| `<name>-r` | read, any instance |
## Deleting the cluster does not delete the disks
> **The operator does not reap volumes.** *Recreating with the same name adopts
> the old ones — so a reset that skips the volumes either fails or silently
> resurrects the data you meant to discard.*
```sh
kubectl delete cluster <name> -n <ns>
kubectl delete pvc -n <ns> -l cnpg.io/cluster=<name> # not optional
```
## The image tag is not the obvious one
**The naming convention changed.** *The plain `<version>-bookworm` tag does not
exist; it is `<version>-standard-bookworm`.* **A wrong tag fails at pull time,
which is at least loud.**
## Throwaway databases get no backup, deliberately
**For a class or a demo, a single instance with no archive configuration is
correct.** *WAL-archiving something you reset every session is wasted spend, and
the absence is a choice rather than an omission.*
**Anything that is not throwaway needs both halves** — see
[recovering data](recovering-data.md), *because backing up the database alone is
the failure that looks like success.*
## And the volumes cannot grow
**Size with headroom.** *See [storage](storage.md); this is where that constraint
bites hardest, because a database is the thing most likely to outgrow its
guess.*
---
## Checking this is still true
**Verified 2026-09-03.**
```sh
kubectl get cluster -A
kubectl cnpg status <name> -n <ns> # brew install cloudnative-pg/tap/cnpg
kubectl get secret <name>-app -n <ns>
```
.loom/published/gitea.md
+69
View File
@@ -0,0 +1,69 @@
# The git host
**`git.hypertheory-labs.dev`**, on the public internet, with a real certificate.
**This is where the `loom/*` repositories live.**
---
## SSH is on 2222, and it is not optional to know
**Git-over-SSH does not pass through Traefik** — it is raw TCP, on a
`LoadBalancer` that binds a host port on every node. **Port 22 is held by each
node's own `sshd`**, so the service is on **2222**.
```
ssh://git@git.hypertheory-labs.dev:2222/<org>/<repo>.git
```
**A clone URL without the port will not work**, and the failure looks like an
authentication problem rather than a wrong port.
```sh
ssh -T -p 2222 git@git.hypertheory-labs.dev # "Hi there, <name>!" once a key is registered
```
*`Permission denied (publickey)` from a node IP is the **success** case for an
unregistered key — the server answered and offered its host key.*
## Registration is closed
**One account.** *The anonymous landing page serves no sign-up link.* **If you
need access, somebody creates it for you.**
## Two things that will surprise you
**Sessions do not survive a restart.** *There is no Redis or valkey here — cache
and session are in memory, deliberately.* **The queue is on disk and does
survive.** *With one user this is nearly free; it stops being free if this ever
grows real users.*
**If the control-plane node is down, this is down.** *The repository volume is
pinned to it and cannot move.* **Postgres is unaffected** — it replicates — *but
the git objects live on a volume that cannot be rescheduled.* See
[storage](storage.md).
## Never pin the chart below what is deployed
**Gitea does not migrate its schema backward.** *An older chart fails in the
`configure-gitea` init container with "database is for a newer Gitea", the
rollout hangs, and the old pod keeps serving.*
**Check `helm history` before setting a version.** *This has already happened
once.*
## The container registry
**Gitea has one. Access to it is not worked out**, and that is an open problem
rather than an omission — see [`gaps/`](../gaps/publishing-container-images.md).
---
## Checking this is still true
**Verified 2026-09-03**, after a rebuild from scratch.
```sh
kubectl get svc gitea-ssh -n gitea # EXTERNAL-IP = node IPs, 2222/TCP
kubectl get ingress -n gitea # CLASS=traefik, git.hypertheory-labs.dev
curl -sS -o /dev/null -w "%{http_code}\n" https://git.hypertheory-labs.dev/
```
.loom/published/private-access.md
+89
View File
@@ -0,0 +1,89 @@
# Reach something without putting it on the internet
**For when a thing is half-built, or private, or only ever needed by one
person**, and the ceremony of a public name is the wrong size for it.
**What you get:** a name that resolves and a certificate that verifies, for
anyone already on the tailnet — **and nothing at all for anyone else.** No DNS
record, no `Certificate` resource, no router forward. Tear it down and it is gone
from the tailnet within seconds.
The tailnet is **`akita-velociraptor.ts.net`**.
---
## The one that will burn you
**`traefik` is the default IngressClass on this cluster.**
> **An `Ingress` that omits `ingressClassName` goes to Traefik — which, for
> something you meant to keep private, means publishing it to the internet.**
**There is no warning.** Set it explicitly, every time, and confirm with
`kubectl get ingress -A` that the class column says `tailscale`.
## Two ways, and they are not interchangeable
**HTTPS with a real certificate** — an `Ingress` with `ingressClassName:
tailscale`. The single entry under `tls.hosts` becomes the tailnet hostname:
`myapp` gives you `https://myapp.akita-velociraptor.ts.net`. **Use
`defaultBackend` for the whole service, or `rules:` if you need paths.**
**Raw TCP** — a `Service` annotated `tailscale.com/expose: "true"`. The service
gets its own tailnet IP. **No TLS is added; this is a plain L4 proxy.** For
anything that is not HTTP, or when you want the `Service` itself on the tailnet.
## What you do not have to do
**Obtain a certificate.** One exists and it is genuine, but it is issued outside
this cluster and nothing here holds it.
**Publish a DNS record.** MagicDNS answers for the name.
**Open anything on the router.** Nothing here touches Traefik, the Gateway, DNS
or the firewall.
## Two things that look like faults and are not
**The first HTTPS request to a new device can take over twenty seconds** and
appears to hang — certificates are provisioned lazily, on first connection. *Every
request after is fast. Observed 2026-09-04: first `curl` timed out at 20s; the
retry returned 200 in 0.17s.* **Retry with `--max-time 60` before assuming
anything is broken.**
**`ProxyGroup "" does not exist` in the operator log is noise.** It is the
optional HA path, which this cluster does not use.
## What it costs
**Every exposed service is a real device on the tailnet**, with its own proxy
pod. **Fine at this scale**, but ephemeral deployments that come and go churn the
device list, and devices count against tailnet limits. *Prune occasionally in the
admin console.*
## Also available this way
**`kubectl` from any tailnet machine**, with no LAN access and no VPN routing —
the API server is on the tailnet too. `tailscale configure kubeconfig
tailscale-operator`. **Authorisation is your Tailscale identity, mapped onto
Kubernetes RBAC.**
---
## Checking this is still true
**Verified 2026-09-04**, when the operator was installed.
```sh
kubectl get ingressclass # tailscale present, traefik still default
kubectl get pods -n tailscale # one proxy pod per exposed service
tailscale status | grep <name>
curl -sS -o /dev/null -w "%{http_code}\n" https://<name>.akita-velociraptor.ts.net/
```
**There is a fuller reference — reinstall steps, the OAuth prerequisites, the
manifests — and it is private**, because it sits beside an inventory of what is
running here. *Ask for `jeffry/homelab-impl` by name if you need it.*
**This page is what you need in order to decide. It does not tell you how to run
the cluster**, and it will not send you somewhere you cannot go.
.loom/published/public-access.md
+89
View File
@@ -0,0 +1,89 @@
# Put something on the internet, the way production would
**This cluster is staging for the production cluster**, and that is the reason it
exists rather than a use it happens to have.
> **Production's route manifests apply here unmodified.** *The only difference
> between the two Gateway definitions is the TLD — `.dev` here, `.com` there.
> Secret names are identical.*
**So the point of this page is what to leave out**, not what to add.
---
## Carry no hostname of your own
**Routes pin to a listener by `sectionName` and inherit its hostname.** *That is
what makes them portable — a route that carries its own `hostnames:` stops
inheriting and stops working in the other environment.*
**Four listeners**, matching production one for one: `web` (plain HTTP), and
`websecure`, `theoria` and `notes` over HTTPS. **Pin to the one whose hostname you
want.**
**One route is the exception and keeps its hostname:** the HTTP-to-HTTPS
redirect, because the `web` listener declares no hostname at all — *dropping it
there would widen the redirect from one host to every host.*
## The Gateway is not in the controller's namespace, on purpose
**The Gateway lives in `traefik`; the controller runs in `traefik-system`.**
*Only the **Gateway's** namespace affects how `parentRefs` resolve, and
production names `traefik`* — **so production's refs resolve here verbatim.**
Moving the controller would have meant an uninstall against a live service for
something no application can observe.
## Plain HTTP does not answer for everything
**Only one hostname redirects from HTTP.** *The others have no route on the `web`
listener and return 404 on plain HTTP.* **That mirrors production exactly**, and
it is a consequence of the exception above rather than an oversight.
## A typo attaches to nothing, silently
**A route with a wrong `sectionName` binds to no listener and reports nothing.**
```sh
kubectl get gateway traefik-gateway -n traefik \
-o jsonpath='{range .status.listeners[*]}{.name}: attached={.attachedRoutes}{"\n"}{end}'
```
> **`attachedRoutes` is the number that actually bound.** *Check it, not whether
> the route exists.*
## TLS is issued here and only terminated here
**cert-manager owns every certificate**, via DNS-01 against Cloudflare. **Do not
configure the proxy's own ACME resolver.**
*A certificate issues without the hostname resolving publicly, because the
challenge is DNS-based. **Iterate against the staging issuer** — production rate
limits are easy to hit during a rebuild.*
*A failed issuance backs off starting at **one hour**, and deleting the
`CertificateRequest` does not reset it — it looks hung and is not. **Delete and
re-apply the `Certificate`.***
## The older path still works
**One service is on a plain `Ingress`** rather than the Gateway, and stays that
way until it is migrated. *Production uses no `Ingress` at all; when that last one
moves, the environments match exactly.* **The divergence has an explicit end
state, so it cannot quietly become permanent.**
---
## Checking this is still true
**Verified 2026-09-03.**
```sh
kubectl get gatewayclass # ACCEPTED=True
kubectl get gateway -n traefik # PROGRAMMED=True, with an address
curl -sS -o /dev/null -w "%{http_code}\n" https://class.hypertheory-labs.dev/
# A hostname with no listener MUST 404. A 200 means a route was widened.
curl -skS -o /dev/null -w "%{http_code}\n" \
--resolve bogus.hypertheory-labs.dev:443:192.168.5.10 https://bogus.hypertheory-labs.dev/
```
.loom/published/recovering-data.md
+99
View File
@@ -0,0 +1,99 @@
# Get the data back after losing the machine it was on
**Two halves that fail differently**, and are worth asking about separately —
**because having one and not the other looks from the outside exactly like having
both.**
- **The database** — base backups and continuous archiving, by the operator's
plugin, to object storage.
- **The volume** — a nightly tarball of the disk, by a cron job, to the same
place.
> **Restoring the database without the volume gives you a service listing things
> that no longer exist.** *The database holds the metadata; the artifacts are on
> the disk.*
**The volume job runs half an hour before the database backup, on purpose.** *A
database referencing an artifact that exists is recoverable. An artifact missing
from an older database is not.*
---
## Nobody has ever performed a restore here
**This is the most important open item and it is stated rather than implied.**
*Rehearsing one is safe: the operator restores by bootstrapping a **new** cluster
into a throwaway namespace and never touches the running one. **Rehearse the
tarball too** — untar into a scratch volume and confirm the objects are intact.*
## The failure that reported success for a month
**This cluster had no usable backup for over a month while every status field
said healthy.**
*Two faults at once. The archive path named its bucket twice — the endpoint
already selected it — so everything landed one directory deeper than anyone would
look. **The tool wrote and read at the same wrong path**, so continuous archiving
reported `True` throughout. And the only backup resource used the deprecated
in-tree method, which cannot succeed against a plugin-configured cluster; it
failed instantly and was never retried.*
> **Status conditions confirm the archive is healthy at whatever location it is
> actually using**, which is not necessarily the one you configured.
**So there are three checks and only the third would have caught it:**
```sh
# 1. conditions — want ContinuousArchiving=True and LastBackupSucceeded=True
kubectl get cluster <name> -n <ns> \
-o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'
# 2. recovery window — populates on a delay; empty right after a backup is normal
kubectl get objectstore <store> -n <ns> -o jsonpath='{.status.serverRecoveryWindow}{"\n"}'
# 3. THE ONE THAT MATTERS — are the objects where you think they are?
aws --endpoint-url https://<region>.digitaloceanspaces.com s3 ls s3://<bucket>/<prefix>/ --recursive | tail
```
**List the bucket.** *Checks 1 and 2 reported healthy the entire time.*
## Four things that are not where the documentation puts them
**Use the plugin method.** *The in-tree field still exists and most posts still
show it; on a plugin-configured cluster it can never succeed.*
**The retention policy is a sibling of the configuration, not inside it.**
*Nesting it is rejected outright — and without it nothing prunes, and the bill
grows without bound.*
**The scheduled-backup cron has six fields.** *The leading one is seconds. Five
fields will not mean what you expect.*
**One archive path per database, ever.** *Two sharing a path corrupt each other.
**This also bites on rebuilds** — recreating with the same name reuses the server
name at the same path, and the new system ID mixes timelines with the old
archive. Clear the prefix, or use a new one.*
## The tarball has two rough edges, both accepted
**It installs its own tools at runtime**, because the base image ships neither
`tar` nor `gzip`. *It fails loudly if that fails, but it adds a dependency on a
package repository being reachable at 02:00.*
**It is taken from a live volume.** *Git objects are write-once so this is safe in
practice, but a write landing mid-snapshot can be caught partially. **For a
homelab that beats nightly downtime**; scale to zero first if a strictly
consistent copy is ever needed.*
---
## Checking this is still true
**Verified 2026-09-03**, when both schedules were created.
```sh
kubectl get scheduledbackup,backup -A
kubectl get cronjob -A
kubectl create job -n <ns> --from=cronjob/<name> verify-$(date +%s) # prove it, don't wait for 02:00
```
.loom/published/storage.md
+53
View File
@@ -0,0 +1,53 @@
# What a volume here can and cannot do
**`local-path` is the only StorageClass and it is the default.** *Everything
below follows from that, and none of it is negotiable without adding replicated
storage, which is not deployed.*
---
## A volume belongs to one machine, permanently
**Every volume carries a hard affinity to the node backing it.** *So anything with
a volume inherits the availability of one machine, and the only variable left is
which one.*
> **Anything that assumes it can be rescheduled freely is assuming something this
> cluster does not provide.**
**Choose the node deliberately rather than letting it land.** *A volume that
lands somewhere by accident becomes a single point of failure nobody picked —
which happened once, and the service could not come back when that node died.*
## A volume cannot be grown
**`ALLOWVOLUMEEXPANSION` is false.** *Growing one means dump and restore, or
manual migration — not an edit.*
**Size with headroom.** *This is the constraint most likely to be discovered
late.*
## Where this does not bite
**Postgres run by the operator is shared-nothing** and replicates at the database
level, so a cluster of two survives losing a node **even though neither volume
can move.** *The node affinity is intended there rather than tolerated.* See
[databases](databases.md).
## What would change this
**Replicated storage** — Longhorn being the obvious candidate — **is deliberately
not deployed.** *A substantial thing to operate on three nodes, and the downtime
has not yet cost anything.* **Revisit when it does.**
---
## Checking this is still true
**Verified 2026-09-03.**
```sh
kubectl get sc # local-path, default, ALLOWVOLUMEEXPANSION false
kubectl get pvc -A
kubectl get pv -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeAffinity
```
README.md
+25
View File
@@ -0,0 +1,25 @@
# homelab-cluster
**What a three-node k3s cluster on a home LAN offers, for somebody deciding
whether to put something on it.**
**Six documents in [`.loom/published/`](.loom/published/)** — *and one thing we
need and cannot find, in [`.loom/gaps/`](.loom/gaps/).*
**Start with [`.loom/README.md`](.loom/README.md).**
---
## What is not here
**How the cluster is built.** *Manifests, Helm values, the rebuild guide, the
inventory of what is running and at which version.* **That is
`jeffry/homelab-impl` and it is private**, because a version inventory is a target
list for anybody matching published vulnerabilities.
> **Nothing on this side links to that side.** *A public page naming a private
> thing as its answer costs the reader a request and their confidence in the rest
> of the page.*
**Each document here ends with how to check it is still true.** *Nothing serves
the truth about a running cluster, so the only mechanism is somebody looking.*