SKYHOUSE.dev Journal

Why: After the May 27 Portainer container recreation pulled in portainer-ee:latest (which was 2.39.2), the Stacks-create UI started throwing instant "Network Error" toasts and any newly created stack's container was invisible from the global Containers page (visible only by drilling into the stack itself). Upgrading to 2.42.0 to see if a year of fixes would clear it.

Short version: we upgraded to 2.42.0 LTS→STS, the deploys still throw the phantom error and the new containers still don't appear in the global Containers list view. But the symptom is purely cosmetic — backend works correctly, containers run healthy, the per-stack detail view shows them. Documenting the state and the workaround so we don't go down this rabbit hole again.

1. Symptoms (still present on 2.42.0)

• "Network Error" toast on every long-running UI action (deploy stack, restart container, delete stack). Frontend axios client gives up after ~20s but the backend operation completes successfully a few seconds later.
• Newly created stacks appear correctly under Stacks; clicking into the stack shows the container; but the global Containers list omits them.
• Older stacks created before the May 27 image update display normally in all views.

2. What we ruled out

• NPM reverse proxy — added proxy_buffering off; proxy_request_buffering off; proxy_read_timeout 600s; proxy_send_timeout 600s; to host 27 (see earlier entry). Necessary fix but not sufficient.
• Resource controls — new stacks get the same Type:6, AdministratorsOnly:true ResourceControl entries as the working older ones. User damien is Role 1 (instance admin) which bypasses permission filtering.
• Compose labels — new containers have com.docker.compose.project matching their stack name exactly, identical structure to working stacks.
• Endpoint snapshot — the snapshot in portainer.db is refreshed every ~5 min and includes the new containers with all the right fields.
• UI-vs-API auth difference — when queried via an admin API key, Portainer's container list endpoint does return the new containers. The discrepancy with the UI view is on the rendering side and we did not localize it further.

3. Working workarounds

• Deploy stacks via the Portainer API, not the UI. Cleaner, faster, no phantom error, and (in our tests) API-deployed stacks do appear in the API's container list (just not the UI's). Example using a token created via Portainer → User profile → Access tokens:

  curl -X POST \
    -H "X-API-Key: $TOKEN" \
    -H "Content-Type: application/json" \
    "https://portainer.skyhouse.dev/api/stacks/create/standalone/string?endpointId=3" \
    -d '{"name":"mystack","stackFileContent":"services:\n  ...","env":[]}'

• Ignore the phantom Network Error when using the UI; wait 30–60 s and refresh the Stacks page — the deploy completes server-side. Do not re-click Deploy after the error or you'll create duplicate stack DB rows.
• For container management of new stacks, drill into the stack from the Stacks list instead of using the global Containers page.

4. Upgrade details

• Backed up the entire portainer_data volume to /home/plex/backups/portainer_data-pre-2.42.0-20260527-183810.tgz before the upgrade.
• Stopped and removed the running container, pulled portainer/portainer-ee:2.42.0 (pinned tag, no more :latest), recreated with identical bind mounts and port bindings:

  docker run -d --name portainer --restart=always \
    -p 9000:8000 -p 9443:9443 \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v portainer_data:/data \
    portainer/portainer-ee:2.42.0

• Migration ran cleanly: alert rule category backfill, refreshing RBAC roles, refreshing user authorizations, DB migrated to_version 2.42.0. Rollback would require restoring the backup — downgrades aren't supported across schema changes.
• Note: 2.42.0 is STS (Short-Term Support), not LTS. The previous lineage was 2.39.x LTS. Worth considering a return to the 2.39.x LTS branch the next time it sees a patch that addresses these UI issues.

5. New services deployed during the debug session

• sonarr (port 8989, config in /home/sonarr/data, library in /media/plex2/TV)
• radarr (port 7878, config in /home/radarr/data, library in /media/plex1/Movies)
• rdtclient (port 6500, config in /home/rdtclient/db, downloads to /media/plex3/Inbox)

All three were deployed via the API and respond with HTTP 200 on their UI ports.

6. Lessons

• Pin Portainer's image tag. The whole episode started because portainer-ee:latest silently rolled forward from a stable build to 2.39.2. We're now pinned to 2.42.0.
• The Portainer UI is fragile across upgrades; the API is stable. For automation and reliable stack deploys, prefer the API.
• NPM's per-host "Advanced" field is unforgiving — bad nginx syntax (missing semicolons) silently deletes the rendered conf file. Always check meta.nginx_err in the proxy_host DB row when a single host appears to drop offline.

End state: Portainer 2.42.0 pinned, all three new services running, NPM proxy serving long-running deploys without truncation, and the known-cosmetic UI quirk documented so future-me doesn't go chasing it again.