Questions and answers

What happens if the panel goes down?

Backups continue. The schedule and the last accepted plan are kept on each server, and the agent sends backups to the storages directly. It keeps its log locally and sends it once the panel is back. History marks that time: “the panel was unavailable” — History.

While the panel is down, its web interface, “Back up now”, plan changes, notifications, nightly checks and storage cleanup don't work. You can still restore data without it: sudo arcula restore on the server, or with the recovery kit on a new machine — Restore without the panel.

The panel can't report its own outage. Set up an external check of https://<panel address>/api/web/v1/health.

What happens if one storage goes down?

The others keep saving: each storage has its own send queue, and an unavailable one doesn't hold up the rest. On servers, the copy in that storage becomes “… behind”, and after the next access check — “no access”. The storage itself shows “No access”. As long as another storage has a fresh backup, the servers' status is “Needs attention”.

When the storage comes back, the latest finished backup goes there. Backups from the time of the outage won't be in it — they are in the other storage. Details — No access.

Can a compromised server delete backups?

Yes. arcula has no deletion protection: agents write to S3 directly with their own key, and there is no gateway between them and the storage — this was decided on October 24.

• “One key for all servers” mode — a compromised server can delete the backups of all servers in this storage.
• “Own key per server” mode — only its own, if the provider supports keys with access to a single folder.

What lowers the risk: a separate key for each server; a second storage at another provider or in another account; deletion protection at the provider itself, if it offers it — arcula doesn't configure it. After a compromise, rotate the S3 keys: when you disconnect a server, the panel reminds you which ones. The key mode is chosen on the storage — Storages.

Where are passwords and keys stored?

The agent doesn't send your database passwords or the contents of .env to the panel: it reads them inside the container at the moment of the dump. Details — Recovery kit and master key.

What happens if I lose the master key?

It encrypts the storage keys and repository passwords in the panel database. Without it, even a panel restored from a backup of its own database can't read these secrets. What remains is a restore with the recovery kit: it has everything needed and doesn't need the master key.

So keep both offline: the kit as a file on your computer, the master key in a password manager. Master key.

How do I check that a backup can be restored?

The recovery kit opens. arcula kit verify checks that the kit decrypts, that every repository of every server can be read, and that the latest backup is from the right server. No root needed; any Linux machine with restic will do. The command doesn't restore data.

arcula kit verify ./arcula-recovery-kit.age

The data comes back up. A trial restore on the server: in the wizard, choose the component mode. The database is imported into a new database next to the live one and checked; live data isn't touched.

sudo arcula restore

After that, the backup shows “component restored” on the “Backups” tab. Do this regularly, on each server in turn. Details — Restore.

What does “Gone from the server” in the plan mean?

It's in the plan, but on its last scan of the server the agent didn't find it anymore: the volume was deleted, the directory moved, the container removed. There's nothing to copy. When you save the plan, these rows leave it, and old backups with this data stay in the storages.

If it disappeared by mistake, put the data back on the server and click “Discover now”: server → “Protection settings”. Gone from the server.

Why does the log say “Not on the server: path from the plan”?

The agent started a backup, but a path or volume from the plan isn't on the disk. Most often a container refers to a directory that doesn't exist on this server — for example, a Compose file from CI with another machine's path. Or the directory was deleted. There's nothing to copy, and no data is lost.

Remove it from the plan: after “Discover now” it shows up in “Gone from the server”, and saving the plan removes it. While it's in the plan, the server may show “… is not in the backup” — A component is not in the backup.

Why does an incomplete backup still count?

If one component wasn't collected — for example, pg_dump failed — everything else in that backup was collected and saved, and you can restore from it. So “Latest copy” and the state of the copies in the storages count it too.

To keep this from hiding the problem, the server immediately becomes “Critical” with the line “… is not in the backup: reason” — until the next backup collects the component again. Take that component from an earlier backup — Backups.

sudo arcula status on the server counts “If it fails now” from complete backups only, so after such a failure the numbers in the panel and on the server differ.

How do I change the backup frequency?

In the server's plan, the “How often” card:

Choose a profile and click “Save plan”. The agent gets the new plan within a minute. The schedule uses the panel time zone. There is no custom interval besides these three in this version. How the target relates to data loss — How much would be lost.

What is “part 2/30”?

It's the nightly data check of a storage. Every night from 03:00 to 06:00 panel time, it reads one thirtieth of the data of each repository (restic check --read-data-subset=2/30) and verifies encryption and checksums. “2/30” is the part read that night. Over 30 nights all the data is read, and the cycle starts again.

If it finds an error, the cycle stops: the next night the same part is read. “Check data” on the storage page reads the next part right away. Data check.

How do I remove arcula from a server?

On the server:

sudo arcula uninstall

The command unlinks the server from the panel and removes the service, the binary, /var/lib/arcula and /etc/arcula together with the repository passwords. With --keep-config the passwords stay — you can use them to read the repositories without the panel. The command never deletes backups in the storages. If the panel is unavailable at that moment, the server is unlinked only on its own side — disconnect it in the panel manually.

In the panel the server becomes “Server disconnected”; history and backups stay. To remove it completely, with or without its backups — “Protection settings” → “Delete server”. Delete a server or storage.

What time zone does the panel show times in?

The time zone of the device you're viewing from. Open the panel in another country, and times are converted automatically. The log shows the full time with the time zone.

The panel has its own time zone: it is used for the backup schedule, nightly checks and cleanup, and for times in notifications. It's shown in “Settings” → “Time zone”. If you've moved for a long time, the same place has “Make it the panel time zone”.

How do I update agents?

A new agent version comes with the panel image. Agents install only a version signed with your key, so sign it first. Then “Settings” → “Agent updates” → “Update everywhere”, or “Update agent” on a single server's page.

The agent waits for the backup to finish and replaces itself. If the new version doesn't start within 3 minutes, it rolls back to the previous one by itself. The result is in the server log: “Agent updated” or “Agent didn't update”.

From the server — sudo arcula update. An older version can only be installed this way: sudo arcula update --version X.Y.Z. Details — Update agents and Agent updates.

What from the host goes into the backup?

The server settings you checked in the plan, in the “Server settings” card:

• users and groups — /etc/passwd, /etc/group;
• systemd services — /etc/systemd/system;
• cron — /etc/crontab, /etc/cron.d, user crontabs;
• Docker config — /etc/docker/daemon.json;
• Caddy and nginx — /etc/caddy, /etc/nginx;
• Let's Encrypt certificates — /etc/letsencrypt.

Not included: user passwords (/etc/shadow), SSH keys, installed packages, a full system image. On restore, host settings are only extracted to /var/lib/arcula/restore/<id>/host — you put them in place yourself. Backup methods.

Why is the copy “behind”?

The latest backup saved in this storage is older than the profile target. Common causes:

• the storage answers with an error — the reason is shown on the copy on the server's “Storages” tab, explained in Copy errors;
• the link is slower than the backups come — the log has many “Backup skipped”;
• this storage has an “at night” upload window — during the day the copy is behind, as intended;
• uploads to this storage are turned off;
• in “own key per server” mode, no key is set for the server.

Details — Copy is behind.

Why does the log say “Backup skipped”?

While the previous backup was being sent to the storage, a newer one was ready. The agent sends only the latest finished backup and skips the ones in between: there's no queue of dozens of backups.

Now and then this is normal. If there are many skips in a row, the link to this storage can't keep up with the schedule: choose a less frequent profile or send to this storage at night.

The agent is offline — are backups still running?

Yes, if the server is running. The agent makes backups by the last plan without the panel and sends the log later. While it's silent, the panel itself looks at its repositories in the storages once an hour and counts the backups it finds in “Latest copy”. What to check — Agent offline.

The plan is confirmed, but the server is “Not set up” — why?

The first backup isn't saved yet. The agent picks up the plan within a minute and makes the first backup right away, but a large first backup can take a long time to upload. Progress is shown on the server page: “Backing up: …”.

If no backup is saved within two profile targets after confirmation, the status becomes “Critical” with a reason — No fresh backup.

Does arcula delete old backups?

Only if you turned on cleanup for the storage. It's off by default, and backups pile up. Cleanup runs at night in the panel time zone and deletes what isn't covered by the retention rules. Always kept: the latest backups of each server, everything less than a day old, and the latest backup that was successfully restored from. Cleanup never touches pinned points. The agent on the server doesn't delete backups.

Cleanup, Pinned points.

What happens when a storage reaches its limit?

Nothing stops: the limit is your budget in the panel, not the provider's quota. From 90%, the storage shows “… left before the limit”. If the provider has its own quota, backups stop being saved once it's used up. Free up space with cleanup. Little space left before the limit.

Does arcula stop services during a backup?

No. PostgreSQL is dumped with pg_dump inside the running container; restic reads volumes and directories from disk on the fly. The only pause is in the “consistent SQLite copy” method: the container is frozen (docker pause) while the database files are copied, and only if they're 1 GB or less. A restore doesn't start anything either — you start the services. Backup methods.

Why do sudo arcula status and the panel show different things?

arcula status counts from the log on the server itself and works without the panel. The exact statuses are computed by the panel. Differences happen like this:

• after an incomplete backup: arcula status counts loss from complete backups only, the panel — from the latest saved one;
• freshness thresholds on the server are two and four schedule intervals; for the “Rarely changes” profile that's 48 and 96 h, while in the panel it's 36 and 72 h;
• the panel sees its own access and data checks of storages; the agent doesn't know about them.

How do I protect the panel itself?

Like any server: install the agent on the panel's server and include its PostgreSQL in the plan. Don't keep the master key only on that server — keep a copy in a password manager. Keep the recovery kit on your own computer. Without the master key, the secrets from a panel database backup can't be read, while the kit brings the data back even without the panel.

If the panel and Vault are on the same server, restore Vault first, then the panel. Restore without the panel.