Storage checks

The panel watches each storage by itself, without agents: it checks access, measures space and reads part of the data at night. The result is in the “Status” column of the storage list and in the “Now” card on the storage page.

What “Status” shows

Critical first, then warnings, then facts:

Storage status: “Critical” — no access; “Needs attention” — backups aren't being saved, the data check found an error, 90% of the limit is used, or the space will last less than 30 days; otherwise “OK”.

Access check

What is checked. With the storage key, the panel works in the test prefix .arcula-check/: it lists objects (list), writes a test object (write), reads it back and compares it with what was written (read), and deletes it. It also checks multipart upload, versioning and Object Lock. Other data isn't touched.

When. Every 6 hours; with the “Check access” button; right after the key mode changes; when a copy was saved while access was marked as broken. A new storage key is checked before it's saved: the old one stops being used only if the new one passed. A server key in “Each server has its own” mode is checked when it's entered — in that server's folder.

Results — as S3 answered, without conclusions: “✓ passed”, “✕ error”, “allowed — no deletion protection”, “the provider doesn't answer this request (NotImplemented)”, “no permission to check”. The full table is on the “Connection” tab, under “Capability checks”.

If list, read or write failed, the storage is “Critical”: “No access: 403 AccessDenied since 12:02”. Server copies here show “no access”, the nightly data check and cleanup don't run, and the size isn't updated. The other storages keep saving.

What to do:

1. Look at the error code: 403 AccessDenied — the key was revoked or lacks permissions, NoSuchBucket — there's no such bucket, timeouts — the network or the provider.
2. Fix it at the provider or replace the key: “Connection” → “Replace”.
3. Click “Check access”.

Space and “Lasts for”

Measurement. Every 45 minutes, after each saved copy and after space is freed, the panel lists the objects in the bucket and adds up their sizes. “Used” is the sum of our objects, “measured 12 min ago”. S3 doesn't provide the provider's metrics, so this isn't the figure from the provider's bill.

“Lasts for” is the space left before the limit divided by the average growth per day. Growth is taken from the measurements over the last 30 days.

Space doesn't grow when cleanup frees as much as backups add — then it's also “—”. Who adds how much — the “Cleanup” tab, “Where the growth comes from”.

Data check

Every night, between 03:00 and 06:00 panel time, the panel's worker runs this for each server repository in this storage:

restic check --read-data-subset=2/30

restic checks the repository structure — indexes and snapshots — and reads one thirtieth of the data files from S3: it decrypts them and compares them with their checksums. The next night it reads part 3/30, and so on in a cycle: in 30 nights all the data is read, then the cycle starts again.

“Checked 03:18: part 2/30, no errors” means: tonight the panel read part 2 of 30 of each repository from the storage and finished by 03:18 — all files matched their checksums.

• Why in parts: reading everything means downloading the whole size of the backups from S3. In parts, the traffic and time are spread over 30 nights.
• “Check data” reads the next part right away, without waiting for the night.
• At night the check runs only if there is access to the storage and it has repositories, and no more than once every 12 hours. If you ran a check manually in the evening, the nightly one may be skipped.
• While the check runs, the repository is locked: the server's backup to this storage waits for it to finish.
• On a server, the check is shown on the “Storages” tab, in the “Data check” column. In the storage log — “Data check” with a step for each repository.

What it proves and what it doesn't

It proves that the data at the provider is intact and can be read with the repository password. It doesn't prove that the application will come up from the backup — that the dump is correct and the service starts. Only a restore with arcula restore checks that; its result is in the “Restore” column of the server's “Backups” tab (Points and copies).

If an error was found

The storage shows “Needs attention”: “Data check: 1 damaged data file in the prod-db-01 repository — pack 7c1e…: ciphertext verification failed”. The log shows “Data check: error found” with the restic lines. The cycle doesn't move on: the next night the same part is read.

What to do:

1. Click “Check data”. The panel reads the same part of the damaged repository again. If there's no error, it was a read or network glitch. If there is, the log shows “Data check: error confirmed”.
2. Make sure this server has a fresh copy in another storage: damage in one storage doesn't affect the others.
3. Write to the provider and attach the restic lines from the log. arcula doesn't repair the repository itself; you can examine it with plain restic, using the password and keys from the recovery kit.