Storage statuses

The storage status shows in the “Storages” list, in the “Status” column, and on the storage page under its name. The panel checks storages on its own: access every 6 hours, space every 45 minutes, data every night from 03:00 to 06:00 in the panel's time zone. How the checks work — Storage checks.

Levels

A storage with a problem comes first in the list. If the space before the limit will last less than 30 days, “Lasts for” is highlighted and the storage also moves up, but the status doesn't change because of it — Space.

No access

In the panel: “No access: 403 AccessDenied since 12:02”. Status: “Critical”.

The panel's access check failed: list, read or write in the test folder .arcula-check/ returned an error. The code is the first S3 error response, as is. “Since 12:02” is when the checks started failing.

While there is no access:

• copies of all servers here show “no access” and don't count toward the “Latest copy”;
• data checks and cleanup don't run;
• space isn't measured; “Lasts for” shows “—”.

Agents keep sending backups on their own, with their key from the plan. Their errors are in the log, in the “Backup not saved” lines.

When it appears: after a failed check — every 6 hours, after a key change, and when you click “Check access”.

What to do — by code:

A new key is set on the “Connection” tab. The panel checks it first, and the old one stops being used only after a successful check. The address and bucket can't be changed in the panel: if the bucket is different, connect a new storage and use it instead of the old one in the server plans — Connect a storage.

Once it's fixed, click “Check access”.

Goes away on its own: after a successful check — within 6 hours, or right away with the button.

Little space left before the limit

In the panel: “18 GB left before the limit — 95% used”. Status: “Needs attention”.

The limit is a space budget you set for the storage in the panel, not the provider's quota. By default only arcula data counts. If the limit is counted over the whole bucket, data that isn't ours counts too.

When it appears: 90 % of the limit or more is used. Space is measured every 45 minutes.

arcula doesn't stop backups at the limit. If the provider has its own quota, backups stop being saved once it runs out.

What to do:

• Turn on cleanup or shorten the retention rules. The preview shows what will be deleted and how much space will be freed — Cleanup.
• “Clean up now” on the “Cleanup” tab — deletes by the rules and frees space without waiting for the night.
• If the provider has space, raise the limit: the “Connection” tab, the “Limit” row.

Goes away on its own: yes, when used space drops below 90 % of the limit. After cleanup the panel measures space again right away.

Not saved in a row

In the panel: “Not saved in a row: prod-db-01 — 36 backups”. Status: “Needs attention”.

For these servers, backups after the last one saved here didn't arrive: they failed to upload, or were skipped because a newer one was ready during the upload. The number is how many in a row.

When it appears: at least one backup of a server after the last saved one isn't saved here. While the storage has no access, the line isn't shown — the cause is visible anyway.

What to do:

• Open the server → the “Storages” tab: the copy in this storage shows the cause.
• In the storage log, in the “Backup not saved” lines — the storage's answer and the restic lines. What each error means — Copy errors.
• One or two skips during a long upload are normal: the storage gets the latest ready backup.

Goes away on its own: yes, when the server's next backup is saved here.

Data check found damage

In the panel: “Data check: 1 damaged data file in the registry-02 repository — pack 7c1e…: ciphertext verification failed”. Status: “Needs attention”.

The nightly check reads part of the data of each repository (restic check --read-data-subset) and verifies encryption and checksums. It found a file that can't be read or doesn't match. After the dash is the restic line as is.

When it appears: right after the check that found the error. The check cycle doesn't move on: the next night the same part is read again.

What to do:

1. Click “Check data” on the storage page: the same part of the damaged repository is read once more. The error may have been a one-off.
2. The log shows “Data check: error confirmed” — the data in this storage is damaged. Restore this server from another storage.
3. Tell the storage provider.
4. Don't put it off: restic doesn't send again what it considers already saved, so new backups may depend on the damaged file too.

arcula doesn't repair the repository itself. You can repair it manually with the restic repair commands; the repository password and keys are in the recovery kit.

Goes away on its own: yes, when this part is read without errors — at night or with the button.

Same account specified

In the panel: “Same account as Primary”. Doesn't change the status.

Two storages have the same label in the “Account” field. You enter it yourself when connecting: most providers don't expose the account number over S3, so arcula compares only the labels.

What this means: if the account is blocked, hacked or stops being paid for, both storages are lost at once. Such copies aren't independent — in the server plan this shows in the “Where to store” card: “independent copies: 1”.

What to do: if this is intended — nothing. For a real second copy, connect a storage at another provider or in another account — Storages.

Goes away on its own: no. This is a fact about the setup, not a failure.

The bucket has data that isn't ours

In the panel: “The bucket has data that isn't ours: 1.2 TB — arcula doesn't touch it”. Status stays “OK”.

The same bucket holds something besides arcula data: files, old backups, other restic repositories, the arcula/ directory of another panel. Ours are only this panel's server folders arcula/<server id>/ and the test folder .arcula-check/. arcula never writes, changes or deletes anything else — it only counts its size.

When it appears: when the storage is connected, and after a space measurement if something that isn't ours is found.

What to do: usually nothing. By default the limit counts only arcula data; you can count the whole bucket too — this is chosen for the limit. Connecting a found old restic repository to view it in arcula isn't possible yet.

Goes away on its own: yes, if no data that isn't ours is left in the bucket.