22.04.2025 – Pavel Klavík, Kamila Klavíková
It’s important to keep your data safe. That’s why we recently tested our backups by rebuilding OrgPad from scratch. In this post, we explain how data is stored and how we handle backups. You’ll also learn how long recovery would take in case of disaster. And we share our future plans to make OrgPad’s infrastructure and backup system even better.
#security, #backups, #OrgPad, #data, #infrastructure
OrgPad is a productivity tool you can rely on. Our goal is to interrupt you as little as possible, so you can focus on solving your own problems. That’s why we automate many things. Creating a new cell makes space between others. Cell size adapts to its content. When pasting content from other tools, we keep only the important formatting.
We treat your data with the same care. That’s why we back it up regularly and thoroughly. In this post, we explain how it all works — what tools we use and how we test backups. We’ll also share our future plans.
For safety, we use three different providers (Hetzner Online GmbH, Contabo, and Time4VPS) in four locations within the EU. Your data is available in four different places. Any data sent outside the main server is encrypted. So even if it leaked, no one could access it. If someone gained access to the main server, data are protected, so they couldn’t delete backups on the others.
BorgBackup, or just Borg, is a tool for creating regular backups. It lets us keep many backups at once. For example, we can make a backup every day. We store all backups from the past month, so we can go back to any point in time. The key feature is deduplication. Shared parts of files are stored only once. Files are split into blocks. Identical blocks are saved once only. We also compress them using zstd, which saves disk space.
Borg can encrypt backups using a password. In OrgPad, we use long random passwords. Here is an example:
wMk4FC84Hl3UwGQNYkBh1wuOABq7dVMrubLHiOXgH18SggLuViThe 256-bit AES encryption together with a long password makes it impossible to read the backup without the password. Even if someone got the encrypted file, they couldn’t use it.
Rsync copies data from one server to another. Only changed files are transferred. The first sync takes longer, but later ones are much faster.
Rsnapshot is a simpler backup tool than Borg. It also stores multiple backups. But it uses a simpler method for deduplication. Only identical files are shared. It uses rsync and Linux hard links to do this.
Each backup is stored in a separate folder. When a new backup is created, the oldest one is deleted, and the others shift. The last backup is copied using links, so unchanged files are shared. Then rsync adds the changed files. So files that didn’t change don’t take up extra space.
All OrgPad data is stored on servers in the EU. These follow strict privacy and security rules. You can see an overview of our backup setup in this infrastructure diagram.
The main infrastructure runs on servers in Germany, provided by Hetzner Online GmbH. OrgPad and all data run on a single server located in a datacenter in Falkenstein (DE). The data is stored in a PostgreSQL database. This includes tables for documents, their cells and links, users, and more. Uploaded files and images are stored in a Minio file database.
Running OrgPad from one server works surprisingly well and reliably. Thanks to improvements in operating systems and virtualization, OrgPad can run smoothly without interruption. Hetzner can move a running server to a different physical machine. It copies memory data into almost the same state, then pauses the old server for a few seconds to finish copying. After that, the live migration is complete and the server continues on the new machine.
In the years we’ve been running OrgPad, we’ve only had a few short outages. Most of them happen when we upload a new version and restart OrgPad. That takes about one or two minutes. We try to upload new versions outside peak hours. Some updates we can even apply without restarting the app.
The main server backs up the PostgreSQL database every 30 minutes using Borg. We extract a full text dump of the database using pgdump and store it in a Borg backup repository. These backups are encrypted with a long random password and kept for one month.
Of course, your data isn’t stored on just one server. Otherwise it could be lost due to a disk failure or a data center fire. We also run a second backup server with Hetzner, in a different datacenter in Nuremberg (DE).
This server connects to the main server every four hours and copies the backups using rsnapshot. It copies the encrypted PostgreSQL backups made by Borg, and unencrypted files and images stored in Minio. These backups are kept for six days.
We use rsnapshot instead of Borg because it can download backups efficiently. The backup server connects to the main server and copies only changed data. The main server doesn’t have access to the backup server. So if an attacker got into the main server and deleted or encrypted data, they wouldn’t be able to reach or damage the backups.
It’s not a good idea to store all backups with one provider. The whole account could be deleted by mistake. That actually happened in the past, more details in this video. That’s why we store additional backups outside Hetzner, with two other providers. These are called disaster recovery backups.
The first server is with Contabo in Düsseldorf (DE). The Hetzner backup server sends Borg backups there, again encrypted with a new long random password. We store PostgreSQL backups (double-encrypted) and Minio files (encrypted once). Important detail: the data is already encrypted when sent, so Contabo can’t access it. Borg keeps backups for one month. So for PostgreSQL, we have two months of backups total. For uploaded files and images, we keep one month.
We also store these backups on another server, hosted by Time4VPS in Vilnius (LT). This server regularly connects to Contabo and copies the full Borg repository using rsync.
Having many backups is useless if they’re not tested. Many companies found during a disaster that their backups were broken or incomplete.
At OrgPad, we monitor that backups are happening. Backup info is shown at the top of our internal status dashboard, which we check several times a day. If a backup fails, a red box shows up. Missing backups show as missing boxes. For example, when Contabo had a short technical outage, we quickly noticed backups weren’t working and contacted their support. We also track disk space and increase it when needed.
Here’s how we test backups. We created a new server at Hetzner and launched OrgPad on it. We automated the process with Ansible. The scripts set up servers, secure them, install OrgPad and the databases, create certificates, and more. The whole setup takes only a few minutes.
Then we copied the full encrypted Borg repository from Contabo and restored the data. This also verified that older backups were usable. The total size was around 600 GB. Copying took almost 5 hours. This was the slowest part. We also tried restoring over the network, but that was slower.
Next, we restored the encrypted Borg storage containing the PostgreSQL backups. That took 15 minutes. Uploading the latest backup into PostgreSQL took 30 more. After that, we could run OrgPad without the uploaded files and images. Restoring the uploaded files took another two and a half hours.
In total, the restore process took around 8 hours. We automated it with Ansible. We also validated the restored data, including checking all uploaded files and images. As OrgPad grows, restore time increases slightly, but not much. If we restored from Hetzner backups instead, which is more likely, the restore would be much faster.
We have many ideas to improve backups in OrgPad, like with everything else. Since we’re a small team, we have to carefully choose what to focus on.
For backups, we want to set up a secondary Hetzner server with PostgreSQL and Minio installed. We’d create a read-only replica of the main server’s databases. That means every change would be sent over the network and saved on the secondary server. This would bring these benefits:
We’ll update this text as we keep improving OrgPad backups. If you want updates on OrgPad and how we handle technical issues, follow us on social media.
