gem-hunter
Language: Python
Private, self-hosted Zotero WebDAV file sync over Tailscale or Headscale, plus zodav-audit: conformance, integrity and repair checks for any Zotero WebDAV server.
Private Zotero WebDAV sync, plus a conformance and integrity checker for any Zotero WebDAV server.
Quick start · zodav-audit · Compatibility · Troubleshooting · Changelog · Security · Contributing
ZoDAV is Zotero WebDAV: a private file-sync server for Zotero that is never on the public internet.
ZoDAV stores your Zotero attachment files (PDFs, snapshots, notes) on a server you control. The server is reachable only over a private Tailscale network (or your own Headscale). It has no public address and no open port. It replaces paid Zotero Storage for a personal library, and it is started with one .env file and one command.
This repository also contains zodav-audit, a checker that works against any Zotero WebDAV server (ZoDAV, Nextcloud, Synology, Koofr, pCloud, nginx, Caddy, rclone and others). Other projects will give you a WebDAV server. None of them check the things Zotero depends on. zodav-audit does:
.zip / .prop pair in the store consistent and readable?Zotero forum users who hit sync problems are usually told to "Reset File Sync History". zodav-audit tells you what is actually wrong.
Server:
mod_dav. Only the methods Zotero uses are allowed (OPTIONS, PROPFIND, GET, HEAD, PUT, DELETE, MKCOL)../zodav, for everything.Audit tool (zodav-audit):
getlastmodified in listings, a byte-exact .zip / .prop round trip, real 404s for missing files, and redirects.--json) and a single self-contained HTML report (--html) you can attach to a forum post.Zotero Desktop ──tailnet tcp/80──▶ [tailscale] ──bridge──▶ [webdav :8080] ── volume zodav-data
other containers on the host ──docker network "zodav"──▶ [webdav :8080]
[audit] ── zodav-audit loop (read-only data mount) ──▶ alerts (webhook/Telegram/email)
[backup] ── restic → off-host repository (read-only data mount, optional)
Everything is defined in one compose.yaml. There are four containers:
| Container | What it does |
|---|---|
webdav |
Apache httpd 2.4 with mod_dav and mod_dav_fs. /zotero/ is the only WebDAV location and needs a login. /healthz is the only page without a login. Everything else is denied. Runs as www-data on a read-only root filesystem. Passwords are hashed with bcrypt into a memory-only (tmpfs) file when the container starts. |
tailscale |
A Tailscale client sidecar in userspace mode. It is only a node that joins your existing tailnet or Headscale server; ZoDAV runs no coordination server, relay or exit node (no NET_ADMIN, no /dev/net/tun). It forwards tailnet TCP port 80 to webdav:8080. It uses Headscale when ZODAV_HEADSCALE_URL is set. |
audit |
Runs zodav-audit watch every 24 hours (by default) on the data volume, mounted read-only. Writes latest.json and latest.html reports, prints one summary line per run, and sends alerts. |
backup |
Optional. Runs restic every 24 hours against an off-host repository, with the data volume mounted read-only. Turned on by COMPOSE_PROFILES=backup. |
Your files live in the Docker named volume zodav-data, under zotero/. Each attachment is stored the way Zotero stores it: KEY.zip (the file) and KEY.prop (its modification time and MD5 checksum).
docker compose).ZoDAV serves one personal library. Group library files cannot use WebDAV in Zotero, so ZoDAV cannot store them.
These steps are for Tailscale. For Headscale see Using Headscale instead of Tailscale.
Sign up at https://login.tailscale.com/start. Free personal plans are enough.
ZoDAV joins your network with a tag (tag:zodav), so you can control exactly who may reach it. In the admin console open Access controls (https://login.tailscale.com/admin/acls) and add the two blocks below to the policy file. If the file already has tagOwners or grants, add the new entries to the existing ones.
{
"tagOwners": {
"tag:zodav": ["autogroup:admin"]
},
"grants": [
{
"src": ["autogroup:member"],
"dst": ["tag:zodav"],
"ip": ["tcp:80"]
}
]
}This lets every device that belongs to a user of your tailnet connect to ZoDAV on port 80, and nothing else. ZoDAV itself cannot start connections to your other devices. If your policy still contains the default "allow all" grant, the rule above does not narrow anything: remove the allow-all grant if you want ZoDAV limited to the rule.
In the admin console go to Settings > Keys (https://login.tailscale.com/admin/settings/keys) and choose Generate auth key:
tag:zodav. The key must be tagged, because ZoDAV asks for that tag when it joins.Copy the key (it looks like tskey-auth-xxxxx). It is shown once.
After ZoDAV first appears under Machines in the admin console (step 4), revoke or expire this key there. It is only used for the first login: ZoDAV stores its login in a Docker volume and does not use the key again.
Install Docker Engine and the Compose plugin by following the official guide: https://docs.docker.com/engine/install/. Then:
git clone https://github.com/ali-rajabpour/ZoDAV.git
cd ZoDAV
./zodav setup./zodav setup asks a few questions:
.env, see Monitoring and alerts),It writes a .env file readable only by you, with a generated password. It will not overwrite an existing .env.
Start ZoDAV:
./zodav start
The first start builds the images and takes a few minutes. When it finishes it prints the values to use in Zotero. To see them again later:
./zodav settings ./zodav settings --show-password
The computer that runs Zotero must be on the same tailnet. Install Tailscale on it (https://tailscale.com/download) and log in with the same account. The address printed by ZoDAV looks like zodav.your-tailnet.ts.net. It only resolves while your computer is connected to Tailscale.
In Zotero 7 or newer:
./zodav settings, for example zodav.your-tailnet.ts.net. Do not add http://, a port or /zotero. Zotero adds the zotero/ folder itself../zodav settings (default zotero)../zodav settings --show-password.zotero folder, accept.Why plain http is fine here: the traffic never crosses the internet unprotected. Tailscale (WireGuard) encrypts it from your computer to the server, end to end, inside the tailnet. A public HTTPS certificate would add nothing. Mobile apps are a different case, see the To-do.
If Verify Server fails, see Troubleshooting. To test the server from the outside, run ./zodav check.
Changing the file-sync method in Zotero does not move your files. ZoDAV starts empty.
./zodav check. When everything is clean and verified on every computer, then (and only then) remove files from the old service.Zotero Storage is separate from WebDAV and is not read by ZoDAV. Group library files stay in Zotero Storage.
If you run your own Headscale server, ZoDAV joins it as one more node. ZoDAV never runs a control server itself: its tailscale container is a client sidecar that logs in to your Headscale with a tagged pre-auth key and forwards mesh port 80 to the WebDAV container. The steps below assume Headscale 0.29 or newer with a grants policy.
Allow the tag in your policy. Add one entry to tagOwners and one grant. An empty owner list means no user can put the tag on a device themselves; only the administrator applies it, through a tagged key:
{
"tagOwners": {
"tag:zodav": []
},
"grants": [
{"src": ["you@"], "dst": ["tag:zodav"], "ip": ["tcp:80"]}
]
}Replace you@ with your Headscale user (or a group). Default is deny, so this is the only traffic allowed to ZoDAV, and ZoDAV cannot open connections to your devices. Validate with headscale policy check and load the policy the way your server does.
Create a single-use tagged key on the Headscale server:
headscale preauthkeys create --tags tag:zodav --expiration 1h
On Headscale 0.29 a tagged key needs no --user; tagged nodes belong to no user and never expire. Older versions may need --user; see headscale preauthkeys create --help.
Set two variables in .env (or your hosting panel), then start ZoDAV within the key's lifetime:
ZODAV_HEADSCALE_URL=https://headscale.example.com TS_AUTHKEY=hskey-auth-xxxxx
With Headscale, ZODAV_TS_TAG is ignored: ZoDAV does not advertise tags itself (with an empty tagOwners that would be refused), the tag comes from the key.
After the first start, headscale nodes list shows zodav with tag:zodav. The login is stored in the tailscale-state volume, so you can clear TS_AUTHKEY. Only if you delete that volume does ZoDAV need a new key.
Connect Zotero using the MagicDNS name, zodav.<your MagicDNS base domain> (for example zodav.mesh.internal), with protocol http. MagicDNS must be on in Headscale and accepted on your computer. ./zodav settings prints the exact name.
compose.yaml works unchanged as a compose application.
compose.yaml and keep the webdav/, audit/ and backup/ folders next to it, because the images are built from them).compose.yaml..env.example. At minimum set ZODAV_PASSWORD, and TS_AUTHKEY for the first start (it can be cleared once ZoDAV has joined your network). Replace every CHANGE_ME value.tailscale container logs) and use it as the Zotero URL, with protocol http.The panel writes your settings, passwords included, to a .env file next to the compose file, readable by every account on the host (mode 644), and rewrites it on each deploy. Lock down the application's folder once, on the host, as root. The panel runs as root, so deploys keep working, and the setting survives redeploys because only the code folder inside it is recreated. For Dokploy:
chmod 700 /etc/dokploy/compose/<app-name>
The ./zodav script is optional on these platforms. Everything it does can be done from the panel, as described next.
Automatic. The audit container checks every stored file once a day (interval: ZODAV_AUDIT_INTERVAL_HOURS) and sends alerts to the channels you configured. In the panel, open the audit service logs: each run prints one line such as zodav-audit watch: 152 attachments, 0 error(s), 0 warning(s) - OK. To prove alerts arrive, set ZODAV_IDLE_DAYS=0 and redeploy: every run then raises the idle warning. Set it back afterwards.
On demand, in the panel's terminal. Open a terminal (Dokploy: service, then Terminal) in the audit container. The password is already in its environment, so no flags are needed:
# Is the server Zotero-compatible? Writes only Zotero's own test file and deletes it. python /app/zodav_audit.py conformance http://webdav:8080/ # Check every stored attachment now. python /app/zodav_audit.py integrity /data/zotero # Show what repair would do. Changes nothing. python /app/zodav_audit.py repair http://webdav:8080/
In a terminal in the backup container (only when backups are enabled):
restic snapshots # list of backups, newest last restic ls latest | grep ABCD1234 # is attachment ABCD1234 in the latest backup? restic check # consistency of the backup repository
From your computer, over the tailnet. This tests exactly the path Zotero uses and writes HTML reports you can open:
export ZODAV_AUDIT_PASSWORD='your ZODAV_PASSWORD' uvx --from git+https://github.com/ali-rajabpour/ZoDAV zodav-audit \ conformance http://zodav.your-tailnet.ts.net/ --html conformance.html uvx --from git+https://github.com/ali-rajabpour/ZoDAV zodav-audit \ integrity http://zodav.your-tailnet.ts.net/ --html integrity.html
Restore drill. It needs Docker on the host, so run it over SSH in the folder where the platform checked out this repository. Platforms name the compose project after the application, so pass that name (it is the prefix of the container names in docker ps), and make sure the same variables are available (most platforms write them to .env in that folder):
COMPOSE_PROJECT_NAME=<application-name> sh backup/restore-drill.sh compose.yaml
Run it about once a month. It prints restore drill ok when the latest backup restores and checks clean.
Zotero shows no error. The sync button (top right) has no red error mark. Hover over it to see file progress.
The file is on the server. In Zotero, right-click a PDF and choose Show File. The name of the folder that holds it is the attachment key, for example ABCD1234. Then, from a device on the tailnet:
curl -u zotero -I http://zodav.your-tailnet.ts.net/zotero/ABCD1234.zip curl -u zotero -I http://zodav.your-tailnet.ts.net/zotero/ABCD1234.prop
HTTP/1.1 200 OK for both means the file is stored. 404 means it has not been uploaded yet.
Counts grow. The attachments number in the audit log line and report grows as you add files. It counts stored files only, not links or notes.
Download it back (the definitive test). Either open the PDF on a second computer that uses the same Zotero account, the same WebDAV settings and is on the tailnet, or on the same computer: add a test item with a PDF, sync, delete only the file inside ~/Zotero/storage/<KEY>/ in your file manager (not the item in Zotero), then double-click the item. Zotero downloads it again from ZoDAV.
The backup holds it. After the next nightly backup, restic ls latest | grep <KEY> in the backup container shows the file.
Other containers on the same server can read and write the store without going through Tailscale. The webdav container is on a Docker network named zodav.
Give the automation its own login in .env (this is the optional second user):
ZODAV_SERVICE_USERNAME=paperbot ZODAV_SERVICE_PASSWORD=at-least-24-letters-digits-dash-underscore
The service password must also be at least 24 characters of A-Z a-z 0-9 _ -, and the name must differ from ZODAV_USERNAME. Restart with ./zodav start to apply it.
In the other application's compose file, join the existing network:
services: myapp: image: example/myapp networks: [zodav] networks: zodav: external: true
Use http://webdav:8080/zotero/ as the WebDAV address, with the service user name and password.
Separate users mean you can rotate either password on its own, and the access log shows which user wrote what. The service user has the same read and write access as the main user. Files must follow Zotero's layout (KEY.zip and KEY.prop) to be used by Zotero, see zodav-audit to check them.
The data volume is on the same machine as the server. If the disk dies, the server is stolen or someone deletes the files, the volume goes with it. A sync service is not a backup: Zotero treats a deleted attachment as deleted everywhere. Backups must go to a different place.
ZoDAV uses restic: encrypted, deduplicated, with many storage back ends.
Pick one place to send backups to and set RESTIC_REPOSITORY plus the credentials that place needs. Examples:
Amazon S3 or an S3-compatible service:
RESTIC_REPOSITORY=s3:s3.amazonaws.com/your-bucket AWS_ACCESS_KEY_ID=xxxxx AWS_SECRET_ACCESS_KEY=xxxxx AWS_DEFAULT_REGION=eu-west-1
Backblaze B2:
RESTIC_REPOSITORY=b2:your-bucket:zodav B2_ACCOUNT_ID=xxxxx B2_ACCOUNT_KEY=xxxxx
restic REST server:
RESTIC_REPOSITORY=rest:https://backup.example.com/zodav RESTIC_REST_USERNAME=xxxxx RESTIC_REST_PASSWORD=xxxxx
SFTP: the compose file does not pass SSH keys or a known-hosts file into the backup container, so sftp: repositories do not work with the provided compose.yaml. Use one of the repository types above. Azure Blob Storage (azure:) is also wired up, see .env.example.
The backup container refuses a RESTIC_PASSWORD shorter than 24 characters or still starting with CHANGE_ME, and it skips a run (without counting it as a success) while the store holds no attachments, so an empty store never replaces good snapshots.
Easiest: answer the backup question in ./zodav setup. It sets COMPOSE_PROFILES=backup, your repository, and a generated RESTIC_PASSWORD. Add the provider credentials to .env by hand.
By hand, add to .env:
COMPOSE_PROFILES=backup RESTIC_PASSWORD=a-long-random-password RESTIC_REPOSITORY=s3:s3.amazonaws.com/your-bucket
Then:
./zodav start ./zodav logs backup
You should see backup ok. The first run creates the repository.
Save RESTIC_PASSWORD in a password manager now. It encrypts the backups. If you lose it, the backups cannot be read by anyone, including you.
restic check --read-data-subset=5% re-reads a random 5 percent of the stored data to catch damage.unhealthy and the audit sends an alert (BACKUP_STALE). If backups were never configured, the audit warns (BACKUP_NEVER).BACKUP FAILED: ... and retried at the next interval.Prove that the backup can be restored, without touching your live data:
./zodav restore-drill
It restores the latest snapshot into a temporary Docker volume, runs the integrity check on it, prints restore drill ok or restore drill FAILED: ..., and removes the volume. Run it after setting up backups and from time to time after that.
Install Docker and clone this repository on the new server.
Create .env with ./zodav setup, then edit it. Use the same RESTIC_REPOSITORY, provider credentials and RESTIC_PASSWORD as before. A new ZODAV_PASSWORD is fine, but you will have to enter it in Zotero again. COMPOSE_PROFILES=backup must be set.
Start once so that the data volume exists, then stop the services that use it:
./zodav start docker compose stop webdav audit
Restore the latest snapshot into a scratch volume:
docker volume create zodav-restore docker run --rm -v zodav-restore:/restore alpine chown 65534:65534 /restore docker compose run --rm --no-deps -T -v zodav-restore:/restore \ --entrypoint restic backup restore latest --host zodav --target /restore
Copy the files into the live data volume and fix the owner (the web server runs as user 33):
docker run --rm -v zodav-restore:/from:ro -v zodav_zodav-data:/to alpine \
sh -c 'cp -a /from/data/zotero/. /to/zotero/ && chown -R 33:33 /to'Start everything again and check:
docker compose start webdav audit ./zodav check docker volume rm zodav-restore
If the new server has a new Tailscale name, update the URL in Zotero (./zodav settings).
Run the restore drill first so you know the repository and password are good before you need them.
The audit container runs zodav-audit watch on the data volume. By default it runs right after start and then every 24 hours (ZODAV_AUDIT_INTERVAL_HOURS). Each run does all integrity checks and these extra checks:
| Check | Raises | When | Default |
|---|---|---|---|
| Mass deletion | error MASS_DELETION |
The number of attachments fell by more than ZODAV_DELETE_ALERT_PERCENT percent and by at least 5 since the previous run. |
10 percent |
| Idle store | warning STORE_IDLE |
The newest .prop is older than ZODAV_IDLE_DAYS days: no device has uploaded anything. |
30 days |
| Free space | error LOW_DISK |
Free space on the data volume is below ZODAV_MIN_FREE_GB GB, or below 5 percent. |
2 GB |
| Backup age | error BACKUP_STALE / warning BACKUP_NEVER |
The last successful backup is older than 26 hours, or there has never been one. | fixed |
If there is any error or warning and at least one alert channel is set, one message is sent to every configured channel. A channel that fails is logged (reason only, no secrets) and the others still get the message. When the next run is clean after a failing one, a single "ZoDAV recovered" message is sent. If a run itself cannot complete, you get "ZoDAV audit could not run".
You can use any combination of the three channels. Set them in .env.
Webhook (ZODAV_ALERT_URL):
# ntfy: any plain URL gets the message as the body, with a Title header ZODAV_ALERT_URL=https://ntfy.sh/your-secret-topic # Slack incoming webhook (JSON {"text": ...}) ZODAV_ALERT_URL=https://hooks.slack.com/services/T000/B000/XXXX # Discord webhook (JSON {"content": ...}) ZODAV_ALERT_URL=https://discord.com/api/webhooks/0000/XXXX
Slack and Discord are recognised by their host name. Any other URL receives a plain-text POST. Treat the URL as a secret: anyone who has it can post to it.
Telegram (set both or neither):
ZODAV_TELEGRAM_BOT_TOKEN=123456789:AAExampleTokenFromBotFather ZODAV_TELEGRAM_CHAT_ID=123456789
Create the bot by chatting with @BotFather and sending /newbot. Then send your new bot any message, open https://api.telegram.org/bot<TOKEN>/getUpdates in a browser and read the number after "chat":{"id":. Messages are plain text, cut to Telegram's 4096-character limit.
Email (setting ZODAV_SMTP_HOST turns it on, ZODAV_SMTP_FROM and ZODAV_SMTP_TO are then required):
# Gmail or Fastmail: use an app password, not your normal password
ZODAV_SMTP_HOST=smtp.gmail.com
ZODAV_SMTP_PORT=587
ZODAV_SMTP_SECURITY=starttls
ZODAV_SMTP_USERNAME=you@example.com
ZODAV_SMTP_PASSWORD=your-app-password
ZODAV_SMTP_FROM=you@example.com
ZODAV_SMTP_TO=you@example.com,partner@example.comThe subject is ZoDAV: <title>. ZODAV_SMTP_SECURITY is starttls (default), ssl (usually port 465) or none; none sends unencrypted and logs a warning. Certificates are always verified. Username and password are set together or not at all. An incomplete setting (a token without a chat id, a host without FROM or TO) stops the audit with an error instead of silently sending nothing.
Reports: each run writes latest.json and latest.html into the reports volume and prints one line to the container log. Get the HTML report with:
./zodav report
It saves zodav-report-<date>.html in the current folder.
After a MASS_DELETION alert: open the report and check what disappeared. If files were removed by mistake, restore them from a backup (see Backups); the alert stays until the count recovers. If you deleted the files on purpose, run:
./zodav accept-deletions
This records today's attachment count as the new normal, so the next check does not alert about the same deletion.
Run ./zodav help for this list. Docker must be running, except for setup and help.
| Command | What it does |
|---|---|
./zodav setup |
Asks for the auth key (required), a Headscale URL, an alert URL and a backup repository (all optional). Writes .env from .env.example with a generated 32-character ZODAV_PASSWORD, mode 600. If you give a backup repository it also sets COMPOSE_PROFILES=backup and generates RESTIC_PASSWORD. Refuses to overwrite an existing .env. |
./zodav start |
Builds and starts everything, waits until the WebDAV server and the audit container are healthy (not for the first backup), makes .env private if it was readable by others, then prints the Zotero settings. |
./zodav stop |
Runs docker compose down. Containers stop; your files and settings are kept. |
./zodav status |
Shows docker compose ps. |
./zodav settings [--show-password] |
Prints protocol, URL (the tailnet name of the server), username and, with the flag, the password. |
./zodav check |
Runs zodav-audit conformance against the server and integrity on all stored files, now. Exits 1 if either finds an error. |
./zodav report |
Copies the latest HTML audit report to zodav-report-<date>.html in the current folder. |
./zodav accept-deletions |
Tells the audit that the current number of attachments is the new normal. Use it after deleting files on purpose, see Monitoring and alerts. |
./zodav repair [--apply] |
Runs zodav-audit repair on the store. Dry run without --apply. Quarantined files are written to zodav-quarantine-<timestamp>/ in the current folder. |
./zodav restore-drill |
Restores the latest backup into a scratch volume and checks it. See Backups. |
./zodav logs [service] |
Shows the last 100 log lines. Services: webdav, tailscale, audit, backup. |
./zodav help |
Shows the command list. |
All settings are environment variables, set in .env or in your hosting panel. .env.example documents each one.
| Name | Required | Default | Meaning |
|---|---|---|---|
ZODAV_PASSWORD |
yes | none | Password for the Zotero user. 24 to any length, only A-Z a-z 0-9 _ -. ./zodav setup generates one. |
TS_AUTHKEY |
first start | none | Tailscale auth key (tagged tag:zodav) or tagged Headscale pre-auth key. Only used for the first login; may be cleared once ZoDAV has joined. |
ZODAV_HEADSCALE_URL |
no | empty | URL of your Headscale server. Empty means Tailscale. |
ZODAV_HOSTNAME |
no | zodav |
Machine name on the tailnet. The address is <name>.<your-tailnet>. |
ZODAV_TS_TAG |
no | tag:zodav |
Tag requested when joining Tailscale. Ignored with Headscale. |
ZODAV_USERNAME |
no | zotero |
Zotero user name. Starts with a lowercase letter; a-z 0-9 _ -; up to 32 characters. |
ZODAV_SERVICE_USERNAME |
no | empty | Login for a second, automated user. Needs ZODAV_SERVICE_PASSWORD. Must differ from ZODAV_USERNAME. |
ZODAV_SERVICE_PASSWORD |
no | empty | Password for the service user. Same rules as ZODAV_PASSWORD. |
ZODAV_ALERT_URL |
no | empty | Webhook for alerts (ntfy, Slack, Discord, other). Empty means no webhook. |
ZODAV_TELEGRAM_BOT_TOKEN |
with Telegram | empty | Bot token from @BotFather. Needs ZODAV_TELEGRAM_CHAT_ID. |
ZODAV_TELEGRAM_CHAT_ID |
with Telegram | empty | Chat that receives alerts. Needs the bot token. |
ZODAV_SMTP_HOST |
with email | empty | SMTP server. Setting it turns email alerts on. |
ZODAV_SMTP_PORT |
no | 587 |
SMTP port. |
ZODAV_SMTP_SECURITY |
no | starttls |
starttls, ssl or none (unencrypted, logs a warning). |
ZODAV_SMTP_USERNAME / ZODAV_SMTP_PASSWORD |
no | empty | SMTP login. Set both or neither. Use an app |
| |||
| |||
| | ||
| | ||
| | ||
| | ||
| | ||
| |||
| |||
| |||
| |||
| |||
|
| |
| |
| |
| |
| | | |
| | | |
| | ||
| | | |
| | | |
| |||
| |||
| | ||
| |||
| | ||
| | ||
| |||
| | | |
| |
| |||
| |||
| | ||
| | | |
| | | |
| | ||
| |||
| |||
| | | |
| | | |
| |||
| | ||
| | | |
| |||
| |||
| |||
| |||
| | ||
|
| | | |
| | | |
| | ||
| |||
| |
| |||
| ||
| | |
| | |
| ||
| ||
| ||
| ||
| ||
| ||
| ||
| | |
| | |
| | |
| |