Wednesday, 23 December 2020 10:45

Central Rsync backup monitoring with Go-lang and Lumen web API

Rate this item
(0 votes)

Rsync is a robust and reliable solution for Unix systems to quickly and reliably copy data to other devices (e.g. NAS systems). I have been using Rsync for years for my data backup from my Ubuntu server to a NAS system from QNAP, which in turn replicates my data to an external hard drive that then ends up offline in the closet. Rsync does this reliably and without problems every day and is started with a cronjob. The log files log everything down to the last detail - so far, no explicit data backup software was necessary for me.

Rsync Logfile Entry

The problem with multiple devices

In the past few months I have massively converted my home network. A Docker Host and 3 Raspberry Pi's have been added. Both the Docker host and one of the 3 raspberries want to be backed up. Of course, this can be done without problems with Rsync. However, it is cumbersome to monitor that the data backup works reliably. I could check myself regularly for the devices with SSH connections and the log files. Alternatively, I could also put the log files on a released order and check it. However, I would prefer a central monitoring of the data backups without having to read log files.

Current structure of my data backup

DIY data backup based on Rsync

True to the motto "Why simple when things get complicated", I have expanded my data backup process to include central monitoring. My thought was to program a simple web API with lumens (PHP), to which the most important information of the backup will be sent as soon as the data backup has been completed. The analysis of the log files makes for me a program that I wrote and compiled with Go. The Go program sends the host name, the job name (passed via parameters), the time stamp of the backup, and the number of files and new data to the Web API. A simple single page web application (programmed with VueJS) then shows me my data backup information. I integrate this web interface into Nextcloud as an external page and therefore have everything at a glance.

First draft - design is embellished;)

Rsync Backup Monitor

Install RSync Monitor yourself

The complete code of my solution is open source and available on GitHub. The easiest way is to use the finished Docker images to install the application yourself. However, what you have to do is compile the Go program yourself and adjust the target IP address of the Web API beforehand. In the next version I will have the Web API address passed as the start parameter. Likewise, you have to adjust the API address in the frontend via which each client can reach the API, this is usually the IP of your Docker host.

Instructions in English can be found in the readme file on my GitHub Repository.


This software lives from community support. I you think this is helpful or if you want me to improve this application, please star this application on github, or even better: Become a sponsor on Patreon or GitHub

Read 70 times Last modified on Wednesday, 23 December 2020 10:46

Leave a comment

Make sure you enter all the required information, indicated by an asterisk (*). HTML code is not allowed.