Interface the HS100 with everything

I couldn’t resist to imagine many potential applications and possible integrations with online (cloud) services that are publicly and freely available, like IFTTT; I’m also a big fan of Android Tasker, Node-Red, the MQTT pub/sub protocol, and Raspberry Pi / NodeMCU hooked with sensors and hardware pieces attached on the GPIO pins) that make more connected things, so literally ideas are all over the places in my heads, on papers, in Keep notes…

Like this one that just popped in my mind:

Switch off the main lights in the salon, dim the lights around the sofa and TV, power on the media server box (smartplug) and TV when I sit on the sofa, with a NodeMCU controller and a pressure sensor (FSR)! Or maybe I’d prefer to have all that triggered when I power on the TV, but only in a certain time slot?…

Told you I have crasy ideas ;). Totally got to try to do that BTW!

Error: Time not found

What I only lack really is time. First, time to try and make some of those idea work (they do eventually with perseverence and I always learn a lot in the process), then time to write about them (here).

I didn’t realize the time it take to write articles people need to write good articles. I usually take notes of what I do, notes that I understand, but publishing those is no value to anyone. You need to structure the ideas, give them some form so the result make sense for the reader, enough at least so they can be inspired in trying also themselves to do, learn and make something that work for them out of what I wrote. Unfortunately due to the lack of time, I don’t enter into too much details, and I assume the reader is tech-savvy enough to read, understand and try by himself. That means some reader gets frustrated because I missed some parts they didn’t know… Sorry about that.

Wins so far

Back to the HS100: there were some (very good) resources publicly available already out there that showed how to interact with an HS100 plug, from shell scripts based on cURL, php and python libraries or Node.js modules, even a Node-Red. But they all had the same problem: they only worked withing the same local network (WiFi/LAN) the plug was connected to. Ugh bummer.

The Kasa App and the Alexa integration were proof that TP-Link was hosting a cloud service, but no API was publicly available yet.

So I started to work on figuring out how the App talked to the TP-Link Cloud endpoint, and started to publish it here. The series got (modestly) popular among the maker and tinkerers ;), and people used the comments to exchange ideas, issues, and come with solutions. So far, I’ve published (not in chronological order)

• Retrieve token and deviceID from Kasa App on Android (this is how everything started)
• Login to TP-Link (with your username and password) and retrieve token
  • also …with Tasker
• Retrieve the cloud end point URL and list of your TP-Link devices (with deviceID)
• Set the plug state (on/off) remotely (once you have Token, DeviceID, and URL)
  • with cURL or Hurl.it (command line, shell scripts,…)
  • with IFTT
  • with Tasker
  • from your Watch

So far the whole process was rather hacky and manual I reckon, but still one could achieve the goal of switching a smartplug from a remote place (outside of the smartplug local network), like from one’s smartphone (Tasker), or an IFTTT recipe. It already enabled some cool logics.

Pitfalls: what about scenes?

Soon after having found out how to trigger one TP-Link HS100 smartplug, the most frequently asked question in the comments has been ¿how can I power On/Off multiple devices from ITFFF?

After all it’s a feature that people know from the Kasa App: you define buttons in the app that will put several of you smartplugs in a state you define.

Related to my idea above, a scene would be:

“Time to watch some TV” scene: Switch off the main lights in the salon, dim the lights around the sofa and TV, power on the media server box (smartplug) and TV.

You can define that in Kasa, but what about calling it from IFTTT? You can’t. You can only call one POST call (via Maker), in an IFTTT recipe.

The second to most request in the mind of all is probably related to the TOKEN that will eventually expire sooner of later, making it a hassle to edit all your IFTTT Recipes/Tasker tasks/whatever scripts… manually!

Well, I’m happy to say this was until now!

More Wins coming soon…

This week I’ve been working hard on this (it’s the second time I try it, and this time I made it!).

This is what I have done:

• I’ve created a node.js module to easily interact (power On/Off, get state) with any number of HS100/HS110 smartplugs and LB100/LB110/LB120 and LB130 smartbulbs (set on/off, set brightness, set color). The module handles the login, token retrieval, and device triggering, in almost no call.
• I’ve created a Glitch example app in which one can define scenes. As Glitch publishes a URL, you can then trigger one “scene” with only one call (http://url.glich.me/scene1/) and with only one POST call it will do whatever you want to! Not only that but you can even define as many scenes as you want in a single Glitch app ( each with a different end-point: http://url.glich.me/scene2/, http://url.glich.me/scene3/…).
• Next I will create a library of new TP-Link Node-Red nodes to control the smartthings from Node-Red!

All that is already working fine for me, so this is what I’ll talk about in next posts (and I’ll publish the code for everything ) when I get some more time to talk about it. (Believe it or not, it took me more than two hours to write this post)!

So, stay tuned if you are interested. I suggest you follow me on Twitter, and follow this blog, recommend on Disqus (the heart icon).

Ah and well… all this is free, but you might see some interesting (or not) Ads around my content. I won’t mind if you all start clicking everywhere. I’m not suggesting anything… :p

In IFTTT we’ll use the Maker service, which let’s us call a custom HTTP GET/POST request with some Json payload, so that’s exactly what we need here.

Note: If you haven’t already, follow the steps in my previous post, to figure out what your Token and Device ID are.

Then create a new Recipe Applet like you normally would in IFTTT, and choose the “Maker Webhook / Make a Web Request” action service to define a new action.

For the URL, I’ve used:

https://eu-wap.tplinkcloud.com/?token=YOUR_TOKEN_HERE

and the method type is POST.

Note: For the URL: please read this other post, as you might need to use a different URL depending the region where you are from.

The Content Type is application/json, and the payload to switch the plug On should be:

{"method":"passthrough",
"params": {"deviceId": "YOUR_DEVICEID_HERE",
"requestData": "{\"system\":{\"set_relay_state\":{\"state\":1}
}
}" }
}

If you want to switch the plug Off, just replace the state 1 with state 0.

Notice how I have added extra carriage returns to separate the “}” at the end, otherwise IFTTT was giving me some error.

That’s it! Easy, right?

Scalability is the capability of a system, network, or process to handle a growing amount of work, or its potential to be enlarged in order to accommodate that growth.

Looking at scalability is very relevant in Cloud environment, which provide a high level of on demand elasticity and thus allow us to easily implement the scalability patterns most relevant to our applications to cater for our particular business needs, whichever they may be.

App Service Plan Provisioning

Let first have a look at what happen when you provision a new App Service Plan in Azure.

When you request the creation of an Azure Web App onto a new App Service Plan (aka. Serverfarm), Azure is not provisioning a brand new VM from scratch. Instead, I believe, they assign you a recycled VM of the chosen size, for example B1, from a pool of existing VM.

Indeed you can see that by looking at the uptime of the provisioned VM instance (use Kudu advanced tools): uptime is quite random, and not near 0. A near 0 uptime would indicate a recently created VM. So, in my understanding, Azure keeps a pool of VMs just ready for when a new customer is requesting a new service plan, or when a scaling operation is needed.

That make a lot of sense. Provisioning a new VM each time would take quite a lot of time, while a new Web App deployment actually happens in about 15 to 20 seconds. For that purpose, I assume they will always keep a certain number of VMs ready in the pool, enough to respond to customer’s needs at all time in a timely manner.

Horizontal scalability (Scale out / Scale in)

Horizontal scalability is when you change the number of VM instances supporting your App Service Plan. When load on your apps grows, you will likely scale-out (add more VM instances). When the loads decrease, you will scale back in, reducing the number of VM instances.

As I understand it, when scaling-out, an operation similar to the provisioning will happen: Azure will assign you one new VM of the same size, from its pool of available VMs:

When the plan scales back in again, the VM is removed from the Plan and placed back in the correspondin pool. As I understand it, it is recycled so it can be reused again, possibly by another customer.

Vertical scalability (Scale up/down)

Vertical scalability is about changing the App Service Plan instance size. If you need more compute power to run you apps, you can choose to scale-up your plan to bigger VM(s). What happen in that case though?

AFAIK, Azure runs on Hyper-V virtualization platform, and at the time of this writing, Hyper-V doesn’t allow for dynamic CPU/RAM resizing of VMs. So how do they do it?

Well, I did the test on a Plan with a single small instance. I noticed the instance hostname and uptime before scaling up. I then scaled the plan up to medium, and guess what: the hostname and uptime of the only instance in the plan was totally different!

Here’s my educated guess: Azure assigned a medium VM from the pool of medium VM and added it to the Plan (1). All the Apps running in the plan started to run in that new VM as well. Only then, the small VM got removed from the Plan (2), leaving the plan with a single Medium VM.

We can also easily figure how it would happen with more than one VM. Scaling back in would also happen similarly.

App Service on Linux is currently in Public Preview and enables customers to run their web apps natively on a Linux platform. This allows for better application compatibility for certain kinds of applications and makes it easier to migrate existing web apps hosted on a Linux platform elsewhere onto Azure App Services.

In this third post in my series on Azure App Service Architecture, I’ll focus on how Microsoft implemented this new Platform as a Service product. What will actually run your app if you deploy it on Linux, what Linux distribution they have chose, and how are all the apps deployed on those underlying Linux servers. If you haven’t read my preview two articles, I strongly recommend you read them first: Azure App Service Architecture (1) and Azure App Service Architecture (2) as they will explain some basic concepts that are used in this one.

Provisioning a Web App on Linux

This part is really trivial. I’ll show some screenshots, but they are self explaining. Let’s click on the [+] button, and head to [Web + Mobile]. From there, we’ll select Web App on Linux (Preview):

I’ll set a new unique name for my new Web App:

To be able to explore a little further the options available to us, I’ll choose to create a new App Service Plan, instead of using the default recommended to me. Notice that while the product is in Preview, it is not available in all Azure regions yet. In Europe it’s only available in West Europe for example. That’s enough for me to test it anyway.

Particularly, I don’t need to pay for a Standard plan, so I’ll choose a Basic B1 here.

So here we go, just click the Create button, and in less than a minute, you’ll have a Web App up and running, ready for you to use to deploy your app, using your favorite Deployment method, Git, FTP…

App Service on Linux Architecture

To better understand what has been deployed by Azure, let’s head to the Development tools section of the Web App setting menu, and select Advanced Tools:

Then click on the Go link. It will open the traditional Kudu tools console, where we will have access to some internal information regarding our Web App.

Application layer

First let’s figure out what we have at the Application layer. That is, what kind of software is going to serve our application. In this case, from the Environment tab, we can see some interesting Server Environment Variables:

SERVER_SOFTWARE=Apache/2.4.10 (Debian)

So apparently the Web App is deployed on Apache 2.4 on Debian.

Let’s dig a little deeper with the Bash console, and have a look at the processes we can see:

Kudu Remote Execution Console
Type 'exit' to reset this console.
/home> ps -ef
UID PID PPID C STIME TTY TIME CMD
1001 1 0 0 12:54 ? 00:00:00 /bin/sh -c /usr/sbin/apache2ctl -D FOREGROUND
1001 5 1 0 12:54 ? 00:00:00 /bin/sh /usr/sbin/apache2ctl -D FOREGROUND
1001 7 5 0 12:54 ? 00:00:00 /usr/sbin/apache2 -D FOREGROUND
1001 9 1 0 12:54 ? 00:00:00 /usr/bin/mono /usr/lib/mono/4.5/mod-mono-server4.exe --filename /tmp/.mod_mono_server4 --nonstop --appconfigdir /etc/mono-server4
1001 12 7 0 12:54 ? 00:00:00 /usr/sbin/apache2 -D FOREGROUND
1001 13 7 0 12:54 ? 00:00:00 /usr/sbin/apache2 -D FOREGROUND
1001 71 1 6 12:54 ? 00:00:08 /usr/bin/mono /usr/lib/mono/4.5/mod-mono-server4.exe --filename /tmp/mod_mono_server_default --applications /:/opt/Kudu --nonstop
1001 126 71 0 12:56 ? 00:00:00 /bin/bash -c ps -ef && echo && pwd
1001 127 126 0 12:56 ? 00:00:00 ps -ef

We can see very few processes running, and especially interesting, the PID 1 is apache2ctl, so we are most likely running is a container (if we were in a full server, it would be “init”). All the processes run as a non-root user (uid 1001).

We can indeed confirm that our Web App is running inside a Docker container by looking at /proc/1/cgroup:

/home> cat /proc/1/cgroup
11:memory:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
10:cpuset:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
9:hugetlb:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
8:blkio:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
7:perf_event:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
6:net_cls,net_prio:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
5:pids:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
4:cpu,cpuacct:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
3:freezer:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
2:devices:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
1:name=systemd:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b

So, Web App on Linux are deployed in Docker containers (and we know the ID of the container in the host).

Unfortunately, there’s not much we can do from inside the Docker container itself to guess anything about the host running the Docker engine, so we don’t really know what is the OS flavor, version or anything else.

We can have a look at the kernel, which is shared between the VM (host) and all the containers run in the same VM:

/home> uname -a
Linux e30a13645e09 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 GNU/Linux

Regarding the Docker image that was used, we can see it’s based on Debian 8 (Jessie):

/home> cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Data layer: Persistent Storage?

Storage inside a Docker container is, usually, volatile, but if we deploy our App there, we don’t want to loose it when the container stops or the host reboots. Also, we want to be able to scale out our Web App when load increase. So we require some kind of persistency!

Similarly to Web App on Windows, where the persisted files are found in D:\home, here we’ll store the Persisted files in /home. As this Web App is recently created, it’s actually quite empty right now:

/home>pwd
/home
/home>ls
LogFiles site

So how is the storage actually persisted?

/home> mount | grep /home
//10.0.176.8/volume-46-default/20a35f31f26928286ecf/580efcc549c040228b254d82ab4ed6e1 on /home type cifs (rw,relatime,vers=3.0,sec=ntlmssp,cache=strict,username=dummyadmin,domain=RD0003FF1A594C,uid=1001,forceuid,gid=1002,forcegid,addr=10.0.176.8,file_mode=0700,dir_mode=0700,nounix,serverino,mapposix,mfsymlinks,rsize=1048576,wsize=1048576,actimeo=1)

We can see that the /home directory is actually a network shared volume mounted via CIFS protocol (I can only assume here it’s backed by Azure Storage). That way Azure will be able to scale out our Web App, deploying more containers based on the very same stateless image, mounting the same volume which is where our Web App files are deployed.

I think Microsoft might be using here a custom implementation of their Docker Volume Plugin for Azure File Storage. (I haven’t had time yet to play with it and see how similar (or not) things look with this Docker storage plugin). For the persistent storage, Microsoft is using Fileservers that export SMB shares mounted in the host. Those are then mapped as /home into the corresponding docker containers.

Putting everything together

Let step back now. This is how I see Microsoft implemented App Service on Linux. First when we only deploy 1 web app on a single instance:

Now, how does it look likes if we deploy multiple Web Apps in the same plan, and we scale out the Plan to two instances?

I hope this gave you a good understanding of how Microsoft implemented internally this new App Service on Linux product.

Note: This article is based solely on information gathered from publicly available sources, mainly Microsoft Azure documentation site and Github, wrapped with my own understanding and conclusions.

Let’s have a look at what an App Service Plan is actually made of.

Compute

We’ve seen in the first post that the App Service Plan is formed of one or more VMs instances (which can be dedicated or shared depending on the Service Tier). Also we’ve seen you can deploy multiple apps to the same Plan. The apps will then all run on all the instances of the Plan.

Inside each VM Instance of the Plan, you Apps are deployed in Sandboxes:

• Sandbox mechanism

Azure App Services run in a secure environment called a sandbox. Each app runs inside its own sandbox, isolating its execution from other instances on the same machine as well as providing an additional degree of security and privacy.

The sandbox mechanism mitigates the risk of service disruption due to resource contention and depletion in two ways: it ensures that each app receives a minimum guarantee of resources and quality-of-service, and conversely enforces limits so that an app can not disrupt other concurrently-executing apps on the same machine.

Storage

From a storage perspective, and especially when it comes to scale-out (horizontally), it comes handy to understand what are the storage capabilities available to an App deployed in an App Service Plan. There are two kinds: Temporary storage, and Persisted storage.

• Temporary files

Whithin the context of the Application deployed in the WebApp, a number of common Windows locations are using temporary storage on the local machine. For instance:

%APPDATA% points to something like D:\local\AppData.
%TMP% goes to D:\local\Temp.

Unlike Persisted files, these files are not shared among site instances. Also, you cannot rely on them staying there. For instance, if you stop a site and restart it, you’ll find that all of these folders get reset to their original state.

• Persisted files

Every Azure Web App has a home directory stored/backed by Azure Storage. This network share is where applications store their content. The sandbox implements a dynamic symbolic link in kernel mode which maps d:\home to the customer home directory.

These files are shared between all instances of your site (when you scale it up to multiple instances). Internally, the way this works is that they are stored in Azure Storage instead of living on the local file system. They are rooted in d:\home, which can also be found using the %HOME% environment variable.

App Service Architecture

Now if we put everything together, let’s have a look at how it looks. First let’s start with a single App deployed on a Service Plan with a single VM Instance:

Now let’s see how it looks when we scale-out the plan to two VM instances. We can see how each instance of the App will have a separate Temporary storage in each VM, while they share the Persisted storage (where the App files are deployed).

How does it look if we now deploy multiple Apps to this same App Service Plan?

Notice how within its sandbox, every App in a same VM will keep seeing it’s persisted storage as D:\home, and the Temporary storage as D:\local. That’s quite nice!

Console Access

From Azure Portal you can access the App’s console: the Kudu tools give access to the Web site app at the sandbox level. From there you can access D:\local and D:\home. The hostname command shows the hostname of the VM (ie. the instance where the console is being connected to).

Azure App Services is a Platform-as-a-Service (PaaS) cloud service offering by Microsoft focused on providing superior developer productivity without compromising on the need to deliver applications at cloud scale. It also provides the features and frameworks necessary to compose enterprise applications while supporting developers with the most popular development languages (.NET, Java, PHP, Node.JS and Python). With App Service developers can:

• Build highly scalable Web Apps
• Quickly build Mobile App back-ends with a set of easy to use mobile capabilities such as data back-ends, user authentication and push notifications with Mobile Apps.
• Implement, deploy and publish APIs with API Apps.
• Tie business applications together into workflows and transform data with Logic Apps.

To be able to deploy any App on Azure App Service you’ll need an App Service Plan:

App Service Plans

An App Service Plan represents a set of features and capacity (compute, storage) that you can share across multiple apps. in Azure you deploy a Web App on an App Service Plan, which is formed of one or more (VM) Instance.

App Service Plans are specified by two characteristics:

• The Pricing Tier: from Free, Shared, Basic, Standard to Premium
• The Instance Size: from Small, Medium, Large to Extra Large

The Instance sizes define some compute and storage characteristics of the underlying VM instances that will form the App Service Plan. The CPU and Ram is the same for every Pricing Tier (Basic, Standard or Premium. In Free or Shared you can’t select the Instance Size). Here are the size available at the time of this writing:

The Pricing Tier define some other capacity characteristics as well as features and services that will be available to the App that we deploy in the Plan.

Note that in the Free and Shared tier, the VM Instances are shared in a multitenant fashion (possibly with other customers), while from Basic to Premium tiers, the VM Instances are dedicated.

Internally (in API documentation,…), App Service Plan are actually called “hosting Plan” or “serverFarm”, which reveal the true nature of what they really are.

Deployment model

From a deployment perspective, Apps in the same subscription and geographic location can share a plan. All the apps that share a plan can use all the capabilities and features that are defined by the plan’s tier. All apps that are associated with a plan run on the resources (VM Instances) that the plan defines:

Scalability Up & Out

Scalability of the App Service Plan is achieve either horizontally by adding more VM Instances to the Plan (Scale-Out), or vertically, by changing to  a larger Instance Size (Scale-Up).

Scaling-Up is a manual action, which occurs without service impact. Scaling-Out can be done manually, scheduled to happen periodically, or automated based on some performance metrics (from the Plan itself, or other resources like the Storage, Service Bus,…).

I’ve dedicated a whole post to how Azure achieves Scalability of the App Service Plans.

Watch out for my next post where I’ll explain how App Service Plans are architected internally.

Ideally, my requirements were:

• Backup should be offsite, preferably in the cloud
• Solution should support some kind of backup policies (full, incremental, and retention time)
• Solution should be secure, so support encryption
• Solution should be portable, so same solution would play nicely with the servers, not mess with their OS (dependencies,…)
• Solution should be as cheap as possible (or ideally free!)

For the offsite storage I first though of using AWS Glacier, a cloud storage meant for cold data that you archive, and don’t need to retrieve often, but the cost was not so cheap. While storage in itself was not too expensive, the cost of retrieving the data (in case you want to restore, which is the point of having a backup, right?), was kind of prohibitive (for the budget I had in mind). So I started to look for alternatives.

For the backup solution, I wanted to use duplicity. Duplicity supports full and incremental backups, using the rsync protocol, and have support for a lot of storage backend: file, FTP, SSH, AWS S3, Azure, Dropbox, OneDrive, Mega,… Most of those backends are either illimited but paid services, or free but rather limited (in capacity) storage. All but Mega, which offer for free 50GB of storage, which is quite nice for backup purpose. Perfect fit in my case.

Regarding the portability requirement, I love Docker containers, and all I deploy now for my personal projects is Dockerized. This wasn’t going to be an exception. Especially I’d hate to install all kind of dependencies for duplicity and the storage backend plugins in all my servers!

So Docker it would be.

Now back to the storage layer in our solution: although duplicity supposedly have support for a Mega backend, it seems Mega changed their API/SDK, and the current plugin is not working anymore, and would need to be reworked totally. So as an alternative, I turned to using MegaFuse, a Fuse module to mount Mega storage in Linux: so the idea is we first mount the Mega account as a filesystem in Linux, and then we use it as destination of our backup with duplicity (using the file backend). Not as cool as having duplicity talk directly to Mega, but that seems to work equally.

So to recap, we have a container with duplicity and MegaFuse. As the container is stateless, we’ll map some volumes from the host to the container so the container gets the needed information:

• /vol/dupmega/megafuse.conf, containing some config for MegaFuse, like the credentials to the Mega account (see below),
• As duplicity and MegaFuse both keep a local cache with some metadata, having those stored in the container would do no good, so I also put that in host mapped folder (/vol/dupmega/cache/duplicity/ and /vol/dupmega/cache/megafuse/)
• Of course, we want to backup the host, not the container, so we need to map that as well into the container to /source

The megafuse.conf contains:

USERNAME = [email protected]
PASSWORD = aV3ryComplexMegaP4ssw0rd
MOUNTPOINT = /mega
CACHEPATH = /dupmega/cache/megafuse

So the Mega account is mounted in /mega in the container, and the MegaFuse cache will be in /dupmega/cache/megafuse (host mounted volume).

Here is the Dockerfile I have used to create my duplicity container with MegaFuse support. Right now it’s not yet published to Docker Hub. Right now it’s very rudimentary, there’s not even a CMD.

FROM ubuntu:14.04
MAINTAINER Alexandre Dumont <[email protected]>

ENV DEBIAN_FRONTEND=noninteractive

RUN sed -i -e ‘/^deb-src/ s/^/#/’ /etc/apt/sources.list && \
echo “force-unsafe-io” > /etc/dpkg/dpkg.cfg.d/02apt-speedup && \
echo “Acquire::http {No-Cache=True;};” > /etc/apt/apt.conf.d/no-cache && \
apt-get update && \
apt-get -qy dist-upgrade && \
apt-get install -y libcrypto++-dev libcurl4-openssl-dev libfreeimage-dev libreadline-dev libfuse-dev libdb++-dev duplicity git g++ make && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
git clone https://github.com/matteoserva/MegaFuse && \
cd MegaFuse && \
make

I use the following command to build the image:

docker build -t adumont/dupmega --no-cache=true .

And here’s the image:

REPOSITORY TAG IMAGE ID CREATED SIZE
adumont/dupmega latest 3bf1d313aa56 7 days ago 581.8 MB


I still have to work on simplifying the whole process of running a backup. For now, I do it rather manually, but it will become a script and be scheduled in cron most likely.

Right now that’s how I run it, notice the volume mapping from host to containers. Also notice the container has to run as privileged, so it can use Fuse from inside the container. The source to be backed up is mounted read-only (least privilege).

host# docker run --rm -h $( hostname ) -ti --privileged \
-v /vol/dupmega:/dupmega \
-v /root/.gnupg:/root/.gnupg \
-v /vol/dupmega/cache/duplicity:/root/.cache/duplicity \
-v /:/source:ro adumont/dupmega


Then from inside the container, I run:

mkdir /mega; MegaFuse/MegaFuse -c /dupmega/megafuse.conf &>/dev/null &
sleep 10
[ -d /mega/backups/$(hostname) ] || exit 1

export PASSPHRASE=AnotherVeryComplexPassphaseForGPGEncrypti0n

and finally the duplicity command which will backup /source to /mega. The command is different for each server, as I tweak which files/folders I want to include/exclude in the backup:

duplicity --asynchronous-upload \
--include=/source/var/lib/docker/containers \
--include=/source/var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Preferences.xml \
--exclude=/source/dev \
--exclude=/source/proc \
--exclude=/source/run \
--exclude=/source/sys \
--exclude=/source/zfs \
--exclude=/source/mnt \
--exclude=/source/media \
--exclude=/source/vol/dupmega/cache \
--exclude=/source/tank \
--exclude=/source/vbox \
--exclude=/source/var/lib/docker \
--exclude=/source/var/lib/plexmediaserver/ \
--exclude=/source/tmp \
--exclude=/source/var/tmp \
--exclude=/source/var/cache \
--exclude=/source/var/log \
/source/ file:///mega/backups/$(hostname)/ -v info


And this would be a sample output:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Oct 20 22:51:23 2016
Deleting /tmp/duplicity-Cwz5Ju-tempdir/mktemp-0lEM40-2
Using temporary directory /root/.cache/duplicity/185b874acbbae73c5807a4cc767e4967/duplicity-kYzeVT-tempdir
Using temporary directory /root/.cache/duplicity/185b874acbbae73c5807a4cc767e4967/duplicity-jWJXmd-tempdir
AsyncScheduler: instantiating at concurrency 1
M boot/grub/grubenv
A etc
A etc/apparmor.d/cache
M etc/apparmor.d/cache/docker
[...]
---------------[ Backup Statistics ]--------------
StartTime 1477499247.82 (Wed Oct 26 16:27:27 2016)
EndTime 1477499514.97 (Wed Oct 26 16:31:54 2016)
ElapsedTime 267.15 (4 minutes 27.15 seconds)
SourceFiles 449013
SourceFileSize 3239446764 (3.02 GB)
NewFiles 75
NewFileSize 189457 (185 KB)
DeletedFiles 135
ChangedFiles 68
ChangedFileSize 176144101 (168 MB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 278
RawDeltaSize 9317325 (8.89 MB)
TotalDestinationSizeChange 1913375 (1.82 MB)
Errors 0
-------------------------------------------------


And backup are really stored on Mega :