Scalability is the capability of a system, network, or process to handle a growing amount of work, or its potential to be enlarged in order to accommodate that growth.

Looking at scalability is very relevant in Cloud environment, which provide a high level of on demand elasticity and thus allow us to easily implement the scalability patterns most relevant to our applications to cater for our particular business needs, whichever they may be.

App Service Plan Provisioning

Let first have a look at what happen when you provision a new App Service Plan in Azure.

When you request the creation of an Azure Web App onto a new App Service Plan (aka. Serverfarm), Azure is not provisioning a brand new VM from scratch. Instead, I believe, they assign you a recycled VM of the chosen size, for example B1, from a pool of existing VM.

Indeed you can see that by looking at the uptime of the provisioned VM instance (use Kudu advanced tools): uptime is quite random, and not near 0. A near 0 uptime would indicate a recently created VM. So, in my understanding, Azure keeps a pool of VMs just ready for when a new customer is requesting a new service plan, or when a scaling operation is needed.

That make a lot of sense. Provisioning a new VM each time would take quite a lot of time, while a new Web App deployment actually happens in about 15 to 20 seconds. For that purpose, I assume they will always keep a certain number of VMs ready in the pool, enough to respond to customer’s needs at all time in a timely manner.

Horizontal scalability (Scale out / Scale in)

Horizontal scalability is when you change the number of VM instances supporting your App Service Plan. When load on your apps grows, you will likely scale-out (add more VM instances). When the loads decrease, you will scale back in, reducing the number of VM instances.

As I understand it, when scaling-out, an operation similar to the provisioning will happen: Azure will assign you one new VM of the same size, from its pool of available VMs:

When the plan scales back in again, the VM is removed from the Plan and placed back in the correspondin pool. As I understand it, it is recycled so it can be reused again, possibly by another customer.

Vertical scalability (Scale up/down)

Vertical scalability is about changing the App Service Plan instance size. If you need more compute power to run you apps, you can choose to scale-up your plan to bigger VM(s). What happen in that case though?

AFAIK, Azure runs on Hyper-V virtualization platform, and at the time of this writing, Hyper-V doesn’t allow for dynamic CPU/RAM resizing of VMs. So how do they do it?

Well, I did the test on a Plan with a single small instance. I noticed the instance hostname and uptime before scaling up. I then scaled the plan up to medium, and guess what: the hostname and uptime of the only instance in the plan was totally different!

Here’s my educated guess: Azure assigned a medium VM from the pool of medium VM and added it to the Plan (1). All the Apps running in the plan started to run in that new VM as well. Only then, the small VM got removed from the Plan (2), leaving the plan with a single Medium VM.

We can also easily figure how it would happen with more than one VM. Scaling back in would also happen similarly.

App Service on Linux is currently in Public Preview and enables customers to run their web apps natively on a Linux platform. This allows for better application compatibility for certain kinds of applications and makes it easier to migrate existing web apps hosted on a Linux platform elsewhere onto Azure App Services.

In this third post in my series on Azure App Service Architecture, I’ll focus on how Microsoft implemented this new Platform as a Service product. What will actually run your app if you deploy it on Linux, what Linux distribution they have chose, and how are all the apps deployed on those underlying Linux servers. If you haven’t read my preview two articles, I strongly recommend you read them first: Azure App Service Architecture (1) and Azure App Service Architecture (2) as they will explain some basic concepts that are used in this one.

Provisioning a Web App on Linux

This part is really trivial. I’ll show some screenshots, but they are self explaining. Let’s click on the [+] button, and head to [Web + Mobile]. From there, we’ll select Web App on Linux (Preview):

I’ll set a new unique name for my new Web App:

To be able to explore a little further the options available to us, I’ll choose to create a new App Service Plan, instead of using the default recommended to me. Notice that while the product is in Preview, it is not available in all Azure regions yet. In Europe it’s only available in West Europe for example. That’s enough for me to test it anyway.

Particularly, I don’t need to pay for a Standard plan, so I’ll choose a Basic B1 here.

So here we go, just click the Create button, and in less than a minute, you’ll have a Web App up and running, ready for you to use to deploy your app, using your favorite Deployment method, Git, FTP…

App Service on Linux Architecture

To better understand what has been deployed by Azure, let’s head to the Development tools section of the Web App setting menu, and select Advanced Tools:

Then click on the Go link. It will open the traditional Kudu tools console, where we will have access to some internal information regarding our Web App.

Application layer

First let’s figure out what we have at the Application layer. That is, what kind of software is going to serve our application. In this case, from the Environment tab, we can see some interesting Server Environment Variables:

SERVER_SOFTWARE=Apache/2.4.10 (Debian)

So apparently the Web App is deployed on Apache 2.4 on Debian.

Let’s dig a little deeper with the Bash console, and have a look at the processes we can see:

Kudu Remote Execution Console
Type 'exit' to reset this console.
/home> ps -ef
UID PID PPID C STIME TTY TIME CMD
1001 1 0 0 12:54 ? 00:00:00 /bin/sh -c /usr/sbin/apache2ctl -D FOREGROUND
1001 5 1 0 12:54 ? 00:00:00 /bin/sh /usr/sbin/apache2ctl -D FOREGROUND
1001 7 5 0 12:54 ? 00:00:00 /usr/sbin/apache2 -D FOREGROUND
1001 9 1 0 12:54 ? 00:00:00 /usr/bin/mono /usr/lib/mono/4.5/mod-mono-server4.exe --filename /tmp/.mod_mono_server4 --nonstop --appconfigdir /etc/mono-server4
1001 12 7 0 12:54 ? 00:00:00 /usr/sbin/apache2 -D FOREGROUND
1001 13 7 0 12:54 ? 00:00:00 /usr/sbin/apache2 -D FOREGROUND
1001 71 1 6 12:54 ? 00:00:08 /usr/bin/mono /usr/lib/mono/4.5/mod-mono-server4.exe --filename /tmp/mod_mono_server_default --applications /:/opt/Kudu --nonstop
1001 126 71 0 12:56 ? 00:00:00 /bin/bash -c ps -ef && echo && pwd
1001 127 126 0 12:56 ? 00:00:00 ps -ef

We can see very few processes running, and especially interesting, the PID 1 is apache2ctl, so we are most likely running is a container (if we were in a full server, it would be “init”). All the processes run as a non-root user (uid 1001).

We can indeed confirm that our Web App is running inside a Docker container by looking at /proc/1/cgroup:

/home> cat /proc/1/cgroup
11:memory:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
10:cpuset:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
9:hugetlb:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
8:blkio:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
7:perf_event:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
6:net_cls,net_prio:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
5:pids:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
4:cpu,cpuacct:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
3:freezer:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
2:devices:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b
1:name=systemd:/docker/fb05a97d54930566111197c8959a5a33ac9af29b6491bf1ca8158b18df50264b

So, Web App on Linux are deployed in Docker containers (and we know the ID of the container in the host).

Unfortunately, there’s not much we can do from inside the Docker container itself to guess anything about the host running the Docker engine, so we don’t really know what is the OS flavor, version or anything else.

We can have a look at the kernel, which is shared between the VM (host) and all the containers run in the same VM:

/home> uname -a
Linux e30a13645e09 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 GNU/Linux

Regarding the Docker image that was used, we can see it’s based on Debian 8 (Jessie):

/home> cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Data layer: Persistent Storage?

Storage inside a Docker container is, usually, volatile, but if we deploy our App there, we don’t want to loose it when the container stops or the host reboots. Also, we want to be able to scale out our Web App when load increase. So we require some kind of persistency!

Similarly to Web App on Windows, where the persisted files are found in D:\home, here we’ll store the Persisted files in /home. As this Web App is recently created, it’s actually quite empty right now:

/home>pwd
/home
/home>ls
LogFiles site

So how is the storage actually persisted?

/home> mount | grep /home
//10.0.176.8/volume-46-default/20a35f31f26928286ecf/580efcc549c040228b254d82ab4ed6e1 on /home type cifs (rw,relatime,vers=3.0,sec=ntlmssp,cache=strict,username=dummyadmin,domain=RD0003FF1A594C,uid=1001,forceuid,gid=1002,forcegid,addr=10.0.176.8,file_mode=0700,dir_mode=0700,nounix,serverino,mapposix,mfsymlinks,rsize=1048576,wsize=1048576,actimeo=1)

We can see that the /home directory is actually a network shared volume mounted via CIFS protocol (I can only assume here it’s backed by Azure Storage). That way Azure will be able to scale out our Web App, deploying more containers based on the very same stateless image, mounting the same volume which is where our Web App files are deployed.

I think Microsoft might be using here a custom implementation of their Docker Volume Plugin for Azure File Storage. (I haven’t had time yet to play with it and see how similar (or not) things look with this Docker storage plugin). For the persistent storage, Microsoft is using Fileservers that export SMB shares mounted in the host. Those are then mapped as /home into the corresponding docker containers.

Putting everything together

Let step back now. This is how I see Microsoft implemented App Service on Linux. First when we only deploy 1 web app on a single instance:

Now, how does it look likes if we deploy multiple Web Apps in the same plan, and we scale out the Plan to two instances?

I hope this gave you a good understanding of how Microsoft implemented internally this new App Service on Linux product.

Note: This article is based solely on information gathered from publicly available sources, mainly Microsoft Azure documentation site and Github, wrapped with my own understanding and conclusions.

Let’s have a look at what an App Service Plan is actually made of.

Compute

We’ve seen in the first post that the App Service Plan is formed of one or more VMs instances (which can be dedicated or shared depending on the Service Tier). Also we’ve seen you can deploy multiple apps to the same Plan. The apps will then all run on all the instances of the Plan.

Inside each VM Instance of the Plan, you Apps are deployed in Sandboxes:

• Sandbox mechanism

Azure App Services run in a secure environment called a sandbox. Each app runs inside its own sandbox, isolating its execution from other instances on the same machine as well as providing an additional degree of security and privacy.

The sandbox mechanism mitigates the risk of service disruption due to resource contention and depletion in two ways: it ensures that each app receives a minimum guarantee of resources and quality-of-service, and conversely enforces limits so that an app can not disrupt other concurrently-executing apps on the same machine.

Storage

From a storage perspective, and especially when it comes to scale-out (horizontally), it comes handy to understand what are the storage capabilities available to an App deployed in an App Service Plan. There are two kinds: Temporary storage, and Persisted storage.

• Temporary files

Whithin the context of the Application deployed in the WebApp, a number of common Windows locations are using temporary storage on the local machine. For instance:

%APPDATA% points to something like D:\local\AppData.
%TMP% goes to D:\local\Temp.

Unlike Persisted files, these files are not shared among site instances. Also, you cannot rely on them staying there. For instance, if you stop a site and restart it, you’ll find that all of these folders get reset to their original state.

• Persisted files

Every Azure Web App has a home directory stored/backed by Azure Storage. This network share is where applications store their content. The sandbox implements a dynamic symbolic link in kernel mode which maps d:\home to the customer home directory.

These files are shared between all instances of your site (when you scale it up to multiple instances). Internally, the way this works is that they are stored in Azure Storage instead of living on the local file system. They are rooted in d:\home, which can also be found using the %HOME% environment variable.

App Service Architecture

Now if we put everything together, let’s have a look at how it looks. First let’s start with a single App deployed on a Service Plan with a single VM Instance:

Now let’s see how it looks when we scale-out the plan to two VM instances. We can see how each instance of the App will have a separate Temporary storage in each VM, while they share the Persisted storage (where the App files are deployed).

How does it look if we now deploy multiple Apps to this same App Service Plan?

Notice how within its sandbox, every App in a same VM will keep seeing it’s persisted storage as D:\home, and the Temporary storage as D:\local. That’s quite nice!

Console Access

From Azure Portal you can access the App’s console: the Kudu tools give access to the Web site app at the sandbox level. From there you can access D:\local and D:\home. The hostname command shows the hostname of the VM (ie. the instance where the console is being connected to).

Azure App Services is a Platform-as-a-Service (PaaS) cloud service offering by Microsoft focused on providing superior developer productivity without compromising on the need to deliver applications at cloud scale. It also provides the features and frameworks necessary to compose enterprise applications while supporting developers with the most popular development languages (.NET, Java, PHP, Node.JS and Python). With App Service developers can:

• Build highly scalable Web Apps
• Quickly build Mobile App back-ends with a set of easy to use mobile capabilities such as data back-ends, user authentication and push notifications with Mobile Apps.
• Implement, deploy and publish APIs with API Apps.
• Tie business applications together into workflows and transform data with Logic Apps.

To be able to deploy any App on Azure App Service you’ll need an App Service Plan:

App Service Plans

An App Service Plan represents a set of features and capacity (compute, storage) that you can share across multiple apps. in Azure you deploy a Web App on an App Service Plan, which is formed of one or more (VM) Instance.

App Service Plans are specified by two characteristics:

• The Pricing Tier: from Free, Shared, Basic, Standard to Premium
• The Instance Size: from Small, Medium, Large to Extra Large

The Instance sizes define some compute and storage characteristics of the underlying VM instances that will form the App Service Plan. The CPU and Ram is the same for every Pricing Tier (Basic, Standard or Premium. In Free or Shared you can’t select the Instance Size). Here are the size available at the time of this writing:

The Pricing Tier define some other capacity characteristics as well as features and services that will be available to the App that we deploy in the Plan.

Note that in the Free and Shared tier, the VM Instances are shared in a multitenant fashion (possibly with other customers), while from Basic to Premium tiers, the VM Instances are dedicated.

Internally (in API documentation,…), App Service Plan are actually called “hosting Plan” or “serverFarm”, which reveal the true nature of what they really are.

Deployment model

From a deployment perspective, Apps in the same subscription and geographic location can share a plan. All the apps that share a plan can use all the capabilities and features that are defined by the plan’s tier. All apps that are associated with a plan run on the resources (VM Instances) that the plan defines:

Scalability Up & Out

Scalability of the App Service Plan is achieve either horizontally by adding more VM Instances to the Plan (Scale-Out), or vertically, by changing to  a larger Instance Size (Scale-Up).

Scaling-Up is a manual action, which occurs without service impact. Scaling-Out can be done manually, scheduled to happen periodically, or automated based on some performance metrics (from the Plan itself, or other resources like the Storage, Service Bus,…).

I’ve dedicated a whole post to how Azure achieves Scalability of the App Service Plans.

Watch out for my next post where I’ll explain how App Service Plans are architected internally.