,

Scaling Sitecore Azure Web App

Avatar de Vinicius

Before scaling Sitecore Azure Web App, you must read the experience I had and going to share in this post.

Azure Web App

In Azure, you can easily change existent resources almost in real time to accomodate business needs, and this is also applies to Web Apps where you can Scale Up and/or Scale Out your application.

Sitecore

Sitecore offers different architecture models to fit client’s needs, and its roles vary depending on the topology is choosen. Nevertheless, you have the ability to scale either vertical and/or horizontally.

The environment

Everything is hosted in Azure, using Web Apps, and provisioned following Sitecore XP Scaled topology. Here is an idea in how the environment looked like

sitecore architecture azure paas blog vinicius deschamps

Before moving forward it is important to know about the difference between Azure Web App and App Service Plans

Azure Web Apps vs App Service Plans

During Azure Web App creation, you are prompted to provide the Instance Details (1) that corresponds to the Web App, and App Service Plan (2) which must be one of the following: create new or choose an existent one from the dropdown list.

azure azure app service vs azure app service plan create blog vinicius deschamps

The relation between Web App and App Service Plan can be illustrated as the image below

azure app service vs app service plan blog vinicius deschamps

N Web Apps can use the same App Service Plan, as long as the resources the App Service Plan can handle.

In addition to that, an idea that helps to understand is to think like

  • App Service Plan is the physical or virtual server which has the resources (CPU, memory, etc)
  • Web App is the web server where configuration and settings sits

That said, we wanted to Scale Out both CM and CD, and everything went well with the CD, so let’s attempt to perform the same on CM.

Scale Out CM idea

When you Scale Out your application, a Load Balancer is added in front of your Azure Web App, and Sitecore CM expects exactly that for a horizontal scaling

You can set up multiple instances behind a load balancer. All instances must connect to a centrally located master database

https://doc.sitecore.com/developers/100/platform-administration-and-architecture/en/scaling-and-configuring-content-management.html

Thus we have decided to simply Scale Out the CM Web App

Everything went wrong #1

Once all instances were up and running, a couple tests were made and things were working just fine.

However, when multiple users started to work, things started to become out of control with instances randomically restarting out of the blue with the following message

WARN Shutdown message: CONFIG change

Always look for the tiny details

By scaling out the CM, the environment was going from 1 to N servers, and Sitecore has different requirements to make this to happen

InstanceName parameter

You must ensure that each CM instance has a unique name, and that’s OK because the default value set in Sitecore.config file is (machine name and IIS site name), and when you scale out you will get a clone App Service that runs with a unique name.

<!--  INSTANCE NAME
            Unique name for Sitecore instance.
            Default value: (machine name and IIS site name)
      -->
    <setting name="InstanceName" value="" />

Publishing

Only one CM instance can perform publishing, and you must ensure that each CM instance points to the publishing CM instance, and again that’s OK because the environment has a dedicated Publishing Service.

ValidationKey and DecryptionKey

Ensure that the values is the same on all the Sitecore instances within the environment and that the IsolateApps modifier is not present in either value, and in the environment configuration these parameters were set fixed and statically.

Indexing

The indexing sub-role is enabled on the Content Mangement role by default. If you have more than one instance of the Content Management, you must disable indexing on all except one instance.

Since a Scale Out simply clone the file system, and doesn’t allow us to have control in each new server’s file system to modify the Indexing configuration, Sitecore CM was getting confuse with multiple servers performing the indexing role.

Now, that the issue has been identified, the idea was to provisioning a new Content Management using its own App Service (and App Service Plan) where the file system could be modified as needed.

So, let’s call this new Content Management CM02 and the existent CM01.

Enable Indexing sub-role CM02

The idea is to have CM02 with the Indexing sub-role, and to accomplish that a modification must be done in the Web.config

<add key="role:define" value="ContentManagement, Indexing, Indexer"/>

Starting in Sitecore 9.2, when you assign the indexing sub-role to a server role, that role manages content indexing automatically and the other server roles that have not been assisgned this sub-role do not index content

https://dev.sitecore.net/Downloads/Sitecore%20Experience%20Platform/92/Sitecore%20Experience%20Platform%2092%20Initial%20Release/Release%20Notes

Disable Indexing sub-role CM01

IMPORTANT: If you are using Sitecore 9.2 and newer, you don’t need to disable indexing on other CMs

For each Index you have configured, you must change its indexUpdateStrategies to manual like the example below

<index id="sitecore_index" type="Sitecore.ContentSearch.SolrProvider.SolrSearchIndex, 
 Sitecore.ContentSearch.SolrProvider">
 <param desc="name">$(id)</param>
 <param desc="core">$(id)</param>
 <param desc="propertyStore"
 ref="contentSearch/indexConfigurations/databasePropertyStore"
 param1="$(id)" />
 <strategies hint="list:AddStrategy">
 <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/manual" />
 </strategies>

Once these changes were applied, here is how the environment look like

sitecore architecture azure paas scaled out blog vinicius deschamps

Everything went wrong #2

Now, we have everything that is required to have a multiple CM environment

  • Load Balancer in front of the CMs
  • InstanceName parameter with Unique Names
  • A Publishing Server
  • Same ValidationKey and DecryptionKey for all CMs
  • One CM to handle Indexing

And things were looking very promising, well… well… until the instances from the scaled out CM01 started to fall apart and, again, randomly and with the exactly same message.

WARN Shutdown message: CONFIG change

What else could be wrong? After some research, I found something answered our question

Accordingly to Sitecore Managed Cloud documentation

Scale out is not supported for CM

https://doc.sitecore.com/developers/managed-cloud/en/scaling-options-for-managed-cloud.html

The solution

That said, the only possibility would be to have additional CMs each of them in their own App Service Plan.

However, since we are not scaling the CM App Service anymore, we don’t have a load balancer to split the load between the multiple CMs and there are some important things to know

  • Azure Load Balancer does not work with Azure App Services, even if you configure VNet for them
  • WAF and Application Gateway are not compatible with Content Management servers

So, you either come up with a third party solution that will assist you in have a Load Balancer for your CMs or you can use Azure Front Door but please keep in mind the following

Use of this technology within a Sitecore solution is permitted, but the Sitecore Support team can only assist with Sitecore product issues that are not unique to the use of the technology, that is, a Sitecore product issue that can be replicated without the technology present.

In some cases, the Sitecore Support team may ask for replication of the issue without the use of the technology, although they will try to avoid this.

https://kb.sitecore.net/articles/682999#Notes

In my case, I am going to use it and here is how it suppose to be

sitecore content management azure front door blog vinicius deschamps

Configure Azure Front Door

Search for Front Doors, and click Front Doors in Azure Search field

azure search front doors blog vinicius deschamps

Click Add to create a new Front Door

azure front doors add blog vinicius deschamps

In Basics choose the appropriate Resource Group or create a new one, and Next : Configuration >

azure create a front door basics blog vinicius deschamps 1

At Configuration you will see 3 steps, let’s start by clicking at the + sign on Step 1

azure create a front door configuration frontends domains blog vinicius deschamps 1

In Add a frontend host provide Host name (1), ensure to choose Enabled for Session Affinity (2) and disabled for Web Application Firewall (3) – since Sitecore does not like it – and click Add

azure create a front door configuration frontends domains add a frontend host blog vinicius deschamps

Let’s move to the Step 2 by clicking in the + sign

azure create a front door configuration backend pools blog vinicius deschamps

Add a backend pool by giving it a Name (1) and + Add a backend (2)

azure create a front door add a backend pool blog vinicius deschamps

Choose a Backend host type (1), in our case an App Service, then choose the Backend host name (2) and its Backend host header (3), and hit Add

azure create a front door configuration backend pools add a backend app service blog vinicius deschamps

Ensure to repeat the step above by clicking + Add a backend until you have all CM Web Apps added

azure create a front door add a backend pool repeat add a backend app services blog vinicius deschamps
azure create a front door add a backend pool repeat complete add a backend app services blog vinicius deschamps

Once you have all Web Apps, you can continue on Add a backend pool options, and simply click Add

azure create a front door configuration backend pools add a backend health probe load balancing blog vinicius deschamps

Finally, let’s configure the Step3 by clicking at + sign

azure create a front door routing rules blog vinicius deschamps

Give it a Name (1), and choose its Frontends/domains (2) and the Backend pool (3), then press Add (4)

azure create a front door routing rules add a rule blog vinicius deschamps

Now that you have all 3 steps configured hit Review + Create

azure create a front door review create blog vinicius deschamps

Review the information and hit Create

azure create a front door review create create blog vinicius deschamps

You should wait until Front Door is provisioned, and you can access from your browser the host name you have set, in my case is https://blog-vinicius-deschamps.azurefd.net

azure front door access sitecore cm blog vinicius deschamps

Great but you probably don’t want to share an azurefd.net domain with your Content Editors so they can access your CM. Let’s see how to configure a custom domain.

Custom domain Azure Front Door

Go to the Front Door (1), select Front Door designer (2), then click in the + (3) at Frontends/domains

azure front door front door design frontends domains add new domain blog vinicius deschamps

You should choose a Custom host name, and you will receive a message saying that a CNAME configuration must be done at your DNS

azure front door add a custom domain blog vinicius deschamps

I am using Azure DNS, so I went to my domain viniciusdeschamps.cloud (1), selected Overview (2) and + Record set (3)

azure dns new record set blog vinicius deschamps

In Add record set, add a Name (1), select the Type (2) and add the Alias (3), then hit Ok (4)

azure dns add record set azure front door cname blog vinicius deschamps

Back to Front Door, and type your Custom host name (1) again, choose Enabled (2) for Custom Domain HTTPS, and in a Production environment use your own certificate but in my case I’m going to use Front Door managed (3), and don’t forget to Enabled (4) Session Affinity and press Add (5)

azure front door add a custom domain after adding dns cname blog vinicius deschamps

In Front Door Designer you will see the following warning

The host ‘your-custom-host.domain’ is not present in any of the routing rules.

The host ‘your-custom-host.domain’ is missing a default route ‘/*’ path for both the HTTP and HTTPS protocolos

azure front door the host is not present in any of the routing rules blog vinicius deschamps

Select the existing Routing rules

azure front door routing rules update existing rule blog vinicius deschamps

And in Frontends/domains choose the recently added domain from the list

azure front door update routing rule frontends domains blog vinicius deschamps

Then, press Update

azure front door update routing rule hit update button after select new frontends domains blog vinicius deschamps

Once you have done, click Save in Front Door designer

azure front door there are pending changes click save to apply blog vinicius deschamps
azure front door with custom domain and new rules blog vinicius deschamps

Please wait for a while until you get a confirmation the configuration has been applied, in my case, it took a while because I choose the Front Door managed certificate

azure front door updating the custom https configuration blog vinicius deschamps
azure front door updated the custom https configuration blog vinicius deschamps

Sitecore Identity Server Unauthorized_Client

Once the Azure Front Door was updated with the new Custom Domain, I tried to access https://prodcm.viniciusdeschamps.cloud/sitecore and when hit the Identity Server got the following message

Sorry, there was an error: unauthorized_client

And this happens because in the Identity Server configuration you should set from which URLs authentication requests should be accepted

sitecore identity server sorry there was an error unauthorized client blog vinicius deschamps

In order to fix it, you should go to Sitecore Identity Server App Service (1), Advanced Tools (2) and Go (3)

azure app service sitecore identity server advanced tools go blog vinicius deschamps

Select Debug console, and Powershell

azure app service kudu debug console powershell blog vinicius deschamps

Navigate to site\wwwroot\Config\production, and access the following file Sitecore.IdentityServer.Host.xml then modify the following line

<AllowedCorsOriginsGroup1>https://sc91vdsch-cm.azurewebsites.net|https://</AllowedCorsOriginsGroup1>
sitecore identityserver host xml add new custom domain blog vinicius deschamps

By adding all Content Management URLs as follows, and hit Save

<AllowedCorsOriginsGroup1>https://sc91vdsch-cm.azurewebsites.net|https://sc91vdsch-cm02.azurewebsites.net|https://sc91vdsch-cm03.azurewebsites.net|https://prodcm.viniciusdeschamps.cloud</AllowedCorsOriginsGroup1>
sitecore identityserver host xml add new custom domain save blog vinicius deschamps

Once the modification is saved, you should Restart the Sitecore Identity Server App Service

azure app service sitecore identity server overview restart blog vinicius deschamps
azure app service sitecore identity server are you sure you want to restart blog vinicius deschamps

Once the restart is completed, try to access Sitecore CM using the custom domain configured on Azure Front Door, in my case https://prodcm.viniciusdeschamps.cloud/sitecore and voilá no error

sitcore identity server fully accessible type credentials blog vinicius deschamps

I got redirect to the Sitecore Experience Platform, and as you can see to the CM02 but that’s not I want for my end users, so there are additional steps to address it. Well, at least we know the Front Door is splitting the traffic

sitecore not using azure front door custom domain blog vinicius deschamps

Custom Domains for App Services

Navigate to the App Services, and choose one of the CMs (1), select Custom Domains (2) and copy the Custom Domain Verification ID (3)

azure app service custom domains custom domain verification id blog vinicius deschamps

Go to your DNS provider, in my case Azure DNS, and let’s add a new Record Set

azure dns new record set blog vinicius deschamps

The step below is to check domain ownership, and you should fill the information as follows

  • Name – asuid.SUBDOMAIN (e.g asuid.prodcm)
  • Type – TXT
  • Value – Paste the value you copied from Custom Domain Verification ID
azure dns add record set custom domain verification txt blog vinicius deschamps

Back to the Custom Domains, and click Add custom domain (3)

azure app service custom domains add custom domain blog vinicius deschamps

Type your Custom Domain (1), then click Validate (2)

azure app service add custom domain validate blog vinicius deschamps

Sometimes it take a while until the record is sync, and you might see the following message

Domain ownership

To verify domain ownership create CNAME and TXT record with your DNS provider

Since we already have a CNAME configured for the Azure Front Door, the CNAME record is not required

azure app service add custom domain add custom domain domain ownership blog vinicius deschamps

Wait for a while and try to Validate the Custom Domain again, and if everything is OK, you should see the following and hit Add custom domain

azure app service add custom domain add custom domain blog vinicius deschamps

IMPORTANT: You should add the same custom domain to every CM App Service!

And finally, when you access your Azure Front Door custom domain to hit Sitecore you will notice that now you have a single URL for all Content Management, which in my case is https://prodcm.viniciusdeschamps.cloud

sitecore using azure front door custom domain blog vinicius deschamps

And that’s it!

I hope you liked, and I’ll see you on my next post.

Photo by Marius Masalar on Unsplash

Tagged in :

Avatar de Vinicius

Uma resposta para “Scaling Sitecore Azure Web App”

  1. […] You can get a detailed explanation about Azure App Service (Web App) and App Service Plans here. […]

Deixe um comentário

O seu endereço de e-mail não será publicado. Campos obrigatórios são marcados com *