Troubleshoot Contact Creation Issue in Sitecore

When creating contact in List Manager, you may face issues which block contact either been created or showing. Here are some scenarios of troubleshooting the issues.

1, Error occurred when adding a new contact

When checking server logs, you may find some error like below:

16652 10:29:40 ERROR [Sitecore Services]: HTTP POST

URL https://xxx.com/sitecore/api/lists/b694feae-dd8d-45e2-ff8a-74cf185c8606/contacts

Exception Sitecore.XConnect.XdbCollectionUnavailableException: The HTTP response was not successful: Unauthorized

   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)

   at Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock(Func`1 taskFactory)

From browser developer tool, we can see the same error and request payload:

{“Identifier”:”test@test.com”,”IdentifierSource”:””,”Email”:”test@test.com”,”FirstName”:”test firstname”,”LastName”:”test lastname”}

This type of error is mainly due to the Sitecore environment server and client certificates configuration which might be invalid or non-matched thumbprint value. It will cause other module malfunction too, such as Experience Analytics. Once this certificate related environment issue fixed, the contact creation issue should be gone.

2, Contact doesn’t show in the list

After contact created (blue bar with successful message) and wait even several hours, the contact still doesn’t show in the list.

Once we see blue bar with successful message, that means contact has been created and stored in database. Retrieval of contacts is through Sitecore xdb index. Most of the time, the issue is because of contact data not been indexed.

There is a windows service “xConnect Collection Search IndexWorker” which indexes contact data.

First, check the Solr version, make sure it is compatible to installed Sitecore. Sometime the old Solr version been used with non-matched schema.

Second, the xdb index might be wrong on shard data due to copying from another Solr instance, which is very common when creating Solr instances in a distributed environment. So we need to rebuild xdb index.

Sitecore Index Manager does provide interface to rebuild all indexes except xdb index.

Here are the steps to invoke the rebuild request command:

• Go to the server where the xConnect Search Indexer is running and open a command line.

• Go to the xConnect Collection Search IndexWorker folder: \App_Data\jobs\continuous\IndexWorker.

• Run following command. The command registers a small document in the live core signaling that the rebuild should be started. The xConnect Search Indexer will notice the document and start the rebuild process.

Sitecore.XConnectSearchIndexer.exe -rr

or

Sitecore.XConnectSearchIndexer.exe -requestrebuild

After above troubleshooting, you should see the contact in the list successfully.

Setup SolrCloud with Zookeeper For Sitecore

Apache Solr includes the ability to set up a cluster of Solr servers that combines fault tolerance and high availability called SolrCloud, these capabilities provide distributed indexing and search capabilities.

SolrCloud is flexible distributed search and indexing, without a master node to allocate nodes, shards and replicas. Instead, Solr uses Zookeeper to manage these locations, depending on configuration files and schemas. Queries and updates can be sent to any server. Solr will use the information in the Zookeeper database to figure out which servers need to handle the request.

From Sitecore 9.0 update 2, Sitecore supports SolrCloud – the Solr scaled environment. Here are steps to setup SolrCloud with Zookeeper:

1, Create a SolrCloud farm.

We use 2 Solr nodes plus an external Zookeeper instance to form a farm. Download Solr and Zookeeper from following links and extract them to same folder. Both node1 and 2 have same Solr extract files.

• Solr: https://lucene.apache.org/solr/downloads.html (v7.2.1)

• Zookeeper: https://www.apache.org/dyn/closer.cgi/zookeeper/ (v3.5.5). Choose the.bin.tar file. For example: apache-zookeeper-3.5.5-bin.tar

• Get Solrssl.ps1 from Kam Figy’s blog: https://kamsar.net/index.php/2017/10/Quickly-add-SSL-to-Solr/ This is the Powershell script to create self-signed SSL for installed Solr instances.

Install Apache Zookeeper as a standalone service.

• Copy and Rename “zoo_sample.cfg” to “zoo.cfg” in “\apache-zookeeper-3.5.5-bin\conf”.

• Inside zoo.cfg, edit “dataDir=/tmp/zookeeper” to something like “dataDir=/var/zookeeper” which is the location of Zookeeper snapshot storage.

• Add entries in System Environment Variables: ZOOKEEPER_HOME = C:\Solrs\ apache-zookeeper-3.5.5-bin. Edit System Variable “Path” and append “;%ZOOKEEPER_HOME%\bin” at last.

• Open Cmd window and type zkserver. The Zookeeper will start to run on the default port 2181 setup in zoo.cfg.

• Before you start any SolrCloud nodes, you must configure your solr cluster properties in ZooKeeper, so that Solr nodes know to communicate via SSL.

• Navigate to the folder \solr-7.2.1\server\scripts\cloud-scripts in command prompt and run command below.

       server\scripts\cloud-scripts\zkcli.bat -zkhost localhost:2181 -cmd clusterprop -name urlScheme -val https

• Start Solr on node 1 and 2 with cloud mode. Navigate to node 1 and 2 Solr bin folder from Command Prompt window. These commands create a solr service on port 8983 and 7574, also attach them to Zookeeper “localhost:2128”.

1. Node 1: \node1\solr-7.2.1\bin> solr start -c -p 8983 -z localhost:2181
2. Node 2: \node2\solr-7.2.1\bin> solr start -c -p 7574 -z localhost:2181

• Run command “solr status”, you can see Zookeeper is listening 2 live nodes on ports 8983 and 7574.

• Run Solrssl.ps1 against both node 1 and 2 to add self-signed SSL certificates to Solr instances.

2, Upload the Solr configuration to Zookeeper

• In node 1 folder, locate the configset folder under the “node1\solr-7.2.1\server\solr\configsets” and copy the _default folder to a new sitecore_configs folder.

• In the sitecore_configs/solrconfig.xml file, set the autoCreateFieldssetting to false (update.autoCreateFields:false).

• In this new folder, open the managed-schema file in the conf folder and do the following:

1. Set the value in <uniqueKey>id</uniqueKey> to _uniqueid.
2. In the fields section, add the following field configuration for _uniqueid:

<field name="_uniqueid" type="string" indexed="true" required="true" stored="true"/>

• Go to the “node1\solr-7.2.1\bin” folder under command prompt, run the following command:

solr zk upconfig -d sitecore_configs -n sitecore -z localhost:2181

3, Add collections to SolrCloud farm

• In one of the nodes, navigate to the Collections page in the Solr admin UI, Click Add Collection.

• The collection names need to be matched to with core names in Sitecore index config files. Be aware of the prefix setup in SIF script.

• In the config set field, click the sitecore collection.

• Repeat for all Sitecore indexes.

4, Configure Sitecore to use SolrCloud as search provider

Open the “App_Config\ConnectionStrings.config” file. Specify the connection string key solr.search and add the token solrCloud=true:

<add name="solr.search" connectionString="https://localhost:8983/solr;solrCloud=true" />

5, Populate the Solr schema

• Open the Sitecore Control Panel.

• Click Populate Solr Managed Schema.

• Select all indexes and click Populate.

Sitecore DevOps A-B-C

Back 2 years ago, I had a post Sitecore DevOps 1-2-3. It was mainly based on TFS source control. Now Git repositories becomes more popular and easy adopted from Azure DevOps. I have worked on several projects based on Azure Git repositories. The main flow of Sitecore DevOps pileline remains same. Because of the flexibility of Git flow, branching strategy becomes challenge and most debating area to many organizations. Basically you just need to come up and shape a flow best fit to your team development life cycle.

Here is a branching strategy and Git flow worked well for me.

Starting from a clean Production source code branch, create a Master branch for main stream of development. Then create a Develop branch from Master for integration sandbox. Production, Master and Develop branches are permanent.

Scenario A: Feature A development and Hotfix

1. Developer creates “A” branch from Master then work on it.
2. After local development completed, send PR to Develop branch to test in sandbox.
3. After sandbox test successful, send PR to Master branch.
4. Create QA branch from Master and make deployment
5. After QA successful, create UAT branch from QA and make deployment
6. In UAT, there is issue found, developer work on fix in its own A branch.
7. At the meantime there is a Hotfix happen, round 1 release is put on hold
8. Create a Hotfix branch from Production then work on it.
9. After local development completed, send PR to both Production and Master branches.
10. Developer finish the fix and send anther PR to Master branch
11. Abandon the round 1 QA and UAT branches, recreate them for round 2. This time, the QA and UAT branches contain both feature A and Hot fix.
12. Everything test good in both QA and UAT. Then send UAT PR to Production for release

Scenario 2: Feature B development.

1. Developer create “B” branch from Master then work on it.
2. After local development completed, send PR to Develop branch to test in sandbox.
3. After sandbox test successful, send PR to Master branch.
4. Create QA branch from Master and make deployment
5. After QA successful, create UAT branch from QA and make deployment
6. After UAT successful, it is ready to PR to Production branch for release

So above Scenario A and B are running in parallel on their own paces and procedures. This is how we can achieve continues deployment.

Unable to Update Template Standard Values

When updating template “__Standard Values”, I got issue not be able to update unchecked checkbox, unselected droplink/droplist, even empty single-line text box through TDS update package. After checking the synced serialized item in TDS project, it showed the fields were empty value with content-length 0. When using Sitecore Update wizard to process the package generated by TDS project, it actually ignored these fields.

Based on Sitecore’s explanation, not all Sitecore templates have Standard values(SV) by default and customers/partners often create ones themselves. As a result, In the case where we introduce a SV in a new update and the solution had an existing SV, we need to handle this correctly, customer data in SV item should be preserved. Thus Sitecore development team made a decision to introduce edge case logic here: If the update is adding SV and the item already exists in the solution, the field values are merged to maintain the customer/partners data and changes(e.g. if the value in the package is empty, the data in the field will be left without changes). So this is the expected outcome for standard values with the update wizard.

This is a weird situation. That means the template Standard Values update through Sitecore UI may not be synced to other upper/lower environments by TDS update package which is generated from source code build.

Then what’s the solution? when playing with the serialized item in source,  I found if I removed the empty value field from serialized item file, for example, deleting the field below which unchecked “Enable item fallback” checkbox, it was then processed by Update wizard correctly.

But we can’t rely on manual deleting fields from source file in an automated DevOps process. Then I found an alternative solution: use Sitecore field Reset function instead of regular content UI when such as:

• un-checking Checkbox,
• deselecting Droplink
• deselecting Droplist,
• empty Single-line Text box
• empty Rich Text box

This applies to almost all the field types. It is kind like Reset to original value. Here is how:

• Highlight the “__Standard Values” item, click the “Reset” button under menu “VERSIONS”

• In the reset form, left side is current value, right side is the value will be reset to.

Checkbox

Droplist

Droplink

Single-line Textbox

After save the reset results, then sync the Sitecore “__Standard Values” item to TDS project as normal. They will be actually removed from serialized item in source, then processed properly in Update Wizard.

Sitecore 9 Installation Error – SQL Server Data-Tier Application Framework

During Sitecore 9 installation, no matter standalone or distributed topology, if you see the error like this below, it is mainly related to SQL Server Data-Tier Application Framework (DACFx).

First, make sure you have installed “MS SQL Server Data-Tier Application Framework (DACFx) v17.1”. This is the prerequisite of Sitecore 9 installation. It will deploy Sitecore 9 SQL Server database objects and data with DACPAC file.

Second, check database server version, especially error message like “…..Microsoft.Data.Tools.Schema.Sql.Sql120DatabaseSchemaProvider is not valid…..”. Sitecore 9 requires MS SQL Server 2016 or above. The compatibility level should be 130. Here is the SQL script to check compatibility level.

select name, compatibility_level from sys.databases

Third, if all above are OK, then what’s the issue? This is the hardest part which I was facing. At the end I found there were several different versions of DACFx installed in the server. They might come with SQL Server Management Studio, Visual Studio 2015, 2017. So after registering DACFx path to specify which one to use, then ran the script and it was successful.

 

Sitecore 9 Installation – Distributed

If you have experience on Sitecore 9 installation with standalone instance, the prerequisite of a distributed environment is pretty much similar. You can refer Sitecore 9 Installtion – Standalone for the instruction of Solr, SQL Server and other prerequisites installation.

The main difference of distributed environment installation is using topology XP1.

• Download Sitecore 9.0.2 rev. 180604 (WDP XP1 packages).zip from https://dev.sitecore.net/
• Extract Sitecore 9.0.2 rev. 180604 (WDP XP1 packages).zip to an install folder.
• Extract XP1 Configuration Files 9.0.2 rev. 180604.zip from previous step. Here are the files extracted.

Here is the full scaled Sitecore 9 architecture. Above each install packages and associated configuration files match each instances below.  

Here is a list of instances all installed.

• xConnect Colletion
• xConnect Colletion Search
• xConnect Marketing Automation
• xConnect Marketing Automation Reporting
• xConnect Reference Data
• Sitecore Content Management
• Sitecore Content Delivery
• Sitecore Processing
• Sitecore Reporting

Real installation is very flexible, you can install one instance per server or combine several instances in one server depending on the infrastructure architecture. The most important thing in distributed installation is the configuration of service connection. Here is a relationship between instances and services.

When editing Powershell Script and configuring instances, you can refer above table to make sure all parameters been setup.

• xConnect service instances

#deploy xconnect instance -collection
$xconnectParams = 
@{
Path = "$PSScriptRoot\xconnect-xp1-collection.json" 
Package = "$PSScriptRoot\Sitecore 9.0.2 rev. 180604 (OnPrem)_xp1collection.scwdp.zip"
LicenseFile = "$PSScriptRoot\license.xml"
Sitename = $XConnectCollectionService 
XConnectCert = $certParams.CertificateName 
SqlDbPrefix = $prefix 
SqlServer = $SqlServer 
SqlAdminUser = $SqlAdminUser
SqlAdminPassword = $SqlAdminPassword 
} 
Install-SitecoreConfiguration @xconnectParams -Verbose

#deploy xconnect instance -collectionsearch
$xconnectParams = 
@{
Path = "$PSScriptRoot\xconnect-xp1-collectionsearch.json" 
Package = "$PSScriptRoot\Sitecore 9.0.2 rev. 180604 (OnPrem)_xp1collectionsearch.scwdp.zip"
LicenseFile = "$PSScriptRoot\license.xml"
Sitename = $XConnectCollectionSearchService 
XConnectCert = $certParams.CertificateName 
SqlDbPrefix = $prefix 
SqlServer = $SqlServer 
SolrCorePrefix = $prefix
SolrURL = $SolrUrl 
} 
Install-SitecoreConfiguration @xconnectParams -Verbose

#deploy xconnect instance -MarketingAutomation
$xconnectParams = 
@{
Path = "$PSScriptRoot\xconnect-xp1-MarketingAutomation.json" 
Package = "$PSScriptRoot\Sitecore 9.0.2 rev. 180604 (OnPrem)_xp1marketingautomation.scwdp.zip"
LicenseFile = "$PSScriptRoot\license.xml"
Sitename = $XConnectMarketingAutomation 
XConnectCert = $certParams.CertificateName 
SqlDbPrefix = $prefix 
SqlServer = $SqlServer 
SqlAdminUser = $SqlAdminUser
SqlAdminPassword = $SqlAdminPassword 
} 
Install-SitecoreConfiguration @xconnectParams -Verbose

#deploy xconnect instance -MarketingAutomationReporting
$xconnectParams = 
@{
Path = "$PSScriptRoot\xconnect-xp1-MarketingAutomationReporting.json" 
Package = "$PSScriptRoot\Sitecore 9.0.2 rev. 180604 (OnPrem)_xp1marketingautomationreporting.scwdp.zip"
LicenseFile = "$PSScriptRoot\license.xml"
Sitename = $XConnectMarketingAutomationReporting 
XConnectCert = $certParams.CertificateName 
SqlDbPrefix = $prefix 
SqlServer = $SqlServer 
} 
Install-SitecoreConfiguration @xconnectParams -Verbose

#deploy xconnect instance -ReferenceData
$xconnectParams = 
@{
Path = "$PSScriptRoot\xconnect-xp1-ReferenceData.json" 
Package = "$PSScriptRoot\Sitecore 9.0.2 rev. 180604 (OnPrem)_xp1referencedata.scwdp.zip"
LicenseFile = "$PSScriptRoot\license.xml"
Sitename = $XConnectReferenceData 
XConnectCert = $certParams.CertificateName 
SqlDbPrefix = $prefix 
SqlServer = $SqlServer 
SqlAdminUser = $SqlAdminUser
SqlAdminPassword = $SqlAdminPassword 
} 
Install-SitecoreConfiguration @xconnectParams -Verbose

• Sitecore instance – reporting

$sitecoreParams = 
@{ 
Path = "$PSScriptRoot\sitecore-XP1-rep.json"
Package = "$PSScriptRoot\Sitecore 9.0.2 rev. 180604 (OnPrem)_rep.scwdp.zip" 
LicenseFile = "$PSScriptRoot\license.xml"
SqlDbPrefix = $prefix 
SqlServer = $SqlServer 
Sitename = $sitecoreRepSiteName 
} 
Install-SitecoreConfiguration @sitecoreParams -Verbose

• Sitecore instance – processing

$sitecoreParams = 
@{ 
Path = "$PSScriptRoot\sitecore-XP1-prc.json"
Package = "$PSScriptRoot\Sitecore 9.0.2 rev. 180604 (OnPrem)_prc.scwdp.zip" 
LicenseFile = "$PSScriptRoot\license.xml"
SqlDbPrefix = $prefix 
SqlServer = $SqlServer 
SqlAdminUser = $SqlAdminUser 
SqlAdminPassword = $SqlAdminPassword 
SolrCorePrefix = $prefix 
XConnectCert = $certParams.CertificateName 
Sitename = $sitecorePrcSiteName 
XConnectCollectionService = "https://$XConnectCollectionService" 
} 
Install-SitecoreConfiguration @sitecoreParams -Verbose

• Sitecore instance – content management

$sitecoreParams = 
@{ 
Path = "$PSScriptRoot\sitecore-XP1-cm.json"
Package = "$PSScriptRoot\Sitecore 9.0.2 rev. 180604 (OnPrem)_cm.scwdp.zip" 
LicenseFile = "$PSScriptRoot\license.xml"
SqlDbPrefix = $prefix 
SqlServer = $SqlServer 
SqlAdminUser = $SqlAdminUser 
SqlAdminPassword = $SqlAdminPassword 
SolrCorePrefix = $prefix 
SolrUrl = $SolrUrl 
XConnectCert = $certParams.CertificateName 
Sitename = $sitecoreCmSiteName 
XConnectCollectionSearchService="https://$XConnectCollectionSearchService"
XConnectReferenceDataService= "https://$XConnectReferenceDataService"
MarketingAutomationOperationsService="https://$MarketingAutomationOperationsService"
MarketingAutomationReportingService= "https://$MarketingAutomationReportingService" 
} 
Install-SitecoreConfiguration @sitecoreParams -Verbose

• Sitecore instance – content delivery

$sitecoreParams = 
@{ 
Path = "$PSScriptRoot\sitecore-XP1-cd.json"
Package = "$PSScriptRoot\Sitecore 9.0.2 rev. 180604 (OnPrem)_cd.scwdp.zip" 
LicenseFile = "$PSScriptRoot\license.xml"
SqlDbPrefix = $prefix 
SqlServer = $SqlServer 
SolrCorePrefix = $prefix 
SolrUrl = $SolrUrl 
XConnectCert = $certParams.CertificateName 
Sitename = $sitecoreSiteName 
XConnectCollectionService = "https://$XConnectCollectionService" 
XConnectReferenceDataService= "https://$XConnectReferenceDataService"
MarketingAutomationOperationsService="https://$MarketingAutomationOperationsService"
MarketingAutomationReportingService= "https://$MarketingAutomationReportingService" 
} 
Install-SitecoreConfiguration @sitecoreParams -Verbose

 

The installation has sequence. Install xConnect service instances first, then Sitecore instances.

From license perspective, XP Services (xConnect Collection, xConnect Collection Search, xConnect Reference Data, Marketing Automation, Marketing Automation Reporting) are not licensed Sitecore installs. Same goes for the worker roles (Search Index worker and Marketing Automation Engine worker). CD, CM, PRC, RPT are all ‘licensed’ instances. They count against a Sitecore license. Processing(PRC) and Reporting(RPT) are sometime treated differently than CD/CM in a standard on-premise perpetual instance-based licensing model.

Sitecore 9 Installation – Standalone

Here is a step by step guide of Sitecore 9 on-premise installation as a standalone instance. It is mostly for local workstation and integrated development environment.

Key points of this installation:

• Topology: XP0

• Solr is installed by default. It serves both content search and xConnect services. It requires secure access.
• Sitecore 9 is installed by SIF(Sitecore Installation Framework) PowerShell scripts.

Prerequisites:

• Windows 8.1, 10, Server 2012 R2
• Net Framework 4.6.2 or 4.7
• SQL Server 2016 or above
• IIS 8.5 or 10
• Java Runtime Environment 1.8+(JRE)
• PowerShell 5.1+
• Web Platform Installer 5.0
• Web Administration Module (IIS)
• Web Deploy 3.6 for Hosting Servers
• URL Rewrite 2+
• MS SQL Server Data-Tier Application Framework (17.1)
• MS SQL Server Transact-SQL ScriptDom

Download and Collect Installation Related Files:

• Sitecore License.xml
• Download Sitecore 9.0.2 rev. 180604 (WDP XP0 packages).zip from https://dev.sitecore.net/
• Extract Sitecore 9.0.2 rev. 180604 (WDP XP0 packages).zip to an install folder. For example, C:\Sitecore\sc902.
• Extract XP0 Configuration Files 9.0.2 rev. 180604.zip from previous step to C:\Sitecore\sc902.
• Download Solr 6.6.3 from http://archive.apache.org/dist/lucene/solr/6.6.3/ and extract to C:\Sitecore\solr-6.6.3
• Download NSSM from https://nssm.cc/download and extract to C:\Sitecore\nssm
• Here are the files collected.

Install Solr:

• Install JRE
• Download Java SE Runtime Environment 8+ from http://www.oracle.com/technetwork/es/java/javase/downloads/jre8-downloads-2133155.html
• Setup JAVA_HOME variables

• Start Solr – run following script in Windows PowerShell as Administrator, then check solr instance at http://localhost:8983/solr

cd C:\Sitecore\solr-6.6.3\bin

.\solr start -p 8983

• Stop Solr – run following script in Windows PowerShell as Administrator

.\solr stop -p 8983

• Set up Solr as service – run below script in Command Prompt as Administrator, then fill the installer as the picture below. Once done, verify the service running from Service Manager.

cd C:\Sitecore\nssm\win64

nssm install solr6

• Add SSL to Solr instance (generate self-signed certificate and key)
  • Save following script to file “solrssl.ps1” (from Kam Figy’s blog: https://kamsar.net/index.php/2017/10/Quickly-add-SSL-to-Solr/).

 

param(
[string]$KeystoreFile = 'solr-ssl.keystore.jks',
[string]$KeystorePassword = 'secret',
[string]$SolrDomain = 'localhost',
[switch]$Clobber
)

$ErrorActionPreference = 'Stop'

### PARAM VALIDATION
if($KeystorePassword -ne 'secret') {
Write-Error 'The keystore password must be "secret", because Solr apparently ignores the parameter'
}

if((Test-Path $KeystoreFile)) {
if($Clobber) {
Write-Host "Removing $KeystoreFile..."
Remove-Item $KeystoreFile
} else {
$KeystorePath = Resolve-Path $KeystoreFile
Write-Error "Keystore file $KeystorePath already existed. To regenerate it, pass -Clobber."
}
}

$P12Path = [IO.Path]::ChangeExtension($KeystoreFile, 'p12')
if((Test-Path $P12Path)) {
if($Clobber) {
Write-Host "Removing $P12Path..."
Remove-Item $P12Path
} else {
$P12Path = Resolve-Path $P12Path
Write-Error "Keystore file $P12Path already existed. To regenerate it, pass -Clobber."
}
}

try {
$keytool = (Get-Command 'C:\Program Files\Java\jre1.8.0_172\bin\keytool.exe').Source
} catch {
$keytool = Read-Host "keytool.exe not on path. Enter path to keytool (found in JRE bin folder)"

if([string]::IsNullOrEmpty($keytool) -or -not (Test-Path $keytool)) {
Write-Error "Keytool path was invalid."
}
}

### DOING STUFF

Write-Host ''
Write-Host 'Generating JKS keystore...'
& $keytool -genkeypair -alias solr-ssl -keyalg RSA -keysize 2048 -keypass $KeystorePassword -storepass $KeystorePassword -validity 9999 -keystore $KeystoreFile -ext SAN=DNS:$SolrDomain,IP:127.0.0.1 -dname "CN=$SolrDomain, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country"

Write-Host ''
Write-Host 'Generating .p12 to import to Windows...'
& $keytool -importkeystore -srckeystore $KeystoreFile -destkeystore $P12Path -srcstoretype jks -deststoretype pkcs12 -srcstorepass $KeystorePassword -deststorepass $KeystorePassword

Write-Host ''
Write-Host 'Trusting generated SSL certificate...'
$secureStringKeystorePassword = ConvertTo-SecureString -String $KeystorePassword -Force -AsPlainText
$root = Import-PfxCertificate -FilePath $P12Path -Password $secureStringKeystorePassword -CertStoreLocation Cert:\LocalMachine\Root
Write-Host 'SSL certificate is now locally trusted. (added as root CA)'

Write-Host ''
Write-Host '########## NEXT STEPS ##########' -ForegroundColor Green
Write-Host ''
Write-Host '1. Copy your keystore to $SOLR_HOME\server\etc (MUST be here)' -ForegroundColor Green

if(-not $KeystoreFile.EndsWith('solr-ssl.keystore.jks')) {
Write-Warning 'Your keystore file is not named "solr-ssl.keystore.jks"'
Write-Warning 'Solr requires this exact name, so make sure to rename it before use.'
}

$KeystorePath = Resolve-Path $KeystoreFile
Write-Host ''
Write-Host '2. Add the following lines to your solr.in.cmd:' -ForegroundColor Green
Write-Host ''
Write-Host "set SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.jks" -ForegroundColor Yellow
Write-Host "set SOLR_SSL_KEY_STORE_PASSWORD=$KeystorePassword" -ForegroundColor Yellow
Write-Host "set SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.jks" -ForegroundColor Yellow
Write-Host "set SOLR_SSL_TRUST_STORE_PASSWORD=$KeystorePassword" -ForegroundColor Yellow
Write-Host ''
Write-Host 'Done!'

• Get it to your local and modify the “keytool.exe” path inside if need(line 37), see below.

• • Run follwing command in Windows PowerShell as Administrator.

cd C:\Sitecore\sc902

.\solrssl.ps1 -KeystoreFile C:\Sitecore\solr-6.6.3\server\etc\solr-ssl.keystore.jks

• • Enable a section in file “C:\Sitecore\solr-6.6.3\bin\solr.in.cmd” (line 94-97), see below.
  • Re-start the Solr service and check instance at https://localhost:8983/solr.

Install and Configure SQL Server:

• The SQL Server installtion is pretty much same as Sitecore previous version installation.
• When creating a SQL Server DB user, make sure it has “sysadmin” role. The login Id and password will be used in Sitecore installation script next.
• Enable Contained Database Authentication – run following sql script in SQL Management Studio

sp_configure 'contained database authentication', 1; GO RECONFIGURE; GO

Install Sitecore with SIF:

Sitecore Installation Framework(SIF) is a PowerShell module for handling the installation and configuration of Sitecore instances. It will install and configure Sitecore eco-system.

Run following script in Windows PowerShell as Administrator and type “Y” to continue if need.

• Register Sitecore Repository

Register-PSRepository -Name SitecoreRepo -SourceLocation https://sitecore.myget.org/F/sc-powershell/api/v2

• Install Framework module

Install-Module SitecoreInstallFramework

• Install Fundamentals module

Install-Module SitecoreFundamentals

• Import above modules

Import-Module SitecoreFundamentals

Import-Module SitecoreInstallFramework

• Configure the PowerShell script below and save as “InstallSitecore9.ps1”. Before running, check and update parameters values to you environment.

#define parameters 
$prefix = "sc90"
$PSScriptRoot = "C:\sitecore\sc902"
$XConnectCollectionService = "$prefix.xconnect"
$sitecoreSiteName = "$prefix.local"
$SolrUrl = "https://localhost:8983/solr"
$SolrRoot = "C:\sitecore\solr-6.6.3"
$SolrService = "solr6"
$SqlServer = ".\SQLEXPRESS"
$SqlAdminUser = "sc9user" # replace it with your local SQL Server installation
$SqlAdminPassword = "Test12345" # replace it with your local SQL Server installation

#install client certificate for xconnect 
$certParams = 
@{ 
Path = "$PSScriptRoot\xconnect-createcert.json" 
CertificateName = "$prefix.xconnect_client" 
} 
Install-SitecoreConfiguration @certParams -Verbose

#install solr cores for xdb 
$solrParams = 
@{
Path = "$PSScriptRoot\xconnect-solr.json" 
SolrUrl = $SolrUrl 
SolrRoot = $SolrRoot 
SolrService = $SolrService 
CorePrefix = $prefix 
} 
Install-SitecoreConfiguration @solrParams -Verbose

#deploy xconnect instance 
$xconnectParams = 
@{
Path = "$PSScriptRoot\xconnect-xp0.json" 
Package = "$PSScriptRoot\Sitecore 9.0.2 rev. 180604 (OnPrem)_xp0xconnect.scwdp.zip"
LicenseFile = "$PSScriptRoot\license.xml"
Sitename = $XConnectCollectionService 
XConnectCert = $certParams.CertificateName 
SqlDbPrefix = $prefix 
SqlServer = $SqlServer 
SqlAdminUser = $SqlAdminUser
SqlAdminPassword = $SqlAdminPassword
SolrCorePrefix = $prefix
SolrURL = $SolrUrl 
} 
Install-SitecoreConfiguration @xconnectParams -Verbose

#install solr cores for sitecore 
$solrParams = 
@{
Path = "$PSScriptRoot\sitecore-solr.json"
SolrUrl = $SolrUrl
SolrRoot = $SolrRoot
SolrService = $SolrService 
CorePrefix = $prefix 
} 
Install-SitecoreConfiguration @solrParams -Verbose

#install sitecore instance 
$sitecoreParams = 
@{ 
Path = "$PSScriptRoot\sitecore-XP0.json"
Package = "$PSScriptRoot\Sitecore 9.0.2 rev. 180604 (OnPrem)_single.scwdp.zip" 
LicenseFile = "$PSScriptRoot\license.xml"
SqlDbPrefix = $prefix 
SqlServer = $SqlServer 
SqlAdminUser = $SqlAdminUser 
SqlAdminPassword = $SqlAdminPassword 
SolrCorePrefix = $prefix 
SolrUrl = $SolrUrl 
XConnectCert = $certParams.CertificateName 
Sitename = $sitecoreSiteName 
XConnectCollectionService = "https://$XConnectCollectionService" 
} 
Install-SitecoreConfiguration @sitecoreParams -Verbose

• Open the script file “InstallSitecore9.ps1” in Windows PowerShell ISE to execute or copy the scripts to run in Windows PowerShell as Administrator.
• Confirm Sitecore 9 installed successfully.

Sitecore CMS-Only Mode and Submit Queue

Prior to version 9, Sitecore use MongoDb to store analytics data. If you don’t need analytics features and don’t want to maintain MongoDb,  you may like to configure Sitecore as CMS-only mode. Based on Sitecore document, it is pretty straightforward.

In Sitecore configuration file “Sitecore.Xdb.config”

<setting name="Xdb.Enabled" value="false" />

It prevents the data from being saved to the database. Now it is safe to remove MongoDb connection strings.

But the tracking is still happening. there is a Sitecore ‘Submit Queue’ that flushes captured data to the server disk (the ‘Data\Submit Queue’ folder by default). It expects MongoDb comes back online, a job process submits the data to MongoDb from the disk “Submit Queue” folder.

But this may cause performance issue to authoring server and even possible item publish error when server runs for a period. To solve this,

In file “Sitecore.Xdb.config”

<setting name="Xdb.Tracking.Enabled" value="false" />

In file “Sitecore.Analytics.Tracking.config”.

<setting name="Analytics.UseSubmitQueue" value="false" />

Sitecore SPEAK: ProgressBar Event Trigger

In Sitecore SPEAK development, the ProgressBar control displays a horizontal bar. It can be specified a value between 0 and 100, and the control displays part of the bar in color to indicate the level of progress.

How to update the progress? This control has a property “UpdateInterval” which specify an update interval in milliseconds. The ProgressBar control raises an event at the end of each of these intervals. The event is called intervalCompleted + the ID of the control. custom code can listen for this event in page code. For example the control ID is “ProgressBar1”. Here is the code.

initialized: function () {
 this.ProgressBar1.on("intervalCompleted:ProgressBar1", this.intervalCompletedprgBar, this);
 },

intervalCompletedprgBar: function ()
 {
 var currBar = this.ProgressBar1.attributes.value;
 this.ProgressBar1.set("value", currBar+(100 - currBar) / 5);
 }

The event is set during code initialized and supposed to be triggered at the end of each of intervals. But the event has never been triggered. After contacting Sitecore Support several rounds, at the end it was reported and registered as a bug in Sitecore bug tracking system. To track the future status of this bug report, use the reference number 198936.

Is there any workaround? Yes. Here is a solution.

This is the Sitecore outbox ProgressBar JavaScript: \sitecore\shell\client\Business Component Library\Layouts\Renderings\Common\ProgressBars\ProgressBar.js

setupTimer: function () {      
         var updateInterval = this.model.get("updateInterval");      
         clearInterval(this.timer);      
         if (updateInterval <= 0) {       
               return;      
         }      
         var id = this.$el.attr("data-sc-id");      
         this.timer = setInterval(function () {        
              _sc.trigger("intervalCompleted:" + id);     
         }, updateInterval);
},

Highlight line code suppose to trigger ProgressBar event, but it doesn’t. The workaround is, replace it with calling progress bar update function directly. The only thing need to be aware is future Sitecore upgrade.

setupTimer: function () {      
         var updateInterval = this.model.get("updateInterval");      
         clearInterval(this.timer);      
         if (updateInterval <= 0) {       
               return;      
         }      
         var id = this.$el.attr("data-sc-id");      
         this.timer = setInterval(function () {        
              //_sc.trigger("intervalCompleted:" + id); 
              _sc.app.intervalCompletedprgBar(); 
         }, updateInterval);
},

 

Sitecore Request Validation Configuration

This was an interesting experience and definitely I’d like to share the tricks learned during this website developed in Sitecore for one of our clients.

This site was migrated from an existing one. Everything was fine in UAT/staging and ready for release.  But when switching the DNS, instead of seeing beautiful home page, there was ugly yellow/red .net error message showing up like:

HttpRequestValidationException: A potentially dangerous Request.Cookies value was detected from the client… 

Immediately, questions came up, what’s that? why?

Since the message mentioned cookies, then I checked browser and found there was a cookie created by previous site(migrated from) when I browsed it. The cookie value contained html tags like “<b>”, “<br>”, even they were harmless. Now it was clear that the new site page request was blocked by .Net Http Runtime request validation.

Sitecore is a platform built upon .Net framework. Request validation is a feature in ASP.NET that examines an HTTP request and determines whether it contains potentially dangerous content. In this context, potentially dangerous content is any HTML markup or JavaScript code in the body, header, query string, or cookies of the request. ASP.NET performs this check because markup or code in the URL query string, cookies, or posted form values might have been added for malicious purposes.

Based on Microsoft HttpRuntimeSection.RequestValidationMode Property Document, here are available settings:

• 4.5(the default). In this mode, values are lazily loaded, that is, they are not read until they are requested.
• 4.0. The HttpRequest object internally sets a flag that indicates that request validation should be triggered whenever any HTTP request data is accessed. This guarantees that the request validation is triggered before data such as cookies and URLs are accessed during the request. The request validation settings of the element (if any) in the configuration file or of the directive in an individual page are ignored.
• 2.0. Request validation is enabled only for pages, not for all HTTP requests. In addition, the request validation settings of the element (if any) in the configuration file or of the directive in an individual page are used to determine which page requests to validate.

When I checked the installed Sitecore instance web.config, by default the “requestValidationMode” was set to 4.0, like below:

<httpRuntime targetFramework="4.5.2" maxRequestLength="512000" executionTimeout="3600" enableKernelOutputCache="false" relaxedUrlToFileSystemMapping="true" requestValidationMode="4.0" enableVersionHeader="false" />

After I changed it to 4.5 which is lazy loading, the site displayed fine. Then we went through all code to make sure the request validation were handled at page level as requested.