Sitecore DevOps 1-2-3

DevOps is a hot topic in today’s software development along with Agile methodology. What does DevOps mean?

The term DevOps is commonly considered a combination of the concepts of development and operations. It is used in IT to refer to roles or processes that bridge various department – usually development and operations teams – to achieve a certain project management philosophy that involves more efficiency in communications between development teams and other parts of a greater business or organization.

– Techopidia

From technical perspective, the key points of DevOps are continue integration and auto-deployment. In real world, we always experience functionalities and features in different stages of development life cycle cross all environments: Dev, QA, Staging, UAT, Production. Sitecore even comes additional complexity with various instances CM, CD, Processing, Reporting, etc.

Below is an overview of Sitecore development life cycle and operation procedure with different stages audience and roles.

Sitecore Development and Publish Procedure

How can DevOps orchestrate Sitecore implementation from various departments and deploy code and items to different environments/instances? Not using Sitecore Ship, Unicom, Gulp, Grunt, Git, TeamCity, you name it, I came up a simple way to create Sitecore DevOps CI/CD procedure with popular .Net development IDE and tools:

  • Visual Studio
  • Team Foundation Server
  • SlowCheetah
  • TDS

First, create code solution based on Sitecore Helix principle and Habitat architecture approach.

This is fundamental to entire DevOps. (Ref: http://helix.sitecore.net/). Because Sitecore platform multitenancy approach results all client websites, applications and components built within Sitecore developed in single code base. So the Modular Architecture is Sitecore recommended best practice. It moves away from the traditional 3 tiered architecture (UI, business logic, data). It breaks the solution into smaller independent and manageable parts. Each module is self-contained with business logic, UI and TDS Sitecore items. It can be built, packed and shipped independently without interfering other modules causing massive regression test. The module should only depend upon modules that are more stable than it.

Second, create build definitions with branch change sets and transformations.

This will differentiate feature stages and environment settings during continue integration and continue deployment. By using TFS merge and branch function, we can move features by checked-in change sets, see below.Sitecore_DevoOps2

In Visual Studio, create configuration profiles for each Sitecore environment: Dev; StgCM; StgCd1; StgCd2; ProdCM; ProdCd1; ProdCd2, etc. Add SlowCheetah extension to Visual Studio.  It will transform all custom Sitecore configuration files at build time based on profile.

Third, use TDS Sitecore Package Deployer to automatically process Sitecore update packages.

  • In Visual Studio TDS project properties, enable “Generate package during build” according to the configuration profile.
  • Create a folder “SitecorePackageDeployer” under “Data”.
  • Install Sitecore Package Deployer. It includes a scheduled task run by the Sitecore task agent. This task runs one per minute and looks for packages in the folder created above
  • In TFS build definition and release task, make sure the generated web code published to Sitecore “Website” folder and update packages published to Sitecore “Data\SitecorePackageDeployer” folder.

 

Advertisements
Posted in Sitecore | Tagged , , , , , , , | Leave a comment

Build a Sitecore Module With External Big Data through Solr

Sitecore stores and manages content through Sitecore items. They are actually data saved in relational SQL Sever database. Normally the content managed in Sitecore are better created as Sitecore items. But in today’s enterprise ecosystem, many external data cannot be created into Sitecore because:

  • There are existing applications managing the external data, no need to create save function in Sitecore. For example, PIM(Product Information/Inventory Management)
  • External data is huge and ETL might take long time to sync data to Sitecore. For example product inventory data in retail business by SKU, Home Depot sells 300K+, BestBuy sells 725K+; in travel booking system by vacation package, Sunwing sells 10M+
  • External data changes very frequently. For example dynamic price of flight ticket and hotel room, real time merchandise inventory.

So the right approach is,

  • Integrate with external data, no ETL to Sitecore
  • Conduct real time search against external data through Solr

Solr is the popular, blazing-fast, open source enterprise search platform built on Apache Lucene. Installing Solr is easy, see Learn more about Solr.  Here is an example with key points to create a core in Solr and configure index to a table in SQL Server.

solr1

  • solrconfig:
    <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" /> 
    
    <lib dir="${solr.install.dir:../../../..}/dist/" regex="sqljdbc42\.jar" />
  • managed-schema:
    <field name="ProductID" type="long" indexed="true" required="false" stored="true"/>
    
    <field name="Name" type="string" docValues="true" indexed="true" stored="true"/>
    
    <field name="ProductNumber" type="string" docValues="true" indexed="true" stored="true"/>
  • db-data-config:
    <dataSource driver="com.microsoft.sqlserver.jdbc.SQLServerDriver" url="jdbc:sqlserver://localhost:8983;databaseName=Northwind" user="sa" password="xxxxxx" />
    
    <entity name="item"  query="select * from Products"  deltaQuery="select ProductID from Products and updateDate > '${dataimporter.last_index_time}‘”  deltaImportQuery=“” deletedPkQuery=“” transformer="RegexTransformer,DateFormatTransformer,TemplateTransformer"> </entity>

SolrNet is an Apache Solr client for .NET to interact with the Solr search engine. Sitecore installation package officially supports Lucene for native and Solr for distributed indexing. Since Sitecore 8.2 the SolrNet is included in package which means SolrNet has been fully test by Sitecore. Programming with SolrNet is easy. Here is an example with steps:

  • Initialize:

     

    Startup.Init<Bike>("http://localhost:8983/solr/Bike");    
    
    ISolrOperations<Bike> solr =  ServiceLocator.Current.GetInstance<isolroperations<Bike>>();
  • Query in SolrNet:
    var qSize = new SolrQueryByRange<int>("Size", 40, 52);
    
    var qSubCat = new SolrQueryByField("ProductSubcategory", "Mountain Bikes");
    
    var qColor = new SolrQueryInList("Color", "Silver", "Black");
    
    SolrQueryResults<Bike> bikes = solr.Query(qSize && qSubCat && qColor);
  • Query Converted to Solr REST Api:
    http://localhost:8983/solr/Bike/select?fq=Color:((Silver)%20OR%20(Black))&fq=ProductSubcategory:%20%22Mountain%20Bikes%22&fq=Size:[40%20TO%2052]&indent=on&q=*:*&wt=json
Posted in Information Technology, Sitecore, Solr | Tagged , , , , , | Leave a comment

A trick to set up 301 url redirect in Sitecore

Recently I am working on a task to redirect whole bunch of page urls to a new location. I want to preserve all of the links from search engines. So I need to make sure that the old links give a 301 (Moved Permanently) response back. This will allow the web crawlers to update their databases with the new links as well as keep the old links working in the meantime.

I downloaded the Sitecore 301 redirect module from marketplace. It comes up with three types redirect setting:

  • Exact Match Redirects
  • Regular Expression Redirects
  • Rules based Redirects

It sounds very straightforward. I started to set a redirect with type “Exact Match Redirects”. For example:

http://somesite.com/en/oldpage.aspx –> Sitecore/Content/somesite/newpage.

But no matter how I tried, the redirect just didn’t happen. What’s wrong?

After reading the document and tried several more different scenarios, I found how the redirect module works. Sitecore allows customizing the rendering pipeline which is exactly the redirect module does. This module kicks in right after Sitecore tries to resolve the item for the request. If Sitecore resolves the item, this module doesn’t do anything. Otherwise, it runs through the rules in the above 3 matching order.

One important thing I found is, Sitecore’s language processor resolves the language before rendering happens by retrieving the language flag in url. That means when this redirect modules kicks in to compare the match url, the language has already been solved and the flag in url is removed. For example above old url becomes http://somesite.com/oldpage.aspx. That’s why my previous redirect settings failed. After I removed the language flag from the url, it succeeded.

Another tip to make the redirect work is, make sure the old link is dead/disabled.

Posted in Information Technology, Sitecore | Tagged , , , , | Leave a comment

An Issue found in Sitecore User Profile Storage

Recently one of our content authors started to get error like this very frequently.

“Length of LOB data (65608) to be replicated exceeds configured maximum 65536. Use the stored procedure sp_configure to increase the configured maximum value for max text repl size option, which defaults to 65536. A configured value of -1 indicates no limit, other that the limit imposed by the data type….”

sitecore-sql1

It only happens to her account, not others. She manages the site content for about one year. From the error message itself, it is very clear that the SQL Server max text replication need to be increased from default 65536 to a bigger size. I know this SQL server configuration change will fix the issue. But the strange part is what cause the issue and why only happen to specific user account?

I start to trace from SQL Server which Sitecore data is stored. The default setting of “Max Text Replication Size” in SQL Server is 65536 bytes = 0.0625MB. This configuration specifies the maximum size of data that can be added to a replicated column in a single INSERT, UPDATE, WRITETEXT or UPDATETEXT statement. This applies to data type text, ntext, varchar (max), nvarchar (max), varbinary (max), xml, and image.

sitecore-sql3

Sitecore has 3 major SQL databases Core, Master and Web. Since this error happens on Sitecore console application to specific user account, I checked the Core database first to review those table columns with data type in text, ntext, varchar (max), nvarchar (max), varbinary (max), xml, and image. One table come to my eyes, it is “aspnet_Profile”.

By default, Sitecore uses ASP.Net Membership Provider to manage authentication and user profile. These tables are in the Core database. The standard Microsoft implementation of the SqlProfileProvider (which is used in Sitecore by default) stores the user profile information in the “aspnet_Profile” table.

SqlProfileProvider persists profile properties in three fields of the “aspnet_Profile” table: PropertyNames, PropertyValuesString, and PropertyValuesBinary.

The following is a synopsis of what’s stored in each field:  PropertyNames holds a string value containing information about the profile property values present in the PropertyValuesString and PropertyValuesBinary fields. The string holds a colon-delimited list of items; each item denotes one property value, and it is encoded in the following format: Name:B|S:StartPos:Length

PropertyValuesBinary is the field I’d like to check further to determine if it grows wild as it is image data type. Then I use following query to watch the data change in the “aspnet_Profile” table.

SELECT [UserId],[PropertyNames],[PropertyValuesString],
[PropertyValuesBinary],[LastUpdatedDate]  
FROM [dbo].[aspnet_Profile] order by LastUpdatedDate desc

The experiment I did included following activities:

  • Create page and content
  • Modify the page content in content/experience editors
  • Submit page content for approval as workflow required
  • Publish page content

Then I found the size of PropertyValuesBinary only increase when executing workflow command from the Sitecore Content Editor, Experience Editor or Workbox on single item with comment, see below. Every workflow command execution increased about 159 bytes.

sitecore-sql2

Now it explains me the reason why some author accounts started experiencing the error. After websites live for a length of period depends how active authors’ actions in workflow, such as submitting draft, approving, rejecting, etc. Once the data size of PropertyValuesBinary exceeds 65536, the error will show up. I don’t know why Sitecore saves these type of information in “aspnet_Profile” table as binary data, and don’t find how I can reduce or remove them from application console. So here are two solutions: run the following script in Core database to increase the limitation, or create a new account for the author to replace her previous one.

exec sp_configure ‘max text repl size’, 2147483647
Go
RECONFIGURE
GO
Posted in Information Technology, Sitecore | Tagged , , , , , , | Leave a comment

Make ReactJS work in Sitecore Environment

 

In today’s front end landscape, the JavaScript technology is playing a very important role in UI and UX.   It is changing very quick and new technology is coming out every month. Currently you probably know the two most popular JS libraries: Angular and React. In Sitecore development, JQuery is very popular adopted pretty much from beginning as part of .Net development. So question is, can we have most modern JS libraries like Angular and React work with Sitecore?

Yes. Before going to the detail, I’d like to simply compare the difference between Angular and React in a high level. In my opinion, Angular is more like a framework. To take it’s advantage, the entire system is better designed to work in Angular way. But React is more like a design pattern. It solves front-end view only and system can be designed in its own way. So to work with Sitecore, I prefer React personally, especially with ReactJS.Net.

The best practice of Sitecore is component-based design or modular architecture. In a Sitecore .Net MVC environment, the base UI element is rendering which is a group of data model, business controller, logic action and view. React is component-based too. It builds encapsulated components that manage their own state, then compose them to make complex UIs. React components implement a render() method that takes input data and returns what to display. You can see both Sitecore and React use the similar term to describe UI element “Rendering” or “Render”.

ReactJS.Net makes .Net developers to work with React and JSX much easier. Thanks Babel to transform JSX to JavaScript and cached server side automatically on-the-fly. It supports ECMAScript 6 features, Server-side rendering and ASP.Net bundling and minification.

Here is a sample I applied React in Sitecore. There is an application which splits traffic flow randomly by percentage. The visitors IP information are stored in database. The requirement is, streaming the traffic split information at real time and display in a grid with additional information by looking up MaxMind Geo data service. It has a drop down search box for filtering the results.

  • Use NuGet to install the ReactJS.NET (MVC 4 and 5) package.
  • Create MVC controller, action and view
  • Create React component in JSX with proper name space for self-contained scope to avoid conflict variables defined globally.
    • main.jsx
      var Traffics = trafficMonitorApp.Components.Traffics
      
      ReactDOM.render(
       <Traffics url="/Home/Traffics" pollInterval={5000} />,
       document.getElementById('content')
      );

      react3

  • Similar to MVC view and partial view, split React component to structured renders for better management.

react2

  • Load data from server-side action initially or retrieve from Web API through AJAX call at real time.
  • Bundle the JSX files in BundleConfig.cs.react1
  • Invoke the JSX bundle from view.
    @Scripts.Render("~/bundles/main")
  • Define the rendering in Sitecore as normal controller or view rendering.

Here is the completed component:

react4

Posted in Information Technology, ReactJS, Sitecore | Leave a comment

Create Custom Pipeline Processor for Queryable Source of Sitecore Rendering Parameter Template Field

When defining the lookup source of Droplink or Treelist in a Sitecore template, It is very straightforward to use Sitecore query to specify the root path and template Id/name in a regular standard template. For example:

query:./ancestor-or-self::*[@@templatename='MyTemplate']

But it doesn’t work in Rendering Parameter Template as not querying in the context content. It fails and just list entire content tree.

Here is a scenario in Sitecore multisites:

  • Global level content is created and cloned to several sites with same structure, same item names and same templates
  • Each site has function using shared common rendering parameter template.
  • There are Droplink/Treelist fields in the rendering parameter template.
  • The source of Droplink/Treelist fields are defined in lookup of cloned content in the scope of current site.

The key to solve the issue is, make sure the queryable lookup is in the current site context content. Let’s create a custom processor to add to Pipelines to tell Sitecore to get context item when lookup in rendering parameter template.

using System.Linq;
using Sitecore.Data;
using Sitecore.Pipelines.GetLookupSourceItems;
using Sitecore.Shell.Applications.ContentEditor;
using Sitecore.Text;
using Sitecore.Web;

namespace MySitecoreProject.App_Help.LookupSource
{
   public class ParameterTemplateSourceQueries
   {
      // Standard Rendering Parameters template ID
      private readonly ID _baseParameterTemplate = new ID("{8CA06D6A-B353-44E8-BC31-B528C7306971}");

      public void Process(GetLookupSourceItemsArgs args)
      {
         // Only patch to Sitecore query
         if (!args.Source.StartsWith("query:"))
         {return; }

         if (args.Item.Template.BaseTemplates.All(bt => bt.ID != _baseParameterTemplate))
         {return;}

         var url = WebUtil.GetQueryString();
         if (string.IsNullOrWhiteSpace(url) || !url.Contains("hdl"))
         {return;}

         var parameters = FieldEditorOptions.Parse(new UrlString(url)).Parameters;

         var currentItemId = parameters["contentitem"];
         if (string.IsNullOrEmpty(currentItemId))
         {return;}

         var contentItemUri = new ItemUri(currentItemId);
         var contextItem = Database.GetItem(contentItemUri);

         args.Item = contextItem;
       }
    }
}

Create a configuration file under “App_Config\Include” and add processor patch to “getLookupSourceItems” before “Sitecore.Pipelines.GetLookupSourceItems.ProcessQuerySource”.

<pipelines>
    <!-- Lookup Source -->
   <getLookupSourceItems>
       <processor patch:before="*[@type='Sitecore.Pipelines.GetLookupSourceItems.ProcessQuerySource, Sitecore.Kernel']" type="MySitecoreProject.App_Help.LookupSource.ParameterTemplateSourceQueries, MySitecoreProject" />
   </getLookupSourceItems>
</pipelines>
Posted in Information Technology, Sitecore | Tagged , , , , , | 2 Comments

Understand the Operation Impacts of Multisites on Single Sitecore Instance

Sitecore license is instance based. In a scaling enterprise environment with DR and HA capability, normally there are multiple CM, CD and Processing servers cross data centres, either active or hot standby(passive mode).  We must license each Sitecore instance in each environment. To host multiple websites, it is almost impossible to host one website per instance, which will double or triple the number of licenses required.

Sitecore platform is designed to manage multiple logic websites in a single instance. What does it mean to interpret development and staging environment? What is the impact to business?

Single Code Base, One Instance, Same Application Pool

Based on this, all websites, applications and components built within Sitecore are developed in single code base and hosted on one instance managed by same application pool. I have a previous post about how multisites solutioned in Sitecore.

Multisites Solution in Single Sitecore Instance

There are also many articles described the challenges imposed by IIS(app pool, SSL certificate, Robots.txt, etc.). For example John West, Sitecore Multisite, Part 1: IIS Challenges

Here I mainly summarised the challenges and impacts to development and business operation, also the measures we came up as guidline.

Impact Areas Challenges Measures
Development Flexibility Single code base requires developers to work with shared libraries between various projects. It is not flexible for developers to apply new or upgrade technology libraries •Assess the architecture solution before project starts

•Rule of KISS. Develop applications based on existing libraries unless necessary

System  Evolvability Certain application technology evolving may require common library upgrade which result non-related applications re-factory •Assess the necessary of the upgrade

•Re-factory the applications necessarily

Production Maintainability •Any functional update to any site will reset the app pool which will cause all sites restart

•Any content publish to any site will clear the html cache of all sites

•Increase CD nodes in production environment to separate sites by group in different hosting server instances.

•Schedule different release or publishing time frame

Posted in Information Technology, Sitecore | Tagged , , , | Leave a comment