Sitecore 8.2 Processing Server - Performance Tuning

Problem

On another Sitecore field experience episode... we performed a Sitecore Upgrade from 8.0.5 to 8.2.5, after that an issue was raised that Sitecore Analytics data and PathAnalyzer reports are not available (out of date/data missing), so we investigated and to make a long story short, based on log errors we identified some issues and after some Sitecore support tickets and research it was determined that we must Rebuild our Reporting database.

That involves a dedicated Processing Server processing xDB (MongoDB) data replicated between 2 geographically separate data centers. Server had 32GB RAM, 8 Core CPU.

Here's the problem discovered after running the RebuildReportingDB:

• Sitecore Processing Server taking an abnormally long time to process data 
• Projected time to process all data is 120 days (based on the History data in Analytics xDB)
• Server not optimally using dedicated resources

Due to many reasons, we can't wait for 120 days. So... we worked with Sitecore support around performance tuning Processing Servers and this article explains the findings. 

In summary, the graph below explains the KPIs, one important one is the end result after tuning:

120 days based on default Sitecore Processing Server configuration down to 4 days with tuned configuration and optimizations!

Important Dependencies

Some things to confirm before running the ReportingDB Rebuild:

• Ensure that all Sitecore servers have been correctly configured according to the role they are supposed to perform (e.g. CM vs CD vs ProcessingServer vs ReportingServer).
  
  Up to and including Sitecore 8.2 this used to be done by enabling/disabling configuration files as specified by Sitecore in an Excel spreadsheet. With Sitecore 9 changes, this can now be done by editing a configuration file(s) that specifies the server role through a setting.
  
  For 8.2, we automated that by re-using a Powershell approach which reads a CSV converted file based on Sitecore's Role Configuration Excel spreadsheet and the PS script enables/disables config files in the App_Config folder. Based on this article:
  [https://www.linkedin.com/pulse/processing-sitecore-config-files-powerhell-robert-senktas/] - Thanks Robert
• Check the Sitecore application log for high error type counts. Do something about them if applicable.
• On running each rebuild, be ready to run the Sitecore SQL Script to copy the Reporting Table data from Primary to Secondary Reporting DB (or opposite) right after you run rebuild using the Admin page: /sitecore/admin/rebuildreportingdb.aspx. By default (configurable) there's a 10 minute window to do this before the Processing Server starts its job (after clicking the Rebuild button on the admin page.).
• Have your Processing Server close to the Primary Mongo source (i.e. within same Data Center/Network).

Optimal Performance Configuration

In our scenario this was the optimal settings:

• Configured the Maximum Batch Size as 1024 KB
• Configured the applicable Sitecore Aggregation Background Processes to use 8 Max Threads each

My next BLOG article will explain the tuning of configuration, what, where and explain findings.

Cheers /F

PS: Thanks to Sitecore Support for your assistance!