This is the presentation deck file that I gave at CRMUG Summit 2015 on Oct 16, 2015. In this session, I talked about some best practices that you can use to speed up your Dynamics CRM data migration and integration.
2. #CRMUGSummit | #INreno15
INTRODUCTION - YOUR PRESENTER
Daniel Cai
Principal Developer @KingswaySoft
7 Years of development experiences with Microsoft Dynamics CRM
– Mostly working as a developer
5 times of Microsoft Dynamics CRM MVP since 2011
About KingswaySoft
– A leading provider of data integration solutions for Microsoft Dynamics software and
beyond
• SSIS Integration Toolkit for Microsoft Dynamics CRM
• SSIS Integration Toolkit for Microsoft Dynamics AX
• SSIS Integration Toolkit for Microsoft Dynamics GP
• SSIS Integration Toolkit for Microsoft Dynamics Marketing
• SSIS Integration Toolkit for Microsoft Dynamics NAV
• SSIS Integration Toolkit for Microsoft Dynamics SL
• SSIS Integration Toolkit for Parature
• and many more…
– Thousands of enterprise clients worldwide depend on our integration solutions to
drive their data efficiency
2
3. #CRMUGSummit | #INreno15
Discuss the techniques that you can use to get
data into Microsoft Dynamics CRM faster
Discuss some best practices that you utilize in
your CRM data integration and migration
projects
3
SESSION OBJECTIVES
4. #CRMUGSummit | #INreno15
SSIS Integration Toolkit is used in some occasions in this presentation to
discuss the techniques
– It is a data integration solution developed by KingswaySoft that I work for
– More information available at http://www.kingswaysoft.com/products/ssis-integration-
toolkit-for-microsoft-dynamics-crm
The techniques should be transferrable to any other third-party or home-
grown solutions in a similar fashion, provided an equivalent option is
available
I didn’t invent any tips or tricks, neither did I discover all of them by myself,
they all come from the community that I have been learning. The credits go
to the generous CRM community and some special individuals.
4
DISCLAIMER
7. #CRMUGSummit | #INreno15
Web Service Interfaces are generally slow
– Dynamics CRM is no exception
You are often constrained by the limited time window
– A couple to a few hours time window for your daily integration
– Typically a ~50 hours of time window for the initial data migration
• Friday 9pm to Monday 6am
Your data volume could be significantly large
You always want to be faster!
7
WHAT ARE THE PROBLEMS
9. #CRMUGSummit | #INreno15
Identify Resources
– Technical Resources
Get to know your infrastructure
– Networking
– Server Capacities
Define a baseline benchmark
– How many records in a second are required in order to meet the time window allowed
Define the infrastructure that can help you achieve the desired benchmark
Decide between Buy vs Write
9
DEFINE YOUR STRATEGIES
10. #CRMUGSummit | #INreno15
Optimized Infrastructure
– Networking
– Firewall
– Security Policies
– Network Load Balancer
Server Capacity (CRM Server, Database Server, Integration
Server)
– CPU
– Memory
– I/O
10
HAVING THE RIGHT INFRASTRUCTURE
11. #CRMUGSummit | #INreno15
CRM Data Import Wizard
Message-driven Integration Solutions
– BizTalk
– Service Bus
– Message Queue
Batch-based ETL Solutions
– SSIS
– Scribe
– Informatica
– IBM CastIron
11
CHOOSE THE RIGHT TOOLS
12. #CRMUGSummit | #INreno15
Each tool works somewhat differently
Establish a good understanding of your tools
– Know the capabilities
– Know the different options to achieve a particular
purpose
– Know the techniques that can be used to help achieve
better performance
12
TRY TO KNOW YOUR TOOLS
14. #CRMUGSummit | #INreno15
MULTI-THREADING
Make sure to update your .config file in order to
overcome the connection limit of concurrent
service calls
<configuration>
...
<system.net>
<connectionManagement>
<add address="*" maxconnection="100"/>
</connectionManagement>
</system.net>
</configuration>
TheSSIS way
16. #CRMUGSummit | #INreno15
BULK API
Bulk API is more beneficial when
network latency is high
There is an API throttling with CRM
Online
– At most 2 concurrent ExecuteMultiple
requests per organization
How to achieve this?
• Simply enter a Batch Size in SSIS destination
component.
17. #CRMUGSummit | #INreno15
The less fields, the better performance
Some fields may require additional services calls using special SOAP messages when
writing to CRM
– statecode/statuscode
– ownerid
– some more
• parentsytemuserid and businessunitid (systemuser entity)
• businessunitid (team and equipment entities)
• Parentbusinessunitid (businessunit entity)
– Those special service calls are not necessary if it is an Update and you are using v7.1 or later
• For Create or Upsert calls, you still need the additional service calls (as of v7.1).
Any fields registered for CRM plugins/workflows or auditing will have certain performance
impact when submitted
17
MINIMIZE THE NUMBER OF FIELDS YOU WORK WITH
18. #CRMUGSummit | #INreno15
Why?
– Smaller SOAP message, which in turn results in less networking time
– Can avoid firing unnecessary CRM plugins/workflows or auditing
that are registered for the fields
How to achieve this?
– Simply turn on “Ignore Unchanged Fields” option in CRM destination
component
18
AVOID SUBMITTING UNCHANGED FIELDS
19. #CRMUGSummit | #INreno15
CRM Diagnostics Tool
– http://CrmServer/tools/diagnostics/diag.aspx
Ideally the latency should be as low as
possible
– Make sure the integration server (or
workstation) is close to the CRM Server
– A poor infrastructure can result in high
network latency, even the servers are close
to each other
Bandwidth should be as high as possible
19
MIND YOUR NETWORK LATENCY
20. #CRMUGSummit | #INreno15
Which servers
– CRM Server
– Database Server
– Integration Server (or workstation)
What to watch
– CPU
– Memory
– I/O
– Other CRM performance counters
20
WATCH OUT RESOURCE USAGES ON SERVERS
21. #CRMUGSummit | #INreno15
Plan CRM database growth
– Have the right server specification to accommodate the growth
– Define data archiving strategies as required
– For initial load, consider sizing the database to the target size after the completion of the load to avoid database growth during
the load
Cleanup PrincipalObjectAccessBase table as required (to remove orphan records)
– http://support.microsoft.com/kb/2664150
• Consider implementing this as a SQL Server agent job to cleanup weekly or monthly to keep your system healthy
Cleanup AsyncOperationBase table as required (to remove completed workflow logs)
– http://support.microsoft.com/kb/968520 for one-time cleanup
• Make sure you implement the index first, otherwise it could a long time to complete depending on how many records you have
• Consider implementing this as a SQL Server agent job to cleanup weekly or monthly to keep your system healthy
– For CRM workflows, consider choosing “Automatically delete completed workflow jobs (to save disk space)” option for automatic
cleanup
– For asynchronous plugin, consider choosing “Delete AsyncOperation if StatusCode = Successful” option for automatic cleanup
21
MANAGE DATABASE GROWTH
22. #CRMUGSummit | #INreno15
If your data integration or migration process involves any querying, you should review your query performance
Depending on the situation, if any query involves non-indexed CRM database field(s), you might want to add custom
index(es)
– CRM On-Prem, add custom indexes to CRM database directly
– CRM Online, request adding custom indexes by raising a support ticket. Alternatively, use the following techniques to add custom
indexes automatically.
• Add the field to CRM quick find view
• Create CRM alternate key – will have to be unique
Note that having an excessive number of indexes in one database table (entity) can adversely affect your performance
Where to start from?
– Look for long running queries in CRM Server Windows Log
• CRM logs any database queries that take longer than 10s
– SQL Server “Missing Index” tables
Query Tuning
– Show Actual Execution Plan – SQL Server Management Studio (SSMS)
– SQL Server Database Engine Tuning Wizard
22
MANAGE DATABASE INDEXES
23. #CRMUGSummit | #INreno15
Reschedule CRM Maintenance Jobs
– By default, the time when CRM maintenance jobs kick off
depends on when the server was installed (or the
organization being provisioned) – which could be in busy
business hours
– Ideally your integration job kicks off shortly after the
maintenance job finishes
Disable the Reindex All job
– What it does?
• Reindex
• Shrink your CRM database
– Create your own CRM database maintenance jobs
23
MANAGE CRM MAINTENANCE JOBS
24. #CRMUGSummit | #INreno15
Manual primary keys could cause performance issue
down the road
– CRM Server generates sequential GUID values for best
performance
– Use manual primary keys only for migration purpose
Further read:
– The Dangers of Guid.NewGuid();
• http://blogs.msdn.com/b/crminthefield/archive/2015/01/19/the-dangers-of-guid-
newguid.aspx
24
AVOID GENERATING PRIMARY KEY (GUID) YOURSELF
25. #CRMUGSummit | #INreno15
Turn on SQL Server RCSI (Read Committed Snapshot Isolation)
– https://msdn.microsoft.com/en-us/library/tcbchxcb(v=vs.110).aspx
ALTER DATABASE CrmOrgName_MSCRM
SET ALLOW_SNAPSHOT_ISOLATION ON
ALTER DATABASE CrmOrgName_MSCRM
SET READ_COMMITTED_SNAPSHOT ON
Add NOLOCK hint to your FetchXML Query
<fetch mapping="logical" no-lock="true">
<entity name="account">
<attribute name="name" />
<filter>
<condition attribute="creditlimit" operator="gt" value="1000000" />
</filter>
</entity>
</fetch>
25
DEAL WITH LOCKS
26. #CRMUGSummit | #INreno15
Help reduce database
deadlocks
Recommended for CRM
database, but not for data
warehouse
26
SQL SERVER – MAX DEGREE OF PARALLELISM
27. #CRMUGSummit | #INreno15
Set CRM database to Simple Recovery mode
– No SQL Server transaction logs
Disable IIS Logging
Disable all CRM workflows / plugins and auditing
Running data integration on CRM server itself
– Best possible network latency
– You need to alternate the target service URL to point to local server as
shown previously
27
EVEN MORE AGGRESSIVE MEASURES – SOME MAY
ONLY BE PRACTICAL FOR INITIAL LOAD
28. #CRMUGSummit | #INreno15
White Papers
– Microsoft Dynamics CRM 2015 White Papers & Technical Documentation
• http://blogs.msdn.com/b/crminthefield/archive/2015/01/05/microsoft-dynamics-crm-2015-white-papers-amp-technical-
documentation.aspx
– Optimizing and maintaining the performance of a Microsoft Dynamics CRM 2011 server infrastructure
• http://www.microsoft.com/en-ca/download/details.aspx?id=27139
Wikipedia
– Dynamics CRM Quick Optimization Guide
• http://social.technet.microsoft.com/wiki/contents/articles/13661.dynamics-crm-2011-quick-optimization-guide.aspx
Blog Posts
– http://blogs.msdn.com/b/crminthefield/archive/2012/04/26/avoid-performance-issues-by-re-scheduling-crm-2011-
maintenance-jobs.aspx
Best Practices
– Best practices for developing with Microsoft Dynamics CRM
• https://msdn.microsoft.com/en-us/library/gg509027.aspx
28
RESOURCES
30. #CRMUGSummit | #INreno15
THANK YOU!
Please complete & turn in your survey now
Session code: ADC26
Visit the online CRMUG Summit 2015 community to
download these slides, ask questions, and connect
with participants
Visit the CRMUG Medics in the HUB to get expert
advice on your CRM-related problems!
30
Editor's Notes
Title slide to be customized
Introduction slide to be customized. If multiple presenters, add additional intro slides
Objectives slide to be customized
Objectives slide to be customized
Agenda slide to be customized
Example of section slide
Example content slide
Example of section slide
Example content slide
Example content slide
Additionally, you can write integration solutions using CRM SDK