SnowMirror 4 – Performance Boost
As the ServiceNow platform is getting more and more popular, the size of ServiceNow databases is increasing over time. Individual tables are becoming so large that the initial loads performed by SnowMirror are sometimes taking very long. That's why we have spent the last six months re-designing the SnowMirror replication core. We are proud to announce SnowMirror 4.0 stable release which significantly improves the replication performance.
ServiceNow does not provide a way how to access its data in bulk, so SnowMirror has to use the out-of-the-box API which is designed for a transactional access only. That is the reason why the performance bottleneck in the whole replication architecture is the ServiceNow instance itself and not the other parts such as the SnowMirror agent or the mirror database.
The original SnowMirror till the 3.9.x series connects to the ServiceNow instance using 3 concurrent HTTP connections. This is a documented approach, to create and maintain a session with ServiceNow and reuse it for connections to the instance. It completely relies on the ServiceNow infrastructure and routing of the requests to individual instance nodes is out of SnowMirror's control. That's why the default was 3 connections in parallel because typically all the requests were routed to the same node, so we wanted to use maximum 3 out of 4 integration semaphores, not to overload a single node.
Another important design feature was that a single synchronization was running in a single thread and using only a single HTTP connection. So, in order to utilize the whole available throughput there had to be at least 3 synchronizations running in parallel.
The SnowMirror 4.0 replication core is completely redesigned. It uses a lower-level approach of connecting to ServiceNow, it better utilizes the ServiceNow cluster and the algorithms are fully parallel. All of this results in a significant performance boost.
The major difference is that SnowMirror is able to identify individual ServiceNow nodes and maintain two sessions with each node separately. This increases the number of replication threads, especially for bigger ServiceNow instances. Most of the instances have only two nodes, but the bigger the instance the more nodes it has.
The second leap forward is the fact that even a single synchronization can utilize the whole thread pool (i.e. all connections) which makes a single-running table several times faster than before. Of course, the general throughput improvement (i.e. if multiple synchronizations are running) is tightly related to the number of nodes. But still, even if your instance has only two nodes then the overall data throughput is 4/3 times higher and an individual table replicates up to 4 times faster.
The SnowMirror 3.x series used only one thread per synchronization and the max number of threads was 3. SnowMirror 4.x can utilize all threads for a single synchronization and the number of threads (i.e. sessions, concurrent connections) increases significantly for bigger instances making the whole throughput a lot higher.
Here are the performance improvements for sample ServiceNow instances (using the default of 2 sessions per node).
- 2 nodes (default) - single table 4 times faster, throughput 1,33 times higher
- 4 nodes - single table 8 times faster, throughput 2,67 times higher
- 8 nodes - single table 16 times faster, throughput 5,33 times higher
- 12 nodes - single table 24 times faster, throughput 8 times higher
Starting with SnowMirror 4.0 we introduce the new licensing model. There is a new edition in our offering called Cluster Edition. The Cluster Edition supports an unlimited number of ServiceNow nodes, it offers the ability to install SnowMirror in a high-available cluster and it includes the Platinum support level with 24x7 support hours and a phone assistance.
The existing Lite and Enterprise editions are limited to two ServiceNow nodes which is the default setup for the majority of ServiceNow customers. Even with this limit of two nodes, the new SnowMirror 4.0 will be up to 4 times faster for our existing users. There is an upgrade path to the Cluster Edition available too.