4.5 Series Snowflake Support
October 24, 2019
From today, we’re supporting Snowflake, an increasingly popular choice used by over a thousand enterprise companies worldwide.
Snowflake Support – Snowflake can be used as a mirror database. It cannot be used as a configuration database.
Parallel Data Processing – The previous versions were updating the mirror database or otherwise processing the data right after downloading it from ServiceNow. With slower databases or during differential strategies this was inconveniently blocking the worker threads.
SnowMirror 4.5 introduces a separate data-processing thread pool. So, there is a set of threads downloading the data and an independent set of threads updating the mirror database. The sizes of the thread pools are configurable to reflect different environments. The new approach improves overall performance and data throughput even for standard databases.
4.4 Series New York Release
September 9, 2019
SnowMirror 4.4 is fully compatible with the latest New York ServiceNow release. There are no major changes in New York that would affect the SnowMirror replication process. Just a few minor meta-data changes.
Upgrades – Please check the ServiceNow Schema Changes before upgrading to New York. This guide might prevent confusion when SnowMirror connects to New York for the first time.
4.3 Series Madrid Release
April 17, 2019
This is a Madrid-compatible release and it also contains an important infrastructure upgrade including Java 11 and Tomcat 9.
IMPORTANT – Users on Linux or users with additional installation adjustments have to finish the upgrade with several manual steps. Please check the Release Notes for more details.
Madrid Compatible Release – SnowMirror has been tested with the latest ServiceNow Madrid release and it is fully compatible with it. As always, SnowMirror stays compatible with the old releases, at least back to Jakarta.
Technology Stack Upgrade – SnowMirror upgrades to Java 11, OpenJDK, and Apache Tomcat 9. This change resolves the new commercial Java licensing conditions by using the open-source version.
Attachment Replication Performance – Improved performance of attachment synchronizations. A synchronization can download data from multiple ServiceNow connections at once.
4.2 Series Feature Requests
January 31, 2019
This release contains a set of smaller feature requests including changes in synchronization of certain table types, changes in Auto Schema Updates and it also supports SAML-based single sign-on.
IMPORTANT – Upgrade to this version requires access to several new ServiceNow system tables (i.e. sys_table_rotation, sys_table_rotation_schedule, sys_archive_log, sys_properties). Please adjust the permissions for your SnowMirror user account according to the Release Notes.
Table Rotation Support – SnowMirror can read rotation/extension tables directly from ServiceNow shards, thus improving the performance of the replication. It can also be configured to store data into multiple mirror tables (i.e. one mirror table per shard).
Configurable Synchronization Execution – A new “Synchronize…” feature enables to run a synchronization and to choose all possible settings how to run it. E.g. threading options, dry run mode, differential load with checking missing updates, etc.
Auto Schema Update (no truncation) – A new option “Enabled (no truncation)” which does not truncate a mirror table when a new column is created. There is also a possibility to exclude a column from Auto Schema Update.
Single Sign-On Authentication – SnowMirror can authenticate users using single sign-on over SAML 2.0 protocol and Web SSO profile.
4.1 Series REST API & London Compatibility
September 5, 2018
SnowMirror has been using Direct Web Services, the out-of-the-box SOAP API in ServiceNow, from the very beginning. This major release comes with an option to switch to the more modern REST API.
ServiceNow REST API – It is possible to configure SnowMirror to use the ServiceNow REST API instead of the traditional SOAP API for replicating the data. The replication algorithms are different for both APIs because of their different feature sets. The REST API is more modern and it is expected that ServiceNow will further improve and evolve only this integration standard. Performance tests indicate that the REST option might have higher data throughput than SOAP, but the exact numbers depend on many factors.
Please note that this new feature is still in Beta and we recommend our users to evaluate it in their sub-production environments first.
London Release Compatibility – SnowMirror has been thoroughly tested with the latest ServiceNow London release and it is fully compatible with it. As always, SnowMirror stays compatible with the old releases, at least back to Istanbul.
4.0 Series Performance Boost
May 2, 2018
The SnowMirror 4.0 replication core is completely redesigned. It uses a lower-level approach of connecting to ServiceNow, it better utilizes the ServiceNow cluster and the algorithms are fully parallel. All of this results in a significant performance boost.
Performance Boost – The major difference is that SnowMirror is able to identify individual ServiceNow nodes and maintain two sessions with each node separately. This increases the number of replication threads, especially for bigger ServiceNow instances. The second leap forward is the fact that even a single synchronization can utilize the whole thread pool (i.e. all connections) which makes a single-running table several times faster than before.
Fast Difference Delete Strategy – Unlike the original Differential strategy, the new Fast Diff Strategy should be much faster. It identifies the parts of a table where a delete could be and compares the keys only in these parts. The rest of the table is not being downloaded which enables the strategy to finish much faster.
3.9 Series Kingston Support & Resume Feature
December 20, 2017
SnowMirror 3.9 is fully compatible with Kingston and introduces an option to resume a failed synchronization.
Kingston Support – The only major challenge with the Kingston release is the new Function Fields feature. Unfortunately, the ServiceNow API does not send these fields together with the records, so temporarily excludes the function fields from the replication. This issue has been reported to ServiceNow and the future releases will hopefully fix that.
Resume Feature – If a synchronization fails for whatever reason (e.g. DB error, network error, ServiceNow outage) then it is possible to resume such a failed synchronization. SnowMirror then starts the replication process from the last successfully replicated record. This is especially useful for long-running full synchronizations.
3.8 Series Jakarta Support & LDAP Integration
June 30, 2017
SnowMirror 3.8 introduces the option to integrate SnowMirror authentication with Active Directory and adjusts the replication algorithms to the latest ServiceNow release called Jakarta.
Jakarta Support – Jakarta release brings several undocumented changes in the out-of-the-box API and related plug-ins. SnowMirror algorithms were revised and adjusted to these changes. Please note that older releases do not work with Jakarta correctly. If you plan upgrading to Jakarta then the upgrade to SnowMirror 3.8 is required too.
LDAP Integration – SnowMirror allows you to integrate it with corporate Active Directory or LDAP. When selecting this security realm then the users are not being stored in the configuration database but they are being authenticated against LDAP. The existing roles are being mapped to groups in LDAP.
3.7 Series Backup Improvements
April 27, 2017
SnowMirror 3.7 focuses on improvements in the SnowMirror Backup module such as the new attachment backup feature and support for differential backups.
Attachment Backup – This new feature enables performing full, incremental and differential backups of attachments. The retention period settings is available as well. The attachments are stored in a directory structure where each record has its own folder named by the record’s display value and the folder contains the attachments.
Differential Backup – Differential backup is a new type of backup storing all the changes since the last full backup.
Independent SnowMirror Backup – SnowMirror 3.7 contains improvements which allow SnowMirror Backup to be purchased as a standalone product or as upgrade to SnowMirror Enterprise Edition.
3.6 Series Data Type Editor
February 10, 2017
SnowMirror 3.6 upgrades the underlying technologies to the latest versions (Java 8, Tomcat 8) and introduces several useful improvements.
Edit Mappings – This new feature allows editing database column data types for individual fields. So it is possible to override the default SnowMirror mapping. E.g. it is possible to extend a varchar column to CLOB and such a setting persists even if the temporary tables are being used for the full loads. See the feature in the user manual.
Synchronize Since – It is possible to trigger an incremental load since a specified time stamp. This feature is useful for fixing inconsistencies even without the need of a full load.
Skipping Failing Records – The setting options allow synchronizations to finish even if there were records failing to insert into the DB. The synchronization then finishes with the warning status. The maximum number of skipped records is configurable.
3.5 Series Istanbul Support & Advanced Filtering
December 7, 2016
The new SnowMirror 3.5 focuses on the ServiceNow Istanbul release and on those customers who utilize SnowMirror for replicating large sets of tables.
Istanbul Support – SnowMirror 3.5 fully supports the latest ServiceNow Istanbul release. The Aggregate Plug-In check was re-worked, several data-type mappings were improved and ServiceNow database view replication algorithm was updated.
Advanced Filtering – SnowMirror allows filtering of the lists of synchronization and the list of historical runs is the new, advanced way which was inspired by the ServiceNow filtering feature. It is possible to define several criteria using a large set of variables and joining them together using boolean operators.
Synchronization Priority – The scheduler settings now allow to specify the synchronization priority. Higher priority synchronization triggers always before a synchronization with lower priority if waiting on a replication thread and it is possible to display the queue of waiting synchronization runs.
3.4 Series Index Replication & Schema Changes Notifications
August 8, 2016
The new SnowMirror 3.4 introduces a set of most requested features we collected at our booth during Knowledge16.
Index Replication– This new feature allows creating the same indexes in the mirror database as they are defined on the ServiceNow side. It makes reporting directly from the large mirror databases much easier. The indexes in ServiceNow are designed very well from the reporting perspective, so usually a very few additional indexes have to be created on the mirror side.
Schema Changes Notifications – SnowMirror notifies every time there is a new field in ServiceNow table. It works for the modified or removed fields as well and for the new indexes if the index replication feature is enabled. The notifications work independently on the Auto Schema Update feature. So even if the schema is not being updated automatically then users get notified about changes on the ServiceNow side.
NOTE: Index replication is disabled for both new and upgrading users. The feature can be enabled manually in Settings -> Generall Settings.
3.3 Series Helsinki & ServiceNow Backup
May 6, 2016
1. ServiceNow Helsinki Support
SnowMirror 3.3 fully supports the latest ServiceNow Helsinki release Patch 0. Table index handling has been re-worked and the replication algorithm together with data mapping has been adjusted. All connection testers and validation functions were improved and the whole application has been tested against a number of Helsinki instances.
2. ServiceNow Backup
This is a completely new module or even a product provided by the SnowMirror team. SnowMirror is now a fully-featured backup tool for individual ServiceNow tables including full and incremental backups, backup retention policies and a restore procedure.
3. Data Archiving Support
ServiceNow contains an optional Data Archiving plug-in which allows moving data that is no longer necessary for immediate day-to-day access from primary tables into a set of archive tables. The latest SnowMirror 3.3 supports removing archived records from a mirror table by using the Audit Delete strategy which is reading the ServiceNow archive log to find the set of removed records. It supports replication of the archive tables themselves as well.
NOTE: Both backup and archiving features are still in a Beta stage. Existing customers interested in evaluating the features please contact our support team at email@example.com.
3.2 Series Consistency Check
March 3, 2016
If enabled, SnowMirror compares the number of records in ServiceNow with the number of records in the mirror DB after each synchronization run. If there is an inconsistency in the numbers then the run finishes with a new warning status. New notification types are available so SnowMirror is able to send emails on warnings.
More features and fixes:
Differential Synchronization – new type of synchronization downloading missing records and deleting the orphans (Enterprise Edition only)
Activity Log Retention – configurable retention of logs and even synchronization run history
NOTE: SnowMirror is no longer distributed in two editions. Both Lite and Enterprise editions are distributed as one package and all the features are driven by licenses.
3.1 Series Geneva Support
January 26, 2016
SnowMirror 3.1 fully supports the ServiceNow Geneva release Patch 1. The data type mapping has been adjusted, all connection testers and validation functions improved and the whole application has been tested against a number of Geneva instances.
Display Values Improvements
Starting with 3.1 SnowMirror is analyzing all the meta-data in the System Dictionary to figure out which field is going to be a display value for a certain record so all the display value columns get correct data types. These new rules apply only to synchronizations created in the new version. Older tables with the 1000-char strings are still supported for data downloads.
IMPORTANT: this new algorithm requires access to additional meta-data table: sys_dictionary_override. Existing users shall adjust privileges of their SnowMirror ServiceNow users.
3.0 Series New Replication Algorithm
November 13, 2015
The replication algorithm no longer uses the sys_created_on column but only the sys_updated_on column. ServiceNow confirms that using this single column is better for the overall performance. The disadvantage is that SnowMirror is not able to count the number of new records before the synchronization begins. It can only count created and updated records together. Using this algorithm ServiceNow strongly recommends creating indexes on the ServiceNow side for larger tables to avoid full-table scans during the incremental loads.
More features and fixes:
Synchronization validation – including the check of existing indexes
Consistency check – at the end of each run
Encoded queries and views replication improved
Synchronization filter and sorting – UI improvement
ServiceNow server time usage – requires Aggregate Web Service plugin
Clean & Synchronize using temporary tables
SnowMirror Monitor – Fuji-compatible ServiceNow application, the old update set is deprecated