Admin Guide, Rest API, Performance API, SnowMirror Monitor, ServiceNow Upgrade, Error Codes, Release Notes

1. Getting Started

SnowMirror

1.1. Introduction

SnowMirror is a data replication tool for ServiceNow. It replicates data from ServiceNow into your local (on-premise) database.

SnowMirror is an on-premise software. You install it into your own ICT infrastructure. All data stays within your organization.

It provides a simple-to-use user interface, where you can choose which tables and columns you want to synchronize and how often you want to do that. The user interface also allows you to see a history of your synchronizations - what tables it synchronized, when it synchronized the data and how many records it downloaded.

SnowMirror does not modify or transform the data. It is up to you, how you use the data. Usually, customers use it for reporting. They connect their reporting tools to the mirror database and run reports against the replica. That is useful because they don’t have to run reports against a live ServiceNow environment. Customers also use SnowMirror for backups or as an integration point for other applications.

1.2. Architecture

SnowMirror reads data from ServiceNow using SOAP or REST API and stores it into your database. We call this process a synchronization. When SnowMirror receives data, it keeps it in memory (RAM). Once the data is permanently stored in mirror database, it may be freed from memory.

Configuration settings are stored in a configuration database. Data is stored in mirror database. SnowMirror does not modify the data - what it receives from API, is what it stores to the database.

SnowMirror’s user interface is accessible via an internet browser. Optionally, you can the check status of your synchronizations directly from within your ServiceNow instance using SnowMirror Monitor.

snowMirrorBasicScheme

Multiple SnowMirror installations with separate databases can be connected to single ServiceNow instance. It can also be installed into a high-availability cluster.

1.2.1. Databases

SnowMirror supports multiple databases: SQL Server, Oracle, MySQL, Postgres, Snowflake and Azure Synapse. See detailed information in Administrator Guide.

  • Configuration database - stores metadata, i.e. what to synchronize, when to synchronize, etc.

  • Mirror database - stores un-modified data mirrored from ServiceNow. The structure of tables and columns is a one-to-one of tables and columns from ServiceNow. You should never manipulate database schema from any other place than SnowMirror’s user interface.

    All tables contain additional columns mirror_created_on and mirror_updated_on. They contain a date and time when SnowMirror inserted/updated a record in the database.

1.2.2. Data to Synchronize

SnowMirror is able to synchronize and backup tables, views and attachments. Synchronization is a process that creates a one-to-one copy of your data without remembering history of the data. Backup is a process where we make a snapshot at the given moment and therefore it keeps history of the data.

SnowMirror is able to synchronize system tables, your own custom tables (u_) and tables from scoped applications. SnowMirror is not able to synchronize tables that do not have SysId column.

Follows a list of supported synchronizations and backups.

Type Target Description

Table

Database

Stores ServiceNow tables into your database. Keeps a one-to-one copy of your data.

View

Database

Stores ServiceNow views into your database. Keeps a one-to-one copy of your data.

Attachment

Filesystem

Synchronizes attachments of ServiceNow tables into your filesystem. Keeps a one-to-one copy of your attachments.

Backup

Filesystem

Stores ServiceNow tables as XML or CSV file to your filesystem. Keeps history.

Backup Attachment

Filesystem

Stores attachments of ServiceNow tables to your filesystem. Keeps history.

1.3. Migration Process

SnowMirror loads data from ServiceNow table using SOAP Web Services or REST API.

SOAP API replication algorithm: SnowMirror requests a batch of 10,000 SysIds which were created or updated in ServiceNow. After receiving SysIds, it loads data using these SysIds in batches of 250 records. It repeats these steps until it downloads all records.

REST API replication algorithm: SnowMirror requests a batch of 1,000 records which were created or updated in ServiceNow. It repeats these steps until it downloads all records.

SnowMirror processes multiple batches at the same time to improve the replication times. This is thoroughly explained in Performance Tuning chapter. Batch sizes are configurable on Performance Tuning page.

SnowMirror does not lock the table in ServiceNow because it only reads the data. If some other process updates and locks your table in ServiceNow, SnowMirror may have to wait until it is free to read the data.

SnowMirror creates and updates records in mirror database. Therefore, it has to lock affected records util it updates them. If some other process is accessing the mirror database table, SnowMirror may have to wait until it is free to commit the update.

To ensure correct synchronization process, please make sure you have correct date and time set on your computer.

1.3.1. Transactions

Data batches are transactional but whole synchronization runs are not. This prevents SnowMirror from reverting data in case of an error.

1.3.2. Migration Steps

synchronization process

  1. Initialization: Loads data from the configuration database and decides which data it will be synchronizing. It downloads meta-data from ServiceNow.

  2. Logs: Logs information about your environment to the activity log. This helps us to diagnose errors.

  3. Auto Schema Update: If enabled, SnowMirror finds out if the schema has changed and makes appropriate changes in mirror database. Optionally, it sends notifications about schema changes to an email of your choice.

  4. Count: Counts records in ServiceNow so that it know how many records it will be synchronizing.

  5. Create or Alter Table: If it does not already exist, it creates a new table in mirror database. Otherwise, it just updates the table (if necessary).

  6. Replication: This is the most important step. SnowMirror synchronizes data from ServiceNow. The replication process is a bit different based on which API you use.

    SOAP API

    SnowMirror requests a list of SysIds which were created or updated in ServiceNow. By default, it requests in a batch of 10,000 SysIds. It uses getKeys method from SOAP Web Services.

    After receiving SysIds, it loads data by these SysIds. By default, data are loaded in batches of 250 records. It uses getRecords method from SOAP Web Services.

    REST API

    SnowMirror requests records which were created or updated in ServiceNow. By default, it requests in a batch of 1,000 records. It uses REST Table Api

    Common to SOAP API and REST API

    If an encoded query is defined, SnowMirror checks for records which matched the query before, but now they don’t match it anymore. Such records are deleted from mirror database. To do that, SnowMirror load SysIds of all updated records (regardless the query) and compares them with SysIds of replicated records.

    As of version 4.0.0, SnowMirror processes multiple batches at the same time to improve the replication times. This is thoroughly explained in Performance Tuning chapter. Batch sizes are configurable on Performance Tuning page.

  7. Delete: Most ServiceNow tables store deleted data in sys_audit_delete table - i.e. deletes audited. This is the preferred way of determining deleted records. However, not every table is audited.

    For such cases, SnowMirror offers multiple deleting strategies. SnowMirror cannot determine if a table is audited or not. Note, that most tables in ServiceNow can be set as audited. If you want more information about auditing tables see ServiceNow Documentation on Auditing.

    Delete Strategies

    Audit

    Audit deleting strategy looks into sys_audit_delete table to find deleted records.

    To find deleted records of a synchronized table since the last synchronization, Audit Delete strategy filters data using tablename and sys_updated_on fields. You should have database index on these fields in ServiceNow.

    This is the most efficient strategy for finding deleted records. To be able to use this strategy, your table needs to audit deleted records in ServiceNow. To find out, whether deletes are audited on your table or not, please go to Edit Settings and use Validate button.

    Advanced topics:

    • Optionally, you SnowMirror can query data using sys_created_on field instead of sys_updated_on field. This is configurable via Advanced Settings using snowMirror.auditDelete.column property.

    • In ServiceNow, some tables make up a hierarchy (for example task → incident). Audit delete strategy has to look for deleted records not only from the table being synchronized but also from child tables. For example task → incident has to filter sys_audit_delete like this: tablename=task^ORtablename=incident.

    Diff

    Diff compares sys_ids in ServiceNow table with the sys_ids in mirror DB. Sys_ids that are present in mirror table, but not ServiceNow table are deleted. sys_updated_on and sys_created_on are intact.

    Fast Difference

    It counts records and downloads max. value of sys_updated_on column. If the values don’t match with the mirror database it splits the SysIds into smaller chunks and repeats until it gets until to 10,000 keys. Only then it downloads the keys and compares them with the mirror database. This strategy is useful when you have a large table and small amount of deletes because it reduces the number of requests to ServiceNow.

    None

    Does not delete any data from the mirror table. Insert and update tasks act as normal. Deleting data can be done manually by Clean and Synchronize button on the detail of the synchronization.

    Truncate (Deprecated)

    Truncate deleting strategy deletes all data from mirror table and downloads all records from scratch. It is deprecated - use Full Load Scheduler instead.
Default deleting strategy is Audit. If the synchronized table is not audited, consider auditing it. Use some other strategy only if the synchronized table is not audited.
When synchronizing a full load view, it is only possible to you use Truncate strategy

  1. Consistency check: Checks that number of records in ServiceNow matches the number of records in mirror database. It is explained in more detail in chapter Consistency Check.

  2. Finalize Migration: Updates the activity logs and finishes the synchronization.

1.3.3. Cache Management

For a better performance, SnowMirror caches ServiceNow metadata (data related to tables and its columns, parent-child structure, etc.). Because of this cache, some changes in ServiceNow can be taken into account with a delay.

If you want to clear the cache manually go to Settings menu, choose General Settings and click on Clear All Caches button.

Cache timeout is configurable through Advanced Settings.

1.3.4. ServiceNow Performance Considerations

  • Download times and number of requests

    The first initial load may take a while to finish. The download time depends on how many rows the table has, how many columns it has and how big the data inside the cells is. SnowMirror downloads data in batches, i.e. it makes a lot of small requests to download huge tables.

    It may take several minutes to download smaller tables or even several hours to download huge tables (millions of records).

    The initial load can be time-consuming. However,subsequent synchronizations will download just the changes (new, update and deleted records). Downloading changes is fast and does not impact the performance of your ServiceNow instance.

  • Database indexes in ServiceNow

    When SnowMirror queries data from ServiceNow, it extensively uses sys_updated_on column. If a table does not have sys_update_on column, it uses sys_created_on instead. To ensure fast processing times:

    • You should have an index on sys_updated_on column in ServiceNow.

    • If the table does not have sys_updated_on column, you should have an index on sys_created_on column.

    • If you have a table hierarchy, only the base table in the hierarchy needs to have the column indexed.

    You can use validate button to verify if you need to create the index. To create an index open ServiceNow and go to System Definition → Tables page. Select your table and open Database Indexes tab. You can create the index there.

1.3.5. Azure Synapse

since 4.10.0 The synchronization process is a bit different for Azure Synapse.

  1. SnowMirror downloads data from ServiceNow.

  2. SnowMirror stores the data into temporary files into local snow-mirror/data/synapse directory.

  3. Once the file reaches 100,000 records, SnowMirror uploads it to Azure Storage.

  4. SnowMirror copies data from Azure Storage into a stage table.

  5. At the end of a synchronization, SnowMirror merges stage table into the target table.

  6. At the end of a synchronization, SnowMirror deletes temporary files from local filesystem.

azure synapse

1.4. Display Values

In addition to raw values, ServiceNow allows its users to define so-called display values. Display
values are usually human-friendly texts which users understand more easily than raw values. For example:

  • Reference fields: When your records reference a user, ServiceNow has to store SysId of that user (raw value). The related display value can contain name of the user.

  • Choice fields: ServiceNow may store choices as numbers. The display values contains human-readable description of the choice.

SnowMirror allows you download display values of:

  • Fields of "reference", "domain_id", "document_id" or "glide_list".

  • Choices where choice value is greater than 0

1.4.1. Synchronization Strategies

SnowMirror provides you with three options how to handle display values.

  • Only sys_ids - SnowMirror does not download display values. It just downloads the raw value. For example reference fields will contain SysId of the referenced record.

  • Only Display Values - SnowMirror downloads and stores display values. It does not store raw values.

  • Both - SnowMirror downloads stores both raw values and display values. Display values are stored to a column with prefix dv_.

Note that there is a limitation imposed on display value synchronization. If display value changes and the record which references the display value does not change SnowMirror cannot discover the change. The display value is only updated when the referencing record changes too.

1.4.2. Display Value Replication Algorithm

Let’s focus on the reference fields. How does ServiceNow evaluate which field to use as a display value? It is not so straightforward as it might seem.
Imagine a table hierarchy Security Incident → Incident → Task. It is possible to specify display values in each table or just in one of them.
It is possible to omit the display values at all and rely on the defaults and finally, have you heard of Dictionary Overrides?

Here is the actual evaluation algorithm ServiceNow uses:

  • Dictionary Override – Current Table – The highest priority in the evaluation algorithm has a flag called Override Display Value in the Dictionary Override (i.e. table called sys_dictionary_override). It is possible to set a different display value then the parent table is specifying. E.g. the Task table has the field number as a display value. In the child table Security Incident it is possible to set up an override configuring short_description as a display value.

  • Dictionary Override – Parent Table – same as previous. If there is no override in the referenced table then the algorithm searches for overrides in the table hierarchy from the child to the parent. E.g. override in the Incident table takes precedence over a display value field specified in the Security Incident table but not over an override in the Security Incident table.

  • Display value field – Current Table – if there are no overrides then the standard display value fields come into play. The display value on the child table overrides the display value on the parent. E.g. Security Incident can specify its own field u_security_code as a display value for security incidents which would override the number field in Task.

  • Display value field – Parent Table – same as previous. If there is no display value field in the referenced table then the algorithm searches for display fields in the table hierarchy from the child to the parent.

  • Default fields – even if there are no display values explicitly set up, ServiceNow searches for default fields to make them as display value fields. These fields are “name“, “u_name“, * “sys_name“, “number“, “u_number“. The algorithm first searches the whole table hierarchy for a field called name. Then, if not found, it searches for u_name, etc. Please note that the child-parent mechanism does not work here. Name has always precedence over Number.

  • SysId – if everything fails then ServiceNow uses sys_id as a display value.

Note that the display value field can be another reference field or can be a choice list field. Transitive relation applies here so ServiceNow is traversing through the display values to find the last in the row.

1.5. Views Synchronization

Migration process described in previous chapters works perfectly when synchronizing tables. Beside tables ServiceNow allows you to define database views. SnowMirror supports views synchronization with some limitations. There are several issues SnowMirror has to tackle:

  • A record in a view is a Cartesian product of records from several tables. That means there is no unique and permanent SysId identifying the record.

  • A record in a view does not have to contain "sys_created_on" and "sys_updated_on" columns.

  • There is no delete log for views in ServiceNow.

SnowMirror provides two options to synchronize a view:

  • Full Load - full load synchronization deletes data from mirror database and then downloads all data from ServiceNow. This way you are guaranteed to have the most recent and 100 percent accurate copy of your data in mirror database. Depending on the amount of your data the synchronization may take a long time to perform.

  • Incremental Load - incremental load synchronization downloads only increments, i.e. new created, updated and deleted data. To use this synchronization type you have to choose a master table. SnowMirror uses this table to find out the increments. This means if records in other tables are changed SnowMirror would not find out. Also, the master table should not contain any duplicate SysId values.
    Incremental Load View synchronization requires less amount of time to perform at the cost of not having 100 percent accurate copy of your data. To tackle this issue we recommend to set up Full Load Scheduler to run once in a while because it clears data from mirror database and downloads all data from ServiceNow.

1.6. Attachments synchronization

Attachments synchronization process is very similar to table synchronization process. This section lists some differences between table synchronization and attachment synchronization.

1.6.1. Migration steps

Create directory if not exists

Besides configuration and mirror database, attachment synchronization needs access to the filesystem so that it can store attachments there. During the synchronization, SnowMirror creates a directory for attachments (if it does not already exist).

Path to the directory is defined by snowMirror.attachment.rootDir property which can be configured in Advanced properties. Be sure to have sufficient rights for creating directories and files in this directory.

Replication Using SOAP API

  1. SnowMirror loads serviceNow.number.of.keys keys (by default, this is 10,000) which were created or updated since the last synchronization run. The keys are loaded from sys_attachment table.

  2. Attachment meta-data are loaded for these keys. Meta-data are loaded in batches by serviceNow.number.of.records records (by default this is 250).

  3. Based on the meta-data, SnowMirror start to download the attachments from sys_attachment_doc table. SnowMirror repeatedly requests data from this table because it returns the attachments divided into small chunks. SnowMirror downloads serviceNow.number.of.attachment.chunks chunks (by default this is 500) per one request. Then it writes the chunks into temporary files and joins them to make up the whole attachment file.

  4. After downloading attachments their meta-data are stored to Mirror table and temporary files are moved to snowMirror.attachment.rootDir. If there already exists file with same name, new file is renamed (suffix is added to filename).

Replication Using REST API

  1. SnowMirror loads attachment meta-data using REST Attachment API. It loads the meta-data in batches of serviceNow.number.of.records records (by default, this is 1,000). It always loads records which were created or updated since the last synchronization run.

  2. Based on the meta-data, SnowMirror downloads the attachments one by one using REST Attachment API.

  3. After downloading attachments their meta-data are stored to Mirror table and temporary files are moved to snowMirror.attachment.rootDir. If there already exists file with same name, new file is renamed (suffix is added to filename).

Delete strategies
Delete Strategies for Attachments

Audit

Audit deleting strategy is not available when synchronizing attachments

Diff

Diff compares sys_ids in ServiceNow table with the sys_ids in mirror DB. Sys_ids that are present in mirror table, but not ServiceNow table are deleted (with their corresponding attachment file). sys_updated_on and sys_created_on are intact.

Fast Difference

It counts records and downloads max. value of sys_updated_on column. If the values don’t match with the mirror database it splits the SysIds into smaller chunks and repeats until it gets until to 10,000 keys. Only then it downloads the keys and compares them with the mirror database. This strategy is useful when you have a large table and small amount of deletes because it reduces the number of requests to ServiceNow.

None

Does not delete any data from the mirror table or attachment directory. Insert and update tasks act as normal. Deleting data can be done manually by Clean and Synchronize button on the detail of the synchronization.

Truncate (Deprecated)

Truncate deleting strategy deletes all data from mirror table and all attachments from local directory before the insert task. Update task is skipped. Note, that sys_updated_on and sys_created_on columns are set to time of last synchronization start. It can not be used to determine the time of actual row insert or update in ServiceNow. It is deprecated - use Full Load Scheduler instead.

1.6.2. Attachments Directory

Content of snowMirror.attachment.rootDir has following structure:
<snowMirror.attachment.rootDir>/<synchronization name>/<display value> where:

  • snowMirror.attachment.rootDir - configurable directory where attachments are stored

  • synchronization name is a name of a synchronization that you filled in Synchronization settings form

  • display value is a value of a field which is marked as display value for the given table

1.6.3. Encoded Query

If you have entered Encoded query it is applied on related table (e.g. incident) and NOT on sys_attachment table.
While getting keys of created (or updated) attachments since the last synchronization another additional request is made on related table (i.e. incident, task, …​) so that SnowMirror can learn which records fit encoded query. This assures that only attachments for records which fit encoded query are downloaded.

1.7. Index Replication

SnowMirror allows you to replicate not only tables and data but also database indexes. If you define an index in your ServiceNow instance SnowMirror will pick up the change and create the index in your mirror database. That is particularly useful if you make complicated SQL queries to your mirror database and need to improve the performance of these queries.

You can enable index replication on General Settings page or per-synchronization in Advanced Settings.

1.7.1. Custom Indexes

What if you need an index in the mirror database, but you don’t want to create it in your ServerNow instance? Don’t worry, we have it covered. You can create the index manually in your database, and we will preserve it intact.

1.7.2. SysId Columns

ServiceNow defines primary keys on sys_id columns. SnowMirror ignores these keys because it always creates its own primary key on "sys_id" column.

1.7.3. Performance Implications

If have you enabled Auto Schema Update or set up Notifications on schema change SnowMirror will send one additional request per synchronization. In the case of views, it will send several requests depending on how many tables the view consists of.

The request itself is not resource expensive and time-consuming. Yet, it should be taken into consideration especially when SnowMirror is configured to run many synchronizations very often.

1.7.4. Limitations

In some cases, ServiceNow Web services return us encoded column names. That happens when a column name is too long. It also happens on few columns on "task" table and its descendants. We cannot then couple the column in mirror database with a column of the ServiceNow index. Therefore, we ignore those indexes.

1.8. Synchronization Types

This chapter describes all types of synchronizations you can encounter in SnowMirror.

1.8.1. Incremental Synchronization

SnowMirror requests and downloads only records inserted, updated or deleted since the last synchronization. This is the most commonly used type of synchronization.

  • New and updated records - incremental synchronization uses sys_updated_on column to query the data. Thanks to that column, SnowMirror is able to download just the new and updated records.

  • Deleted records - there are multiple strategies how to find deleted records.

1.8.2. Clean and Synchronize

SnowMirror cleans all data in a mirrored table and downloads all records from scratch.

After the clean, it may take a long time to synchronize the data again. The old data won’t be available during the synchronization. To tackle the issue you may want to enable "snowMirror.replication.cleanAndSyncToTmpTable" which synchronizes the data to temporary table. Only after the synchronization finishes successfully the original mirror table is removed and replaced by the temporary table.

Optionally, you can enable Clean Synchronization to temporary table. If enabled, SnowMirror downloads data during a clean synchronization to a temporary table. After the synchronization finishes, it drops the old table and replaces it with the temporary table. This feature allows you to have all your data available during a long clean synchronization. On the other hand, you will lose all your custom permissions and indexes defined on the target table.

1.8.3. Differential Synchronization

Inserts all missing and deletes all excessive records. By default, it does not update any records. Optionally, it can be configured to check for updates too.

Differential synchronizations works as follows:

  1. It requests the first 10,000 SysIds from ServiceNow ordered alphabetically.

  2. It loads the first 10,000 SysIds from the mirror database ordered alphabetically.

  3. It computes a difference to find out new and deleted SysIds. If it cannot decide if the record is new or deleted it will process it in the next batch.

  4. It repeats the first three steps until there is no records left to process.

If checking for updates is enabled, it also has to load and compare values of sys_updated_on column.

Differential synchronization is particularly useful if you need to repair your mirror table. See Consistency Check chapter for examples.

1.8.4. Synchronize From

SnowMirror downloads only records whose sys_updated_on column contains a value greater than or equal to the date you chose.

This is useful if your synchronization missed or ignored some records, you have fixed the root cause of the issue, and now you want to download the missing records.

1.8.5. Resume

Let’s say you have a long-running synchronization and halfway through the synchronization process it fails (for whatever reason). It’s inconvenient to run it again because it takes some time to do so. "Resume" functionality is here to rescue.

It allows you to resume a failed synchronization. If you resume a synchronization, SnowMirror starts to download data from the point of the failure.

SnowMirror can resume table and view synchronizations. Backups and attachments are currently not supported.

To resume a synchronization, open Synchronization Detail page and click Resume button.

1.9. Archiving Support

ServiceNow provides optional Data Archiving Plugin. It allows users to store older data into archive tables.

If you use Archiving in ServiceNow, you may want to enable support in SnowMirror too. You can do that globally in General Settings or per synchronization in Advanced Settings.

Here’s what SnowMirror does when archiving is enabled:

  • When Audit Deleting Strategy is used, SnowMirror checks sys_archive_log to find archived records. Then, it deletes the records from mirror database. We use the following columns from sys_archive_log:

    • sys_created_on - to find new records in "sys_archive_log" since the last synchronization.

    • restored - to find whether a record was restored (i.e., is not archived anymore) or not.

    • from_table - to get archived records of a specific table. When a table has child tables, we also look for records related to these child tables. Example:

      • When we look for archived records in "incident", our query contains: from_tableINincident

      • When we look for archived records in "task", our query contains even child tables: from_tableINtask,incident,etc.

  • Checks sys_archive_log table so that it can insert restored records to the mirror table.

  • Synchronization of Archive tables (those starting with "ar_") - New records are picked up using "sys_archived" column. Updated records are picked up using "sys_archive_restored". That’s different from ordinary (non "ar_") tables where we pick up inserted and updated records only using "sys_updated_on" column.

1.10. Consistency Check

In some cases SnowMirror may miss synchronizing some records.

Imagine that you imported data to your table in ServiceNow and all records had sys_updated_on set on some old date. SnowMirror already synchronized newer data hence it will miss the newly imported records. This is when Consistency check comes in handy. After every synchronization run it counts the number of records in ServiceNow and number of records in mirrored table. Then it compares the counts and if it finds out that they are not equal it will put the synchronization into Warning state and send notification email (if configured to do so).

consistency check list

When you encounter such a warning you should start looking for a reason of the inconsistency. To tackle the issue with old dates in imported records you can use differential synchronization. To diagnose any other potential issues you may want to use validate button.

consistency check activity log

See Admin Guide on how to configure consistency check.

1.11. Mirror Database Constraints And Indexes

SnowMirror creates a primary key on "sys_id" column in every mirror table. Thanks to the key, it can update records really fast.

Older SnowMirror installations used a unique key instead of a primary key. Those installations keep using unique key, even after upgrade. New (clean) installations use a primary key.

In case of Full Load Views, the records consist of several joined tables. Therefore, there is no single sys_id, and we cannot create a primary key.

You can even configure SnowMirror to use unique indexes instead of primary keys. Go to Settings → Advanced Settings page and set a property "snowMirror.replication.index.sysIdConstraintType" to "UNIQUE_INDEX".

2. Backups

SnowMirror allows you to copy and archive data from ServiceNow so that it may be restored in case of data loss. It supports backups of tables and attachments.

Data from tables can be stored to CSV or XML files. Those are valid files recognized by ServiceNow. You can use them to import your data back to ServiceNow.

2.1. Backup Strategies

SnowMirror supports three backup strategies. You can run them manually or automatically by a scheduler.

  1. Incremental - downloads changes made since the last backup (no matter if it was full or incremental).

  2. Differential - downloads changes made since the last full backup.

  3. Full - downloads all data.

2.2. Location

All backups are stored to snow-mirror/data/backups directory. It can be changed on General Settings page using "Backups" field. Please make sure to have sufficient permissions for creating directories and files in this directory.

change directory

Each backup synchronization creates its own directory with the name of the synchronization.

Sample Backup Structure:
snow-mirror/data/backups
├───incident
│       2581_FULL_2016-01-08_07h-43m-50s.zip
│       2582_INCREMENTAL_2016-01-08_07h-44m-35s.zip
│       2583_INCREMENTAL_2016-01-08_07h-46m-35s.zip
│
└───sys_user
        2581_FULL_2016-01-08_07h-43m-50s.zip
        2582_INCREMENTAL_2016-01-08_07h-44m-35s.zip
        2583_INCREMENTAL_2016-01-08_07h-46m-35s.zip

2.3. Retention Period

Every day SnowMirror runs a job which removes old backups. Each synchronization defines a "retention period". The job will remove backups older that the retention period.

2.4. Table Backup

The backup synchronization works the same as table synchronization works except a few exceptions:

  • Backups do not use the mirror database at all. They are stored exclusively on filesystem in files.

  • SnowMirror stores durations and timers to mirror database as either a number of milliseconds or as a date and time. Backups do not use these settings so that the fields can be imported back to ServiceNow.
    XML files store timers and durations as a date and time (timestamp). CSV files store them as a number of seconds.

  • SnowMirror allows a user to store dates and times in whatever timezone she wishes (default being UTC). Backups can only store dates and times in UTC so that the values can be imported back to ServiceNow.

  • SnowMirror can’t synchronize columns which are not marked as "active" in the Dictionary. It also can’t synchronize data from child tables - e.g. when exporting "task" (parent) table you won’t see any "incident" (child) records in the export. If you want to back up child tables you have to create a new synchronization for each child.

2.4.1. File Formats

You can choose from the following formats:

  • CSV - records are store in comma separated file format. ServiceNow can run all necessary business rules during the import.

  • XML - records are stored in XML file. Unlike CSV it can even store deleted records. ServiceNow won’t run any business rules during the import!

The CSV or XML files are always compressed into a ZIP file which highly reduces the size of the backup.

The file has the following format: [ActivityLogId_TypeOfTheSynchronization_Date_Time].

  • ActivityLogId - ID of the activity log created during the backup

  • TypeOfTheSynchronization - FULL or INCREMENTAL. See Synchronization Types chapter for details.

  • Date_Time - date and time when the synchronization started

Example:
2581_FULL_2016-01-08_07h-43m-50s.zip

2.4.2. Create and Edit

For detailed description of how to create a new synchronization please New Synchronization Wizard. Also, see Retention Period (Backups Only) and File Format (Backups Only) sections describing backup specific fields.

Here, we will just discuss several specifics related to backups.

  • CSV

    • Deleting Strategy - you cannot choose any deleting strategy, because CSV file can’t store deleted records.

    • Reference Fields - defaults to Only Display Values so that it behaves like ServiceNow does. However, you can choose whatever option you want to.

    • Columns - you can choose columns you want to synchronize.

  • XML

    • Deleting Strategy - if you choose Audit deleting strategy, and your table is audited then all deleted records will be stored in the XML file.

    • Reference Fields - you won’t have to choose any Reference Fields strategy. XML backups have to always download both value and its display value.

    • Columns - all columns are synchronized to stay compatible with ServiceNow XML files.

new sync

2.4.3. Import To ServiceNow

Backup files are valid CSV or XML files used by ServiceNow. See ServiceNow documentation for detail how to import those files.

Please take a time to study how import of CSV and XML files works and know the implications.
Importing CSV files

  1. Load Data into a new table. Open System Import Sets → Load Data page import data into a new table. Let ServiceNow create a new table with all columns for you. All columns will be of a "string" type.

    load data

  2. Create a transform map

    Open System Import Sets → Create Transform Map page and create a new transform map. Choose the new table created in the previous step as a source table and the table where you want to import the data as target table.

    create transform map

    After submitting the form, you have to map fields from the source table to fields in the target table. We recommend using "Auto Map Matching Fields" or "Mapping Assist" for that purpose. Mapping is thoroughly described in ServiceNow documentation. If you have never used transform maps, we recommend you to read ServiceNow documentation. In this tutorial we pinpoint just a few important steps:

    • Coalesce Sys ID column - configure Sys ID column in a target table to coalesce. This makes ServiceNow to match source values with values from existing records. Simply said this will allow you to update existing records with the records in your import set. See ServiceNow documentation for detailed description. Note that you will see two Sys ID columns in the source table: "Sys ID" and "sys_id". Please choose "sys_id" column - that is the one from you import set.

      coalesce sys id

    • Date/Time format - please make sure the mapping of Date and Date/Time fields uses the correct format.

      date field format

  3. Run the transformation - Open System Import Sets → Run Transform page and choose your import set and transform map. Click Transform button and ServiceNow will load the data to the target table.

Importing XML files

Please follow the documentation on ServiceNow documentation.

ServiceNow will store the data directly to the database and won’t run any business rules during the import!

The import procedure is simple. On the target table choose Import XML and then upload your XML backup file to ServiceNow. Then just click "Upload" and ServiceNow will start the import.

import xml

2.5. Attachments Backup

Attachments are stored into a directory structure which is packed into a single ZIP file. See the sample structure below.

Sample Backup Structure:
snow-mirror/data/backups/
├───incident
│       2581_FULL_2016-01-08_07h-43m-50s.zip
│      └───INC0000001
│            └───FirstAttachmentOfInc0000001.png
│            └───SecondAttachmentOfInc0000001.png
│       2582_INCREMENTAL_2016-01-08_07h-44m-35s.zip
│      └───INC0000001
│            └───ThirdAttachmentOfInc0000001.png

It is worth mentioning that such a directory structure cannot store deleted files. If you delete an attachment in ServiceNow it won’t be tracked in ZIP file. Therefore, you should perform Full backup from time to time.

2.5.1. Create and Edit

For detailed description of how to create a new synchronization please New Synchronization Wizard. Also, see Retention Period (Backups Only) section.

3. Cloud Storage

SnowMirror allows you to store data from ServiceNow as CSV files into cloud storage. We support Amazon S3, Azure Storage and Google Storage. Once you have your data stored in cloud storage, you can query it with other cloud tools. For example, Amazon Redshift can query files in S3.

SnowMirror stores data into a bucket. Every synchronization has its own folder there and is further structured into folders. Updated records are written into new files. Therefore, you end up having multiple snapshots of records in multiple files.

3.1. Bucket Structure

SnowMirror stores data into a bucket. Every synchronization has its own folder there and is further structured into folders:

  • data - contains new and updated records

  • deleted - contains sys_ids of deleted records

  • archived - contains sys_ids of deleted records

Sample File Structure in Cloud Storage:
your-bucket
├─── incident
│    ├─── data
│    │    ├─── 2022-12-31_13-00-00-100.csv.gzip
│    │    ├─── 2022-12-31_13-01-00-100.csv.gzip
│    │    └─── 2022-12-31_14-01-00-100.csv.gzip
│    ├─── deleted
│    │    ├─── 2022-12-31_13-01-01-100.csv.gzip
│    │    └─── 2022-12-31_14-01-01-100.csv.gzip
│    └─── archived
│         ├─── 2022-12-31_13-01-02-100.csv.gzip
│         └─── 2022-12-31_14-01-02-100.csv.gzip
└─── sys_user
     └─── data
          ├─── 2022-12-31_13-00-00-100.csv.gzip
          └─── 2022-12-31_14-01-00-100.csv.gzip

3.2. Updates

Updated records are written into new files. Therefore, you end up having multiple snapshots of records in multiple files. There are two options how to recognize which snapshot is newer:

  • File names contain a timestamp of when the file was created.

  • Every CSV file contains a column named "mirror_created_on". It contains a date and time when the record was written into the file.

Updated records are stored to new files and the old files are kept intact. That means, you can find historical values of your records. However, this is not a replacement for "sys_audit" table. SnowMirror does not capture every change. It captures changes at scheduled intervals. If you, for example, download data every 30 minutes, and you update your record multiple times during that timeframe, SnowMirror captures only the last value.

3.3. Deletes

SnowMirror is able to recognize deleted records only if deletes in ServiceNow table are audited in ServiceNow. That means, it supports only so-called Audit Delete Strategy.

3.4. File Format

SnowMirror stores data into CSV files which are further compressed with Gzip. CSV stands for comma-separated values. The first line in each file is a header - i.e. a list of column names. All the other lines contain your ServiceNow data.

The first column is "sys_id" column. The last column is "mirror_created_on". This column does not exist in ServiceNow. SnowMirror adds it so that you can see date and time when the record was written into the file.

Sample CSV File with New and Updated Records
"sys_id","assigned_to","dv_assigned_to","number","mirror_created_on"
"46b66a40a9fe198101f243dfbc79033d","5137153cc611227c000bbd1bd8cd2007","David Loo","INC0000009","2023-02-01 12:35:23"
"46b9490da9fe1981003c938dab89bda3","9ee1b13dc6112271007f9d0efdb69cd0","Don Goodliffe","INC0000010","2023-02-01 12:35:23"

Files with deleted records contain just "sys_id" column and "mirror_created_on" column.

Sample CSV File with SysIds of Deleted Records
"sys_id","mirror_created_on"
"9b33214d1bbf6410593f42e58d4bcb28","2023-02-01 12:43:32"
"2eace1f4db1f73005c7ad92b5e9619a0","2023-02-01 12:43:32"

Files with archived records contain just "sys_id" column and "mirror_created_on" column.

Sample CSV File with SysIds of Archived Records
"sys_id","mirror_created_on"
"9b33214d1bbf6410593f42e58d4bcb28","2023-02-01 12:43:32"
"2eace1f4db1f73005c7ad92b5e9619a0","2023-02-01 12:43:32"

3.5. Consistency Check

SnowMirror accesses cloud storage using libraries provided by the cloud vendor. These libraries do not allow us to query the files. Therefore, we can run a consistency check only when running an uninterrupted clean synchronization.

3.6. Database vs. Files

Storing data into cloud storage is supported since SnowMirror 5.0.0 (February 2023). Up until then, we supported only relational databases. Therefore, it may be useful to summarize differences between synchronization to relational databases and cloud storages.

Feature Relational Database Cloud Storage

Updates

Existing records in the database are updated.

Updated records are written into new files and existing files are kept intact.

Deletes

Can utilize Audit, Differential, Fast Difference delete strategies.

Deleted records can be identified using Audit delete strategy. Differential and Fast Difference delete strategies are not supported.

Differential synchronization

Supported.

Not supported (we cannot query data in cloud storage).

Consistency Check

Executed at the end of every synchronization.

Executed only at the end of an uninterrupted clean synchronization.

Soft Delete

Optionally, records can be "soft" deleted. I.e. instead of deleting a record from database, we keep it there and mark it as deleted.

Sys_id columns of deleted records are written into new files. Existing files are kept intact. Therefore, records are "soft" deleted by default.

4. Installation

For quick installation see Installation section in Admin Guide.

5. Licensing

To be able to synchronize data, you need to enter a valid license key. Please visit our website to see license types, support levels and supported features. You can request a trial key on our website.

If you don’t enter a valid license, SnowMirror will show a warning at the top of each page.

no license entered

If your license is about to expire this message will appear in the page header:

license expiring

6. Logging in/out

Every user has a unique username for logging in and one or more user roles. User roles give him or her a permission to access application functionality. The name of the logged-in user is shown on the right side of top-horizontal menu.

If you want to log out, use the Logout link, which is placed on the right side of top menu next to username of currently logged-in user.

login top menu

User roles can be added or removed only by Super Administrators. For more information about roles and permissions see User Roles).

6.1. Permanent login

After 4 hours of inactivity, SnowMirror logs you out. In case you want to stay logged in longer (2 weeks), you can use "Remember Me" checkbox.

login remember me

When you access a login page directly (URL http://address-of-your-snowmirror:9090/login) it always shows the login page (regardless the "Remember Me" checkbox). All other pages will let you in automatically if "Remember Me" is checked.

The "Remember Me" option creates a cookie in your browser containing an access token. The cookie is removed by your browser after 2 weeks of inactivity or if you click "Logout" button. Otherwise, you stay logged in.

7. User Roles

Every SnowMirror users have to have at least one of the following roles.

Role Permissions

Synchronizations

User Management

Settings

REST API

Read

Run

Create

Edit

Own profile

All Users

Notifications

All Other

Use

Synchronization Reader

✓

N/A

N/A

N/A

✓

N/A

N/A

N/A

N/A

Synchronization Supervisor

✓

✓

N/A

N/A

✓

N/A

N/A

N/A

N/A

Synchronization Administrator

✓

✓

✓

✓

✓

N/A

✓

N/A

N/A

Super Administrator

✓

✓

✓

✓

✓

✓

✓

✓

N/A

API Caller

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

✓

8. Dashboard

Dashboard is the main page of SnowMirror and it is divided into three parts - list of failed synchronizations, graph of migrated data volume and last / upcoming synchronizations.

8.1. Failed Synchronizations

'Failed Synchronizations' shows synchronizations that failed during the last run. Failed synchronizations shows name of synchronization and time of failure. By clicking on the synchronization name you can obtain more information about the synchronization see Synchronization Detail.

8.2. Graph of Migrated Records

Graph of migrated records shows global statistic of all transferred data per day. It counts a sum of all inserted/updated/deleted records from all synchronizations executed during a particular day. Every each color represents a kind of data operation similar to progressbar.

Much more information is available on the detail of the synchronization. It is possible to use a link on every synchronization name in the table to go to synchronization detail.

migrated records graph

8.3. Upcoming Synchronizations

Upcoming synchronizations tab shows all synchronizations which are currently running or will be executed in next 12 hours by default. Last synchronizations show all synchronization that were triggered in past 12 hours. This interval can be set in the property file.
It includes basic details like time of synchronization execution, result of the last execution and progressbar. Contents are sortable.

Last / Upcoming Synchronizations Table:

Name

Name of the synchronization. It is chosen by a user and it does not have to match the table name in mirror database or the name of the table in ServiceNow.

Next Run

Next Run is a date and time of the next scheduled run. Synchronizations with manual execution scheduler have this field left empty.

Starts In

Shows a time left until the next synchronization run starts.

Synchronization Progress

The progress bar shows a ratio of migrated records. It is continuously updated and shows the current progress of the synchronization. The progress bar is made of three colors:

green icon green - data which were inserted in mirror DB

blue icon blue - data which were updated in mirror DB

grey icon grey - records deleted from mirror DB

If the last synchronization had not migrated any records, the progress bar background is white.

8.4. Last Synchronizations

Last synchronizations tab shows all synchronizations that were triggered in past 12 hours.

It includes basic details like the start and end time of synchronization run, result of the last run and progressbar. Contents are sortable.

Last / Upcoming Synchronizations Table:

Result

An icon representing a result of the last synchronization run.

Name

Name of the synchronization. It is chosen by a user and it does not have to match the table name in mirror database or the name of the table in ServiceNow.

Start Date

Date and time when the synchronization started.

End Date

Date and time when the synchronization finished.

Duration

How had the synchronization been running.

Synchronization Progress

The progress bar shows a ratio of migrated records. It is continuously updated and shows the current progress of the synchronization. The progress bar is made of three colors:

green icon green - data which were inserted in mirror DB

blue icon blue - data which were updated in mirror DB

grey icon grey - records deleted from mirror DB

If the last synchronization had not migrated any records, the progress bar background is white.

9. Synchronization Overview

Synchronization Overview includes a list of all created synchronizations. Table includes the preview of basic information and is sortable.

sync overview table
Synchronization Owerview Table

Result

An icon representing a result of the last synchronization run. SnowMirror recognizes the following results:

ok icon

success - data from ServiceNow was successfully synchronized to mirror database

warning icon

warning - data from ServiceNow was synchronized to mirror database but SnowMirror detected potential issues. The synchronization requires your attention.

error icon

fail - data could not be synchronized. An error occurred during the synchronization

running icon

running - the synchronization is currently running

canceling icon

cancelling - synchronization task is being canceled

error icon

unknown result - the synchronization was created but it was not executed yet

waiting icon

waiting - the synchronization is waiting in the queue to be executed +

disabled icon

disabled - the synchronization is deactivated and can’t be executed until you reactivate it.

warning disabled icon

disabled with warning - the last synchronization’s run ended up with a warning. The synchronization is deactivated and can’t be executed until you reactivate it.

error disabled icon

disabled with error - the last synchronization’s run ended up with error. The synchronization is deactivated and can’t be executed. It can be reactivated on Synchronization Detail Page.

Name

Name of the synchronization. It is chosen by a user, and it does not have to match mirror DB table name or the name of the table in ServiceNow.

Next Run

Next Run is a date and time of the next scheduled synchronization run in format mm/dd/yyyy hh:mm. Synchronizations with manual execution scheduler have this field left empty.

Last Run

Last Run is a date and time of the last synchronization run (no matter if it was successful or not). The format of the date ant time is mm/dd/yyyy hh:mm.

Duration

A time how long it took to finish the synchronization.

Progress

The progress bar shows a ratio of migrated records. It is continuously updated and shows the current progress of the synchronization. The progress bar is made of three colors:

green icon green - data which were inserted in mirror DB

blue icon blue - data which were updated in mirror DB

grey icon grey - records deleted from mirror DB

If the last synchronization had not migrated any records, the progress bar background is white.

9.1. Synchronization Overview Controls

  1. New Synchronization - Redirects to the New Synchronization Wizard page.

  2. Synchronize - Triggers synchronization of selected synchronizations.

  3. Drop-down menu

    1. Execute

      1. Clean and Synchronize - Deletes all mirror table data and immediately synchronizes selected synchronizations. Neither sys_updated nor sys_created on are used. That means all data are loaded from ServiceNow table.

      2. Synchronize From - Synchronize From downloads only records whose sys_updated_on column contains a value greater than or equal to the date you chose.

      3. Differential Synchronization - Inserts all missing and deletes all redundant records. It does not update any records.

      4. Resume - Resumes the last synchronization run and continues from the point of failure.

      5. Stop - Stops selected synchronizations if they are running.

    2. Manage

      1. Edit Scheduler - Redirects to edit scheduler page. Can be used to schedule multiple synchronizations (selected by checkbox).

      2. Activate - Enables deactivated synchronizations. After activation, synchronizations will be able to synchronize data from ServiceNow.

      3. Deactivate - Temporarily prevents selected synchronizations from triggering. If you disable synchronizations, you will not be able to manually trigger them.

      4. Delete - Permanently deletes selected synchronizations and (optionally) all its data from mirror database.

      5. Export - Exports definitions of selected synchronizations. See Administrator Guide for details on Import/Export.

9.2. Filtering

Synchronizations can be filtered by all columns available. To open the filtering form click the funnel icon. After filling in filtering conditions click the Run button and SnowMirror will show only synchronizations matching the conditions.

Also note that there is word "All" next to the funnel icon. This is breadcrumbs area. It shows which filter is applied. The word "All" means no filter is applied.

filtering overview filter

9.2.1. Filtering Options

Each field has a predefined set of operators which you can use to filter its values. Generally, there are four types of operators:

  1. Text - is, is not, is empty, is not empty, contains, does not contain, starts with, ends with

    Used for filtering text fields such as:

    • Name - name of the synchronization,

    • Encoded query - encoded query of the synchronization

    • Mirror table - name of the table in mirror database

    • |since 5.7.0Target database - name of the database connection that is used to store synchronization data

    • Created by - name of who created the synchronization

    • Updated by - name of the last person who updated the synchronization

  2. Numbers - is, is not, greater than, greater than or equal to, less than, less than or equal to, between

    Used for filtering numbers in fields such as:

    • Inserted - number of records inserted during the synchronization

    • Updated - number of records updated during the synchronization

    • Deleted - number of records deleted during the synchronization

    • Total - total number of records processed during the synchronization (inserted + updated + deleted)

  3. Options - you are only allowed to choose one of the predefined values. Used on fields:

    • Synchronization Type - Table, view, attachment or backup

    • Result

      • Successful - finished its execution successfully

      • Warning - finished its execution, but a user should check inconsistencies in records count

      • Failed - finished its execution with error

      • New - synchronization has never been executed

      • Scheduled - synchronization is waiting to be executed

      • Running - synchronization is being executed

      • Finished - synchronization finished its execution (no matter if it succeeded or failed)

    • State - Active or inactive

    • Reference Fields - Only sys_ids, only display values, both

    • Delete Strategy - Audit, truncate, difference or none

    • Include Inherited Columns - Yes or no

    • Auto Schema Update - Yes or no

    • Synchronization Interval - Daily, weekly, periodically, cron or manually

    • Synchronization Interval (Full) - Daily, weekly, periodically, cron or manually

    • Execution Type - Type of full load scheduler: Clean and synchronize or differential synchronization

  4. Date and time - today, yesterday, tomorrow, Last minute, Last 15 minutes, last 30 minutes, last hour,

    • Duration - how long it took to finish the synchronization

    • Last Run - date and time when the synchronization finished

    • Created On - date and time when the synchronization was created

    • Updated On - date and time when the synchronization was updated

9.2.2. Cancelling Filter

You can cancel the current filter by clicking the "All" breadcrumb.

9.2.3. Breadcrumbs

Breadcrumbs work the same way as they do in ServiceNow GUI:

  1. Click on one breadcrumb and only the one filter will be applied.

  2. Click the arrow separator, and the following breadcrumb will be removed from filter condition.

filtering breadcrumbs

9.3. Queue

Synchronization queue provides you a list of synchronizations which are currently running or waiting to be executed. Synchronizations are executed from the top of the list to the bottom.

queue sample

To open the queue, click on the "Show Queue" link on Dashboard.

queue link

10. New Synchronization Wizard

To create a new synchronization use the New Synchronization button on the top of Synchronization Overview (you can open it by clicking "Synchronizations" in the top menu). This button starts a three steps wizard which will guide you step-by-step through process of synchronization creation.

10.1. Table to Synchronize

First, you have to choose what kind of synchronization you want to create. You can synchronize or backup tables, views and attachments.

After choosing the type of your synchronization, you can choose a name of a table/view to synchronize. The list of names is alphabetically ordered and supports quick search with autocomplete. Click "Next" button to move to the next step.

SnowMirror caches the list of tables/views so that it does not have to connect to ServiceNow every time you open the page. If you don’t see your table/view, please click on the "Loaded at" link on the bottom-right side of the page and SnowMirror will refresh meta-data.

new sync select table

10.1.1. Client Script For a View

Views support incremental updates only partially. To overcome the limitations on incremental synchronization you can synchronize tables involved in a view and create a view in your database manually. Before creating database view, all tables involved should exist in mirror database. If some tables are missing, please create and run synchronizations first.

Click Generate SQL button to generate the client SQL script.

new view sync
new view sync generate sql

10.2. Synchronization Settings

Second step of the wizard allows you to set synchronization parameters and records of table you want to synchronize.

10.2.1. Synchronization Name

Name of synchronization is chosen by a user. It does not have to match with mirror DB table name or name of table from ServiceNow. It should be any word name or phrase, which characterizes the synchronization (E.g. task_nightly, incident-basic info,…​).

10.2.2. Name of the Table in ServiceNow

Contains a name of the ServiceNow table chosen in the first step of the wizard. It is read-only.

10.2.3. Target Database

since 5.7.0This option appears only if multiple database connections are configured. You can choose which database connection will be used to store synchronization data.

10.2.4. Table Name in Mirror DB

A name of a table in your local mirror DB. By default, the name matches the name in ServiceNow. You can choose a different name. The name can be 255 characters long, and it must contain only letters (without diacritic), numbers, underscores and dashes.

10.2.5. ServiceNow Encoded Query

An encoded query represents a filter of records. An encoded queries can use operators like and, or, not, greater than, lower than, etc. For more information about encoded queries see ServiceNow documentation on Encoded query strings.

Be sure to fill in a valid query. Query will not be validated. An invalid query will not be recognized in SnowMirror logs.
After editing encoded query, all synchronized data from in mirror DB table will be deleted.
On attachment synchronizations, an encoded query is applied on the target table (e.g. incident) and not on sys_attachment table.
If you are not sure how to compose your query or if it is valid, open the table in ServiceNow, choose a filter, right click on it and choose "copy query".
Copy an Encoded Query from ServiceNow

10.2.6. Delete Strategy

This parameter specifies, how will be data deleted from mirror database, based on deletes in ServiceNow. Following strategies are available:

Delete Strategies

Audit

Audit deleting strategy looks into sys_audit_delete table to find deleted records.

To find deleted records of a synchronized table since the last synchronization, Audit Delete strategy filters data using tablename and sys_updated_on fields. You should have database index on these fields in ServiceNow.

This is the most efficient strategy for finding deleted records. To be able to use this strategy, your table needs to audit deleted records in ServiceNow. To find out, whether deletes are audited on your table or not, please go to Edit Settings and use Validate button.

Advanced topics:

  • Optionally, you SnowMirror can query data using sys_created_on field instead of sys_updated_on field. This is configurable via Advanced Settings using snowMirror.auditDelete.column property.

  • In ServiceNow, some tables make up a hierarchy (for example task → incident). Audit delete strategy has to look for deleted records not only from the table being synchronized but also from child tables. For example task → incident has to filter sys_audit_delete like this: tablename=task^ORtablename=incident.

Diff

Diff compares sys_ids in ServiceNow table with the sys_ids in mirror DB. Sys_ids that are present in mirror table, but not ServiceNow table are deleted. sys_updated_on and sys_created_on are intact.

Fast Difference

It counts records and downloads max. value of sys_updated_on column. If the values don’t match with the mirror database it splits the SysIds into smaller chunks and repeats until it gets until to 10,000 keys. Only then it downloads the keys and compares them with the mirror database. This strategy is useful when you have a large table and small amount of deletes because it reduces the number of requests to ServiceNow.

None

Does not delete any data from the mirror table. Insert and update tasks act as normal. Deleting data can be done manually by Clean and Synchronize button on the detail of the synchronization.

Truncate (Deprecated)

Truncate deleting strategy deletes all data from mirror table and downloads all records from scratch. It is deprecated - use Full Load Scheduler instead.
Default deleting strategy is Audit. If the synchronized table is not audited, consider auditing it. Use some other strategy only if the synchronized table is not audited.
When synchronizing a full load view, it is only possible to you use Truncate strategy

10.2.7. Auto Schema Update (only table, view and backup synchronization)

Auto Schema Update check for schema changes in ServiceNow. You can choose how SnowMirror should handle those changes.

  • Enabled - whenever a synchronization is executed, SnowMirror checks for schema changes in ServiceNow. It automatically adds, updates (data type, max. length of a column) and removes columns. If a new column is added SnowMirror clears the mirror table and downloads all records from scratch to get the missing data.

  • Enabled (no truncation)since 4.2.0 - the same as Enabled option. It handles new columns differently, though. If a new column is added, SnowMirror does not clear the mirror table. Instead, it creates the column and populates it with a default value (which is defined in ServiceNow).

    Note: if the column uses a dynamic default value, or it is a display value column, the column is not populated and is empty!

  • Disabled - SnowMirror does not check for schema changes at all.

If you enable Auto Schema Update SnowMirror selects all columns for synchronization. You can exclude a column, though. Just put it back to the list on the left. If a new columns is found in ServiceNow, it is automatically added to the synchronized columns.

It is important to note that some kind of changes (new column, change of data type) have to perform clean synchronization! Some changes do not have to perform clean synchronization (increase of max. length of a column).
new sync auto schema update
Figure 1. Auto Schema Update enabled and one column is excluded

10.2.8. Include Inherited Columns (only table, view and backup synchronizations)

In ServiceNow a table can extend another table. In case the table extends another table then you will see a checkbox called Include Inherited Columns. If you check this checkbox, you can also synchronize columns defined in the parent table. If the checkbox is not checked columns from parent table are hidden and won’t be synchronized.

new sync include inherited

10.2.9. Reference Fields

Allows you to choose how SnowMirror handles Display values.

  • Only sys_ids - SnowMirror does not download display values. It just downloads the raw value. For example reference fields will contain SysId of the referenced record.

  • Only Display Values - SnowMirror downloads and stores display values. It does not store raw values.

  • Both - SnowMirror downloads stores both raw values and display values. Display values are stored to a column with prefix dv_.

10.2.10. Table Rotation

ServiceNow supports table rotation. Table rotation allows you to split large tables into multiple smaller tables (so-called shards). If you access the large table, ServiceNow internally joins all its shards. However, that comes with performance issues. It is faster to access the data directly from the shards.

SnowMirror can be configured to download data from the shards instead of the large table. You have to configure this while creating the synchronization. This setting cannot be changed later on.

  1. Replicate Shards - data will be downloaded from shards each into its own mirror table.

  2. Replicate And Merge Shards - data will be downloaded from shards into a single mirror table.

  3. Ignore - data will be downloaded from a single table (not accessing the shards directly) into a single mirror table.

sync settings table rotation

10.2.11. Columns to Synchronize (only table, view and backup synchronization)

You don’t have to synchronize the whole table. You can select columns to synchronize in a two-sided multi-select component. SysId column is selected by default and is mandatory. You can add all other columns with >> button or select columns separately. Only columns on the right side of multi-select component will be synchronized.

new sync settings

SnowMirror caches the list of columns so that it does not have to connect to ServiceNow every time you open the page. If you don’t see your column, please click on the "Loaded at" link on the bottom-right side of the page and SnowMirror will refresh meta-data.

new sync refresh metadata

10.2.12. Retention Period and File Format (Backups Only)

Old backups are periodically deleted. By setting "Retention Period" you define how long they will be kept on the disc.

settings backup retention period

10.2.13. File Format (Backups Only)

Format allows you to choose how to store backups.

settings backup file formats

10.2.14. Count Records (Count Attachments)

At the very bottom of the Synchronization Settings form there is a button called Count Records (for attachments it is called Count Attachments). It calls ServiceNow to find out how many records are there in the table you are synchronizing. It also takes into account the encoded query. Only if the encoded query is invalid you are notified of the error. Otherwise, a message with a number of records pops up.

new sync records count
new sync attachments count

10.2.15. Validate Button

At the very bottom of the Synchronization Settings form there is a button called Validate. It helps you to discover potential issues which may occur during synchronization.

  1. Form validation - validates values provided in the form. Checks if all required values were provided. Checks that synchronization name is unique in SnowMirror and that mirror table with the same name does not already exist.

  2. Web service access - checks if the accessible using Direct Web Services. If disabled login to your ServiceNow instance and open System Definition → Tables page. Select your table and on "Application Access" tab check "Allow access to this table via web services" option.

  3. Replication columns - checks if columns used during synchronization are present in your table. Checks if "sys_updated_on" column exists. If not, it checks the presence of "sys_created_on" column.

  4. Auditable - checks if Audit deleting strategy is selected then in checks if the table is auditable. If not, SnowMirror won’t be able to synchronize deleted records. To enable auditing of deleted records open your ServiceNow instance and go to System Definition → Dictionary page. Find and open a record where Table is "Some name of your table" and Type is "Collection". Then check "Audit" checkbox.

  5. Table Rotation - checks whether the table is rotated/extended in ServiceNow and checks if appropriate "Table Rotation" is selected in SnowMirror.

  6. Count records - count number of records in the ServiceNow table

  7. Diff deleting strategy - checks the size of the mirrored table and warns you if the table is too big to be synchronized using Diff deleting strategy.

  8. Indexes - validates if ServiceNow table contains all necessary indexes. Warns you if you don’t have the indexes. Without them the performance of your ServiceNow instance may be negatively influenced during synchronization. To create the index login to your ServiceNow instance and go to System Definition → Tables page. Select your table and open Database Indexes tab. You can create the index there.

  9. Archiving support - In case Archiving plugin support is enabled, it validates database indexes of "sys_archive_log" table. We expect indexes on "restored", "from_table" and "sys_created_on" columns. To create the index login to your ServiceNow instance and go to System Definition → Tables page. Select "sys_archive_log" table and open Database Indexes tab. You can create the indexes there.

  10. Table size limit - Lite edition is allowed to download only a limited amounts of records. This validation warns you if you exceed the limit. It is not performed in other editions than Lite.

validate button dialog

10.3. Schedule

Every synchronization can be triggered manually anytime, but it has also mandatory schedulers, which define execution plan.

There are two types of schedulers:

  • Incremental Load Scheduler - When it starts for the first time it downloads all available data. Next synchronizations only download increments (new records, updated records and removes deleted records). This is the preferred way to synchronize data.

  • Additional Scheduler - If "Incremental Load Scheduler" is not good enough, you can define an additional scheduler. Additional scheduler allows you to choose a type of synchronization to run (incremental, clean, differential, synchronize from).

    You can add an unlimited number of additional schedulers. To add a new scheduler, click "Create additional scheduler" button.

    Type Description

    Downloads only records created, updated or deleted since the last successful synchronization.

    Cleans all data from mirror database and then downloads all data.

    Inserts all missing and deletes all redundant records. It does not update any records.

    SnowMirror downloads only records whose sys_updated_on column contains a value greater than or equal to the date you chose. By default, SnowMirror download data from the last successful synchronization. You can specify an offset in minutes. If the last successful synchronized date is 2018-12-31 11:00:00 and the offset is 60 minutes the synchronization would download data from 2018-12-31 10:00:00

    new sync settings scheduler additional scheduler sync from
If Incremental Load Scheduler is running and "Clean and Synchronize" kicks in then the "Clean and Synchronize" is postponed and executed after the Incremental Load Scheduler finishes. SnowMirror expects that the "Clean and Synchronize" runs less often (one a week, one a month, …​) hence it has to wait to execute.
On the other hand if "Clean and Synchronize" is running and Incremental Scheduler kicks in then the Incremental Load won’t be executed - it is not necessary because the "Clean and synchronize" downloads all data anyway.

There are five approaches how to schedule a synchronization - daily, weekly, periodically, manually and cron. See details below.

10.3.1. Daily

Choose this option when you want to execute your synchronization every day exactly at the same time.

  • Active Since - The day, when the synchronization will get active.

  • Time - The time, when the synchronization will be triggered every day.

new sync scheduler

10.3.2. Weekly

This option allows you to run synchronization every week at selected days and time.

  • Active Since - The day, when the synchronization will get active.

  • Days of Week - You have to select one or more days, when the synchronization will be triggered.

  • Time - The time, when the synchronization will be triggered every selected day.

new sync scheduler weekly

10.3.3. Periodically

  • Active Since - The day, when the synchronization will get active.

  • Time - The time, when the synchronization will get active.

  • Interval - Time interval in minutes between synchronization runs. Available values are natural numbers in the range of 1 to 1440 (i.e. one day).

new sync scheduler periodically

10.3.4. Cron

A cron expression is a string consisting of six or seven fields that describe individual details of the schedule. These fields, separated by white space, can contain any of the allowed values, shown in the expected order in the table below, with various combinations of the special characters described in the cron manual.

Cron expression have to be compatible with a format of Quartz Scheduler cron expressions (not Linux Cron expressions). Quartz and Linux crons are similar but not the same. See Official Quartz documentation for details, especially Examples chapter.
Field Allowed Values

Seconds

0-59

Minutes

0-59

Hours

0-23

Day of month

1-31

Month

0-11 or JAN-DEC

Day of week

1-7 or SUN-SAT

Examples:

0 0 12 * * ?

Fire at 12:00 PM (noon) every day.

0 0/5 14 * * ?

Fire every 5 minutes starting at 2:00 PM and ending at 2:55 PM, every day.

0 15 10 ? * MON-FRI

Fire at 10:15 AM every Monday, Tuesday, Wednesday, Thursday and Friday.
Pay attention to the effects of '?' and '*' in the day-of-week and day-of-month fields!

10.3.5. Manually

Last approach of scheduling is the manual execution. There is no automatic plan, you can just run the synchronization whenever it is necessary.

10.4. Scheduler Priority

Each synchronization has a certain priority (the default value being "normal"). If several synchronizations are waiting to be executed then synchronization with a higher priority will execute first.

There are five priorities: Highest, high, normal, low, lowest

The settings can be found on Scheduler page under Advanced Settings section.

settings schedule priority

10.5. Run Immediately

Use this option in case you want to run the new synchronization immediately.

10.6. Finalization

When synchronization configuration is done, synchronization overview is showed. Created synchronization has question mark in the result column because it has not been triggered yet. You can check synchronization settings by clicking on its name.

new sync overview

11. New Bulk Synchronizations

SnowMirror offers a way to create multiple new synchronizations at a time. Pages and forms are very similar to the ones for creating a single synchronization. To create or add new synchronizations use the New synchronization button in the bottom of synchronization overview and then click Bulk Create button.

bulk sync create new

11.1. Tables or Views to Synchronize

On this page you can choose multiple tables or views for which you want to create a synchronization for.

When creating Bulk synchronization you can create either Attachment, Table or View synchronization. You cannot create more types in one wizard.
bulk sync create choose

11.2. Synchronizations Settings

The form is similar to the form for creating a single synchronization, but some fields are missing. SnowMirror assigns a value to these fields automatically:

  • Synchronization name will match the name of the table (view) from ServiceNow (for attachment synchronization "Synchronization name" will match the name of the table from ServiceNow with _attachment suffix)

  • Mirror database table name will match the name of the table (view) from ServiceNow (for attachment synchronization Mirror database table name will match the name of the table from ServiceNow with _attachment suffix)

  • All columns will be added synchronized by default. You can only choose if you also want to synchronize columns inherited from parent tables. For attachment synchronization no columns will be added.

bulk sync create settings

Attachments synchronization

bulk sync create settings attachments

In case a synchronization of a table, or a view already exists, you will be prompted to resolve the conflict.

  • Ignore - a synchronization for with the duplicate name won’t be created

  • Rename - a new synchronization will be created with a new automatically assigned name

bulk sync create resolve duplicates

11.3. Schedule

The form is exactly the same as the one for creating a single synchronization. See Schedule for details.

11.4. Run Immediately

Use this option in case you want to run the new synchronization.

12. Synchronization Detail

The following controls are available for all synchronizations on their detail page. The Detail page is accessible by clicking on it the name of the synchronization on Dashboard or on the List of synchronizations.

12.1. Detail Page

12.1.1. Controls

sync detail controls

Synchronize Now

Manually starts a synchronization out of its schedule. Scheduled starts are not affected.

Clean and Synchronize

Deletes all data in mirror table and synchronizes all data from scratch. Two activity logs will be created. One for data deletion and the second one for data download.

Differential Synchronization

Inserts all missing and deletes all redundant records. It does not synchronize updated records.

Synchronize…​

Opens a new page which allows you to start a synchronization using a fine-grained settings.

Synchronize From

Opens a popup window where you can specify a date and time you want to start the synchronization from. After selecting the date and time the synchronization starts.

Resume

Resumes the last synchronization run and continues from the point of failure.

Deactivate

You can temporarily prevent synchronization from triggering. If you disable a synchronization, you will not be able to manually trigger it.

Edit Settings

Edit synchronization name, name of table in mirror DB, delete strategy, encoded query or synchronized columns. See Synchronization Settings.
Settings are not editable during running synchronization.

Edit Mappings

Allows you to explore and edit data types to which the columns will be mapped.

Edit Scheduler

Edit the interval of triggering of the synchronization.

Delete

Deletes synchronization. Deletes rows from configuration DB and drops table from mirror DB. All data in the mirror DB table will be lost.

12.1.2. Status

Shows the status of the latest synchronization run.

sync detail status
Status Description:

State

If the synchronization is running, the State shows the current progress. If the synchronization has already finished, it shows its result.

Show Activity

A link to the Activity page where you can see details of the latest synchronization run and understand what SnowMirror was doing during the synchronization.

12.1.3. Basic Information

Basic information about the synchronized table.

sync detail basic info

Basic information about the synchronized attachments.

sync detail basic info attachment
Basic Information Table:

ServiceNow Table Name

Name of the table in ServiceNow.

ServiceNow Encoded Query

An encoded query used for restricting synchronized records (attachments). Useful when you want to synchronize - for example - just opened incidents. If empty, no query is applied.

Target Mirror DB

since 5.7.0Name of the database connection that is used to store synchronization data.

Table Name in Mirror DB

Name of the target table in mirror DB.

Local system directory

This is the directory where your synchronized attachments will be saved. Displayed only in attachments synchronization.

Created by

User who created this synchronization.

Created on

Time of synchronization creation.

Updated by

The last user who updated synchronization scheduler or synchronization settings.

Updated On

Time of last synchronization update.

Delete strategy

Strategy to determine which rows have been deleted. For more information see Delete Migration Strategy.

12.1.4. Migrated Records per Synchronization

Chart shows volume of migrated data in each synchronization. By default, it shows 20 latest synchronization runs.

migrated records per sync graph

12.1.5. Synchronization Duration

Chart shows time that each synchronization run has taken. Shows same number of synchronization runs as the chart above.
Number of shown columns is configurable by the same property as the Migrated Records per Synchronization.

sync duration

12.1.6. Schedule

Contains information about Incremental Load scheduling of the synchronization. See Scheduler Settings.

Schedule Table:

Synchronization Interval

Scheduling type. See Scheduler Settings.

Time

Displayed in case of daily or weekly synchronization interval. Shows the time that triggers synchronization on selected days.

Day of Week

Displayed solely in case of weekly synchronization interval. Shows days on which the synchronization is triggered.

Cron

Displayed in case of cron scheduler type. Shows cron expression used to schedule the synchronization. For more info about cron expression see Quartz scheduler.

Next Run

Shows date and time when synchronization will be triggered next time.
sync detail schedule

12.1.7. Full Load Scheduler

Contains information about Full Load scheduling of the synchronization. Contains the same fields as Incremental Load scheduling. See previous chapter for details.

12.1.8. Last Run

Shows comprehensive information about last or currently running synchronization.

sync detail last run
Last Run Table:

Started by

Shows who triggered last run. In case of manual synchronization shows name of user who started it. If a run was triggered automatically by scheduler, it shows value <SCHEDULER> regardless who set up the scheduler.

Started on

Time of last synchronization start.

Duration

Time that synchronization run has taken.

Inserted Records

Number of new records inserted into mirror DB.

Updated Records

Number of records updated in mirror DB.

Deleted Records

Number of records deleted from mirror DB.

Total Processed Records

Sum of inserted, updated and deleted records.

12.1.9. Synchronized Columns

Shows columns that are synchronized and their data type and length in ServiceNow.

sync detail sync columns
Synchronized Columns

Column Name

Name of the column.

ServiceNow Data Type

Data type of the column in ServiceNow.

Length

Max. length of the column.

Index

Show an icon for every database index on the column. If you hover mouse pointer on the icon, you can see details of the index.

12.1.10. Activate/Deactivate

You can suspend launching synchronization by clicking to Deactivate synchronization button. After that you can still change synchronization or scheduler settings. After set up, you can reactivate the synchronization by clicking on Activate Synchronization button.

12.1.11. Synchronize…​

Synchronize…​ allows you to execute a synchronization and fine-tune how to execute it.

  • Synchronization - What kind of synchronization you want to run.

    • Incremental Synchronization

      • Synchronize From - The synchronization download data created/updated/deleted since this date and time.

        • Last Successful Synchronization - SnowMirror will use date and time based on the last successful synchronization run.

        • Selected Date/Time - SnowMirror will use date and time specified by the user on this page.

          • Synchronize From Date/Time - date and time from which to download data

      • Database Update

        • All Records - stores all downloaded records to the database. It overrides the records that were already in the database.

        • Modified Records - stores only records which were updated. Before updating the data it compares timestamps (sys_updated_on from ServiceNow with mirror_updated_on from mirror database). Then it stores just the records whose sys_updated_on is newer than mirror_updated_on.

      • Encoded Query - The synchronization will download records matching the query and will insert/update these records. The query will be used only for a single synchronization run. It won’t be stored for later use. Delete does not use this query at all. Deletes are performed with the original query (if defined).

    • Clean And Synchronize

      • Use Temporary Table - synchronize directly to mirror table or to a temporary table

    • Differential Synchronization

      • Mode

        • Regular - download records to a mirror table as described in Differential Synchronization chapter

        • Dry Run - similar to Regular with one important exception. Data are stored to a separate log file. Mirror table is not modified in any way.

      • Check Updates - Differential synchronization compares SysIds in ServiceNow with SysIds in mirror database. Then it downloads missing and removes excessive records. On top of that it can check for updated records by comparing sys_updated_on from ServiceNow with sys_updated_on from mirror database.

        • Enabled - SnowMirror compares sys_updated_on from ServiceNow with sys_updated_on from mirror database. It downloads and stores updated to mirror database.

        • Disabled - SnowMirror does not compare sys_updated_on column. It does not check for updated records.

  • Scheduler Queue

    • Regular - the synchronization is going to wait in the queue until it gets its turn

    • New Thread - the synchronization is going to skip the queue and will be executed immediately in a new thread

  • Priority - All synchronizations share worker threads. The higher the priority the more often your synchronization gets a worker thread assigned.

  • Advanced Settings - addition settings no so commonly used.

synchronizeDotDotDot

12.2. Activity

Activity page provides you with information on what was happening during the synchronization.

12.2.1. Top-Level Overview

On the top of the page, there is an image providing a top-level overview of the synchronization. Each circle is drawn in a different color.

  • Grey - The step was not executed.

  • Blue - The step is being executed.

  • Red - The step was executed and failed.

  • Yellow - The step was executed and finished with a warning. It requires your attention.

  • Green - The step was executed and finished successfully.

An example of successful synchronization:

sync detail activity workflow

An example of failed synchronization. Red color indicates that the whole synchronization failed. Grey steps were not executed at all.

sync detail activity failed workflow

12.2.2. Details

Right under the top-level overview image, you can find a list of steps with detailed information on what happened during the synchronization. If you need more information, you can click on the desired step to see its sub-steps. If you need even more information, you can click on a sub-step.

sync detail activity accordion

12.2.3. Activity Log

Every synchronization run has its own activity log. It is a text file which contains detailed information about all the activities performed during that synchronization. As a user, you should primarily use Activity page to understand the synchronization process. The log is a low-level instrument. It does not necessarily have to be user friendly. However, it is very helpful to our support team.

log error example

The logs are stored in SnowMirror’s installation directory in "logs/activityLogs". Each synchronization has its own directory there identified by its ID. Every activity logs has also its own meta-data records written in the configuration database.

12.3. Edit Mappings

Edit Mappings page allows you to explore and edit data types to which the columns will be mapped. For each column the table shows data type from ServiceNow, the data type we will use in mirror database, name of the column in ServiceNow and name of the column in mirror database.

Sometimes, ServiceNow exceeds the max. length of a column and returns more data. Because of that, the data may not fit into the mirrored column. The solution is simple - you choose a data type that suits your needs.

If you change the data type SnowMirror immediately changes the type in mirror database.

Change of the name is necessary in some cases. For example, Oracle database allows only 30 characters long names. Long column names are automatically shortened, and you can change the name if you want to.

sync detail edit mappings

The rows you have changed are painted in bold. If you click on a name of a column you will be redirected to the page where you can change the data type and name of that column.

sync detail edit mappings detail

12.4. Advanced Settings

Each synchronization has "Advanced Settings" page which allows you to configure features not usually used. It also allows you to override system properties for a particular synchronization.

  • Synchronization Tasks

    • Use Temporary Tables - This setting applies only to Clean And Synchronize.

      • No - Cleans all data in target table and then re-downloads them from scratch. If the table is large, it may take a while to download everything from scratch.

      • Yes (rename table) - SnowMirror downloads all data to a temporary table. After the synchronization finishes, it removes the original table and replaces it with the temporary table. This feature allows you to have all your data available during a long clean synchronization. On the other hand, you will lose all your custom permissions defined on the target table.

      • Yes (copy data)since 4.9.0 - SnowMirror downloads all data to a temporary table. After the synchronization finishes, it deletes all data from the target table and copy the data there. This feature allows you to have all your data available during a long clean synchronization. You will not lose any custom permissions defined on the target table. On the other hand, this option is slower than "Yes (rename table)."

      • System - Use a global settings. The global settings can be configured using on Settings → General Settings page.

    • Consistency Check - enable/disable the Consistency Check for this synchronization (override global settings)

    • Synchronize Indexessince 4.12.2 - Should we synchronize database indexes or not? Available only in Enterprise edition and higher.

      • System - use global settings. The global settings can be configured using on Settings → General Settings page.

      • Yes - SnowMirror will find out what indexes are defined in ServiceNow and replicate them to the mirror database.

      • No - SnowMirror will ignore indexes.

    • Archiving Support - enable/disable Archiving tables support for this synchronization. If set it overrides global settings from General Settings.

  • Performance

    • Download All Updated - When an encoded query is defined SnowMirror has to download all updated SysIds (without the query applied). It has to be done so that SnowMirror can find records which ceased to match the query. Some tables are never updated. For those tables you can disable the Download of All Updated SysIds and hereby stop making unnecessary requests and improve replication times.

    • Count Records - SnowMirror counts records so that it can show a nice progress bar and verify if it downloaded all records. In case you have a huge table and ServiceNow times out, you can turn off counting records.

  • Check Updates

    • Diff Check Updates - Controls how differential synchronization check updated records.

      • System - uses a global settings. The global settings can be configured using snowMirror.replication.diffSync.checkUpdates property.

      • Yes - checks for updated records regardless the global settings.

      • No - does not check updated records regardless the global settings.

    • Database Check Updates - Controls which records can be updated.

      • System - uses a global settings. The global settings can be configured using snowMirror.replication. databaseCheckUpdates property.

      • Yes - updates only those records whose sys_updated_on column is greater than the one in mirror database.

      • No - updates all downloaded records regardless its sys_updated_on column value.

  • Synchronization Parameters

    • Incremental Column - Name of a column which is used for querying only updated or created records.

    • Incremental Offset - Offset in seconds. Incremental sync. starts at a certain time minus a value of this property. This allows us to pickup changes which started before that certain time but were committed to ServiceNow database later.

      Leave empty if you want to use global (system) settings. To change the system settings go to Advanced Settings page and change a value of "snowMirror.replication.startOffsetInSeconds" property.

  • Auto Schema Update

    • Use Default Value - When Auto Schema Update is Enabled (no truncation) new columns are created. This option allows you to choose how to treat new columns.

      • System - use the global settings. The global settings can be configured using on Settings → General Settings page.

      • Yes - SnowMirror fills in default value from sys_dictionary into new columns.

      • No - SnowMirror does not fill in default value. New columns are left empty.

  • Records Delete - SnowMirror is able to find out when a record is deleted in ServiceNow and then delete it in your mirror database. Records Delete allows you to choose how to delete the record.

    • System - uses a global settings. The global settings can be configured on Settings → General Settings page.

    • Hard - record is irreversibly deleted from mirror database.

    • Soft - record is not deleted and "mirror_deleted_on" column is filled in. The column contains date and time when SnowMirror found out it was deleted.

      SnowMirror deletes all records (including soft-deleted records) in case:

      • Clean and synchronize is called

      • Truncate delete strategy is used

      • A new column is added using Auto Schema Update

        If you switch from Soft to Hard, "mirror_delete_on" column will be dropped during the next synchronization run.

  • Data Truncationsince 4.7.12 - If enabled SnowMirror truncates large String values so that they fit into database columns.

    • System - use a global settings. The global settings can be configured using on Settings → General Settings page.

    • Enabled - SnowMirror truncates large String value regardless the global settings.

    • Disabled - SnowMirror does not truncate any data regardless the global settings.

  • Missing Columns Warning - If enabled a synchronization ends up in a warning status when ServiceNow does not return all columns.

    • System - use global settings. The global settings can be configured using on Settings → General Settings page.

    • Enabled - A synchronization ends up in a warning status when ServiceNow does not return all columns.

    • Disabled - A synchronization does not end up in a warning state because of missing columns regardless the global settings.

  • Directory Name Field - Attachments are stored to a directory with a following structure: "SyncName/RecordDisplayValue". This option allows you to customize the "RecordDisplayValue" part.

    It must contain a column name. A value of this column is used as a directory name.

    If you change this setting, you have to run a clean synchronization to download all attachments from scratch!

  • Read Replica Statesince 4.11.0 - Internally, ServiceNow maintains a copy their database for read-only queries. You can configure SnowMirror to request data from these read-only replicas. That may improve a performance of your ServiceNow instance.

    • System - SnowMirror uses global settings. The global settings can be configured on Settings → ServiceNow page.

    • Enabled - SnowMirror will request the data from read-only replica. The name of the replica must be provided in Read Replica Name field.

    • Disabled - SnowMirror will not read data from read-only replica.

      Synchronizations can overwrite this configuration.

  • Read Replica Namesince 4.11.0 - Internally, ServiceNow maintains a copy their database for read-only queries. You can configure SnowMirror to request data from these read-only replicas. That may improve a performance of your ServiceNow instance.

    In this field, you can provide the name of your read-only replica. Synchronizations can overwrite this configuration.

  • Keys Batch Sizesince 4.14.2 - Number of sys_ids downloaded in a single request. Default is 10,000. Increasing this batch size usually does not have significant performance effect.

    • Empty value - use global settings from Settings → Performance page.

    • Non-empty value - Overwrites the number of keys in a single batch.

  • Records Batch Sizesince 4.14.2 - Number of full records downloaded in a single request. Default is 250 for Soap API and 1,000 for Rest API and Performance API. Increasing this batch size for Soap or Rest usually does not have significant performance effect. For Performance API, see performance tuning chapter.

    • Empty value - use global settings from Settings → Performance page.

    • Non-empty value - Overwrites the number of records in a single batch.

  • Log Levelsince 5.0.6 - How much information to write into activity logs.</p>\

    • System - use global settings (normal)

    • Most Detailed - Most detailed information. Useful for debugging.

    • Detailed - Detailed information. Useful for debugging.

    • Normal - Usual amount of information.

  • Index Typesince 5.5.4 - A type of index to use when creating a table.</p>\

    • System - use global settings.

    • Clustered Columnstore Index - use clustered columnstore index.

    • Heap - use heap.

    • Clustered Index - use clustered index

    Warning: if you change this option, SnowMirror won’t change the existing table. We recommend you to change this option only once before you run the synchronization for the first time.

advanced settings

12.5. History

A list of activity logs for all synchronization runs. These activity logs are listed on the History tab on Synchronization Detail page.

History Table

Result

Result icon shows icon, which represents result of the last synchronization. There are following possibilities:

ok icon

success - data from ServiceNow was successfully synchronized to mirror database

warning icon

warning - data from ServiceNow was synchronized to mirror database but SnowMirror detected potential issues. The synchronization requires your attention.

error icon

fail - data could not be synchronized. An error occurred during the synchronization

running icon

running - the synchronization is currently running

canceling icon

cancelling - synchronization task is being canceled

error icon

unknown result - the synchronization was created but it was not executed yet

waiting icon

waiting - the synchronization is waiting in the queue to be executed +

disabled icon

disabled - the synchronization is deactivated and can’t be executed until you reactivate it.

warning disabled icon

disabled with warning - the last synchronization’s run ended up with a warning. The synchronization is deactivated and can’t be executed until you reactivate it.

error disabled icon

disabled with error - the last synchronization’s run ended up with error. The synchronization is deactivated and can’t be executed. It can be reactivated on Synchronization Detail Page.

Start

Date and time when the synchronization was launched.

End

Date and time when the synchronization was stopped (by any cause - finished, canceled etc.)

Started by

Shows who triggered last run. In case of manual synchronization shows name of user who started it. If a run was triggered automatically by scheduler, it shows value <SCHEDULER> regardless who set up the scheduler.

Duration

Is the time from start to finish of the last synchronization run.

Inserted

Number of new records inserted into mirror DB.

Updated

Number of records updated in mirror DB.

Deleted

Number of records deleted from mirror DB

Total

Sum of inserted, updated and deleted records.

12.5.1. Filters and Sorting

To get a sense of history data you can filter activity logs by its results. Just click the funnel icon to see the filtering form. See more on Filtering.

history filters

12.6. Data

A tab "Data" on Synchronization Detail page provides a simple data explorer of the synchronization’s mirror table.

13. Version Checker

When you open SnowMirror in a browser, it automatically checks for a new version. If a new version exists, you will see a blue notification in a page footer. If your SnowMirror is really old, you will see a red notification.

footer new version blue
footer new version red

13.1. How It Works

Version check runs in your browser (not the server). After you open SnowMirror in a browser, it downloads a small "version.js" file from our website. The file contains a version of the latest release. SnowMirror caches the file in a cookie for the next 24 hours. Only then we will check the latest version again.

The file on our website is available using HTTPS protocol (port 443). If your computer is not permitted to access and see our website we won’t show any notifications.

14. Data Types

SnowMirror stores data the same way as they are stored in the ServiceNow database - not the way how data is displayed in ServiceNow. Therefore, some data types need a special attention:

  • Journal fields - Journal fields are not stored directly in the original table (e.g. incident) but in a special system table called sys_journal_field. To view incidents together with comments you need to synchronize two tables (incident and sys_journal_field) and then perform a join to display the data together.

  • HTML - ServiceNow stores them as HTML. Therefore, SnowMirror stores them as HTML too.

    field html

  • Currency, Price - ServiceNow automatically converts currency fields and price fields to a currency of an active user. SnowMirror cannot change this behavior. The original (not converted) values are stored in fx_currency_instance (for currency fields) and fx_price (for price fields). You should create a synchronization for these tables and access your currency/price fields there.

    Example: How to synchronize currency field "salvage_value" from cmdb_model table.

    1. In ServiceNow, the value was entered in Euros.

      field currency sn

    2. cmdb_model in SnowMirror: Our ServiceNow integration user uses USD. Therefore, SnowMirror received the value converted to USD.

      field currency db cmdb model

    3. fx_currency_instance in SnowMirror: The value in this table is never converted. You can join cmdb_model with fx_currency_instance like this:

      SELECT *
FROM cmdb_model INNER JOIN fx_currency_instance
    ON cmdb_model.sys_id = fx_currency_instance.id
    AND fx_currency_instance.field = 'salvage_value';

      field currency db fx currency instance