Admin Guide, Rest API, Performance API, Data Viewer, SnowMirror Monitor, ServiceNow Upgrade, Error Codes, Release Notes

1. Getting Started

SnowMirror

1.1. Introduction

SnowMirror is a data replication tool for ServiceNow. It replicates data from ServiceNow into your local (on-premise) database.

SnowMirror is an on-premise software. You install it into your own ICT infrastructure. All data stays within your organization.

It provides a simple-to-use user interface, where you can choose which tables and columns you want to synchronize and how often you want to do that. The user interface also allows you to see a history of your synchronizations - what tables it synchronized, when it synchronized the data and how many records it downloaded.

SnowMirror does not modify or transform the data. It is up to you, how you use the data. Usually, customers use it for reporting. They connect their reporting tools to the mirror database and run reports against the replica. That is useful because they don’t have to run reports against a live ServiceNow environment. Customers also use SnowMirror for backups or as an integration point for other applications.

1.2. Architecture

SnowMirror reads data from ServiceNow using SOAP or REST API and stores it into your database. We call this process a synchronization. When SnowMirror receives data, it keeps it in memory (RAM). Once the data is permanently stored in mirror database, it may be freed from memory.

Configuration settings are stored in a configuration database. Data is stored in mirror database. SnowMirror does not modify the data - what it receives from API, is what it stores to the database.

SnowMirror’s user interface is accessible via an internet browser. Optionally, you can the check status of your synchronizations directly from within your ServiceNow instance using SnowMirror Monitor.

snowMirrorBasicScheme

Multiple SnowMirror installations with separate databases can be connected to single ServiceNow instance. It can also be installed into a high-availability cluster.

1.2.1. Databases

SnowMirror supports multiple databases: SQL Server, Oracle, MySQL, Postgres, Snowflake and Azure Synapse. See detailed information in Administrator Guide.

  • Configuration database - stores metadata, i.e. what to synchronize, when to synchronize, etc.

  • Mirror database - stores un-modified data mirrored from ServiceNow. The structure of tables and columns is a one-to-one of tables and columns from ServiceNow. You should never manipulate database schema from any other place than SnowMirror’s user interface.

    All tables contain additional columns mirror_created_on and mirror_updated_on. They contain a date and time when SnowMirror inserted/updated a record in the database.

1.2.2. Workflows

SnowMirror is able to download data from ServiceNow tables, view and is also able to download attachments. Synchronization is a process that downloads data from ServiceNow. We distinguish three different workflows (use-cases): replication, archive and backup.

  • Replication - Replication is a mirror of your ServiceNow data. Whatever change you make in ServiceNow, replication will make the same change too in SnowMirror database. When you create, update or delete record in ServiceNow, SnowMirror does the same in SnowMirror database.

    Replication is the most widely used workflow, and it is mainly used for reporting.

  • Archive: Archive offloads obsolete attachments and data from your ServiceNow instance according to the rules you define. When you delete data from ServiceNow, archive synchronization will not delete the data. Archive keeps your data safe even when you decide to delete it from ServiceNow.

    Archive only inserts data. It does not update or delete data. Therefore, it is useful only for obsolete and unused data.

    It synchronizes data using Differential Synchronization. It may be slower than the most widely used Incremental Synchronization. That should not be an issue because typically, you don’t need to archive your data every day. We expect you to archive data less often, usually once a month.

  • Backup: Backup offloads data into CSV or XML ServiceNow files and stores them into file system. If something happens with your data in ServiceNow, you still have the copy in a safe place.

    Data is being exported to CSV and XML files which are recognized by ServiceNow. Any update to a record in ServiceNow is stored to a new file in SnowMirror. If record gets updated, you will end up having multiple files with multiple snapshots of the records. We don’t try download every single update. We download updates only at scheduled times.

1.2.3. Workflow Comparison

Operations
Workflow Insert Update Delete

Replication

✓

✓

✓

Archive

✓

N/A

N/A

Backup

✓

✓

Partially
Source Data

SnowMirror is able to synchronize system tables, your own custom tables (u_) and tables from scoped applications. SnowMirror is not able to synchronize tables that do not have SysId column.

View synchronizations have limited option how to download deltas. Table synchronization is preferable.

Workflow Tables Attachments Views

Replication

✓

✓

✓

Archive

✓

✓

N/A

Backup

✓

✓

N/A
Target Storage

Currently, the following target data formats are supported.

Workflow Database Table Cloud CSV/Parquet File Local CSV/Parquet File Local XML File Cloud Attachments Local Attachments

Replication

✓

✓

N/A

N/A

N/A

✓

Archive

✓

✓

N/A

N/A

N/A

✓

Backup

N/A

N/A

✓

✓

N/A

✓

1.3. Migration Process

SnowMirror loads data from ServiceNow table using SOAP Web Services or REST API.

SOAP API replication algorithm: SnowMirror requests a batch of 10,000 SysIds which were created or updated in ServiceNow. After receiving SysIds, it loads data using these SysIds in batches of 250 records. It repeats these steps until it downloads all records.

REST API replication algorithm: SnowMirror requests a batch of 1,000 records which were created or updated in ServiceNow. It repeats these steps until it downloads all records.

SnowMirror processes multiple batches at the same time to improve the replication times. This is thoroughly explained in Performance Tuning chapter. Batch sizes are configurable on Performance Tuning page.

SnowMirror does not lock the table in ServiceNow because it only reads the data. If some other process updates and locks your table in ServiceNow, SnowMirror may have to wait until it is free to read the data.

SnowMirror creates and updates records in mirror database. Therefore, it has to lock affected records util it updates them. If some other process is accessing the mirror database table, SnowMirror may have to wait until it is free to commit the update.

To ensure correct synchronization process, please make sure you have correct date and time set on your computer.

1.3.1. Transactions

Data batches are transactional but whole synchronization runs are not. This prevents SnowMirror from reverting data in case of an error.

1.3.2. Migration Steps

synchronization process

  1. Initialization: Loads data from the configuration database and decides which data it will be synchronizing. It downloads meta-data from ServiceNow.

  2. Logs: Logs information about your environment to the activity log. This helps us to diagnose errors.

  3. Auto Schema Update: If enabled, SnowMirror finds out if the schema has changed and makes appropriate changes in mirror database. Optionally, it sends notifications about schema changes to an email of your choice.

  4. Count: Counts records in ServiceNow so that it know how many records it will be synchronizing.

  5. Create or Alter Table: If it does not already exist, it creates a new table in mirror database. Otherwise, it just updates the table (if necessary).

  6. Replication: This is the most important step. SnowMirror synchronizes data from ServiceNow. The replication process is a bit different based on which API you use.

    SOAP API

    SnowMirror requests a list of SysIds which were created or updated in ServiceNow. By default, it requests in a batch of 10,000 SysIds. It uses getKeys method from SOAP Web Services.

    After receiving SysIds, it loads data by these SysIds. By default, data are loaded in batches of 250 records. It uses getRecords method from SOAP Web Services.

    REST API

    SnowMirror requests records which were created or updated in ServiceNow. By default, it requests in a batch of 1,000 records. It uses REST Table Api

    Common to SOAP API and REST API

    If an encoded query is defined, SnowMirror checks for records which matched the query before, but now they don’t match it anymore. Such records are deleted from mirror database. To do that, SnowMirror load SysIds of all updated records (regardless the query) and compares them with SysIds of replicated records.

    As of version 4.0.0, SnowMirror processes multiple batches at the same time to improve the replication times. This is thoroughly explained in Performance Tuning chapter. Batch sizes are configurable on Performance Tuning page.

  7. Delete: Most ServiceNow tables store deleted data in sys_audit_delete table - i.e. deletes audited. This is the preferred way of determining deleted records. However, not every table is audited.

    For such cases, SnowMirror offers multiple deleting strategies. SnowMirror cannot determine if a table is audited or not. Note, that most tables in ServiceNow can be set as audited. If you want more information about auditing tables see ServiceNow Documentation on Auditing.

    Delete Strategies

    Audit

    Audit deleting strategy looks into sys_audit_delete table to find deleted records.

    To find deleted records of a synchronized table since the last synchronization, Audit Delete strategy filters data using tablename and sys_updated_on fields. You should have database index on these fields in ServiceNow.

    This is the most efficient strategy for finding deleted records. To be able to use this strategy, your table needs to audit deleted records in ServiceNow. To find out, whether deletes are audited on your table or not, please go to Edit Settings and use Validate button.

    Advanced topics:

    • Optionally, you SnowMirror can query data using sys_created_on field instead of sys_updated_on field. This is configurable via Advanced Settings using snowMirror.auditDelete.column property.

    • In ServiceNow, some tables make up a hierarchy (for example task → incident). Audit delete strategy has to look for deleted records not only from the table being synchronized but also from child tables. For example task → incident has to filter sys_audit_delete like this: tablename=task^ORtablename=incident.

    Diff

    Diff compares sys_ids in ServiceNow table with the sys_ids in mirror DB. Sys_ids that are present in mirror table, but not ServiceNow table are deleted. sys_updated_on and sys_created_on are intact.

    Fast Difference

    It counts records and downloads max. value of sys_updated_on column. If the values don’t match with the mirror database it splits the SysIds into smaller chunks and repeats until it gets until to 10,000 keys. Only then it downloads the keys and compares them with the mirror database. This strategy is useful when you have a large table and small amount of deletes because it reduces the number of requests to ServiceNow.

    None

    Does not delete any data from the mirror table. Insert and update tasks act as normal. Deleting data can be done manually by Clean and Synchronize button on the detail of the synchronization.

    Truncate (Deprecated)

    Truncate deleting strategy deletes all data from mirror table and downloads all records from scratch. It is deprecated - use Full Load Scheduler instead.
Default deleting strategy is Audit. If the synchronized table is not audited, consider auditing it. Use some other strategy only if the synchronized table is not audited.
When synchronizing a full load view, it is only possible to you use Truncate strategy

  1. Consistency check: Checks that number of records in ServiceNow matches the number of records in mirror database. It is explained in more detail in chapter Consistency Check.

  2. Finalize Migration: Updates the activity logs and finishes the synchronization.

1.3.3. Cache Management

For a better performance, SnowMirror caches ServiceNow metadata (data related to tables and its columns, parent-child structure, etc.). Because of this cache, some changes in ServiceNow can be taken into account with a delay.

If you want to clear the cache manually go to Settings menu, choose General Settings and click on Clear All Caches button.

Cache timeout is configurable through Advanced Settings.

1.3.4. ServiceNow Performance Considerations

  • Download times and number of requests

    The first initial load may take a while to finish. The download time depends on how many rows the table has, how many columns it has and how big the data inside the cells is. SnowMirror downloads data in batches, i.e. it makes a lot of small requests to download huge tables.

    It may take several minutes to download smaller tables or even several hours to download huge tables (millions of records).

    The initial load can be time-consuming. However,subsequent synchronizations will download just the changes (new, update and deleted records). Downloading changes is fast and does not impact the performance of your ServiceNow instance.

  • Database indexes in ServiceNow

    When SnowMirror queries data from ServiceNow, it extensively uses sys_updated_on column. If a table does not have sys_update_on column, it uses sys_created_on instead. To ensure fast processing times:

    • You should have an index on sys_updated_on column in ServiceNow.

    • If the table does not have sys_updated_on column, you should have an index on sys_created_on column.

    • If you have a table hierarchy, only the base table in the hierarchy needs to have the column indexed.

    You can use validate button to verify if you need to create the index. To create an index open ServiceNow and go to System Definition → Tables page. Select your table and open Database Indexes tab. You can create the index there.

1.3.5. Azure Synapse

since 4.10.0 The synchronization process is a bit different for Azure Synapse.

  1. SnowMirror downloads data from ServiceNow.

  2. SnowMirror stores the data into temporary files into local snow-mirror/data/synapse directory.

  3. Once the file reaches 100,000 records, SnowMirror uploads it to Azure Storage.

  4. SnowMirror copies data from Azure Storage into a stage table.

  5. At the end of a synchronization, SnowMirror merges stage table into the target table.

  6. At the end of a synchronization, SnowMirror deletes temporary files from local file system.

azure synapse

1.4. Display Values

In addition to raw values, ServiceNow allows its users to define so-called display values. Display
values are usually human-friendly texts which users understand more easily than raw values. For example:

  • Reference fields: When your records reference a user, ServiceNow has to store SysId of that user (raw value). The related display value can contain name of the user.

  • Choice fields: ServiceNow may store choices as numbers. The display values contains human-readable description of the choice.

SnowMirror allows you download display values of:

  • Fields of "reference", "domain_id", "document_id" or "glide_list".

  • Choices where choice value is greater than 0

1.4.1. Synchronization Strategies

SnowMirror provides you with three options how to handle display values.

  • SysIds - Reference field in mirror database contains SysId of the referenced record. An example: "incident" table contains "assigned_to" column with SysId of the referenced user.

  • Display Values - Reference field in mirror database contains display value of the referenced record. An example: "incident" table contains "assigned_to" column with display value of the referenced user (usually name and surname).

  • SysIds and Display Values - Reference field in mirror database contains SysId of the referenced record. Additional field is created to store the display value. Name of the display value column is prefixed with dv_. An example: "incident" table contains "assigned_to" column with SysId of the referenced user and "dv_assigned_to" column with display value (name and surname of the referenced user).

Note that there is a limitation imposed on display value synchronization. If display value changes and the record which references the display value does not change SnowMirror cannot discover the change. The display value is only updated when the referencing record changes too.

1.4.2. Display Value Replication Algorithm

Let’s focus on the reference fields. How does ServiceNow evaluate which field to use as a display value? It is not so straightforward as it might seem.
Imagine a table hierarchy Security Incident → Incident → Task. It is possible to specify display values in each table or just in one of them.
It is possible to omit the display values at all and rely on the defaults and finally, have you heard of Dictionary Overrides?

Here is the actual evaluation algorithm ServiceNow uses:

  • Dictionary Override – Current Table – The highest priority in the evaluation algorithm has a flag called Override Display Value in the Dictionary Override (i.e. table called sys_dictionary_override). It is possible to set a different display value then the parent table is specifying. E.g. the Task table has the field number as a display value. In the child table Security Incident it is possible to set up an override configuring short_description as a display value.

  • Dictionary Override – Parent Table – same as previous. If there is no override in the referenced table then the algorithm searches for overrides in the table hierarchy from the child to the parent. E.g. override in the Incident table takes precedence over a display value field specified in the Security Incident table but not over an override in the Security Incident table.

  • Display value field – Current Table – if there are no overrides then the standard display value fields come into play. The display value on the child table overrides the display value on the parent. E.g. Security Incident can specify its own field u_security_code as a display value for security incidents which would override the number field in Task.

  • Display value field – Parent Table – same as previous. If there is no display value field in the referenced table then the algorithm searches for display fields in the table hierarchy from the child to the parent.

  • Default fields – even if there are no display values explicitly set up, ServiceNow searches for default fields to make them as display value fields. These fields are “name“, “u_name“, * “sys_name“, “number“, “u_number“. The algorithm first searches the whole table hierarchy for a field called name. Then, if not found, it searches for u_name, etc. Please note that the child-parent mechanism does not work here. Name has always precedence over Number.

  • SysId – if everything fails then ServiceNow uses sys_id as a display value.

Note that the display value field can be another reference field or can be a choice list field. Transitive relation applies here so ServiceNow is traversing through the display values to find the last in the row.

1.5. Views Synchronization

Migration process described in previous chapters works perfectly when synchronizing tables. Beside tables ServiceNow allows you to define database views. SnowMirror supports views synchronization with some limitations. There are several issues SnowMirror has to tackle:

  • A record in a view is a Cartesian product of records from several tables. That means there is no unique and permanent SysId identifying the record.

  • A record in a view does not have to contain "sys_created_on" and "sys_updated_on" columns.

  • There is no delete log for views in ServiceNow.

SnowMirror provides two options to synchronize a view:

  • Full Load - full load synchronization deletes data from mirror database and then downloads all data from ServiceNow. This way you are guaranteed to have the most recent and 100 percent accurate copy of your data in mirror database. Depending on the amount of your data the synchronization may take a long time to perform.

  • Incremental Load - incremental load synchronization downloads only increments, i.e. new created, updated and deleted data. To use this synchronization type you have to choose a master table. SnowMirror uses this table to find out the increments. This means if records in other tables are changed SnowMirror would not find out. Also, the master table should not contain any duplicate SysId values.
    Incremental Load View synchronization requires less amount of time to perform at the cost of not having 100 percent accurate copy of your data. To tackle this issue we recommend to set up Full Load Scheduler to run once in a while because it clears data from mirror database and downloads all data from ServiceNow.

1.6. Attachments synchronization

Attachments synchronization process is very similar to table synchronization process. This section lists some differences between table synchronization and attachment synchronization.

1.6.1. Migration steps

Create directory if not exists

Besides configuration and mirror database, attachment synchronization needs access to the file system so that it can store attachments there. During the synchronization, SnowMirror creates a directory for attachments (if it does not already exist).

Path to the directory is defined by snowMirror.attachment.rootDir property which can be configured in Advanced properties. Be sure to have sufficient rights for creating directories and files in this directory.

Replication Using SOAP API

  1. SnowMirror loads serviceNow.number.of.keys keys (by default, this is 10,000) which were created or updated since the last synchronization run. The keys are loaded from sys_attachment table.

  2. Attachment meta-data are loaded for these keys. Meta-data are loaded in batches by serviceNow.number.of.records records (by default this is 250).

  3. Based on the meta-data, SnowMirror start to download the attachments from sys_attachment_doc table. SnowMirror repeatedly requests data from this table because it returns the attachments divided into small chunks. SnowMirror downloads serviceNow.number.of.attachment.chunks chunks (by default this is 500) per one request. Then it writes the chunks into temporary files and joins them to make up the whole attachment file.

  4. After downloading attachments their meta-data are stored to Mirror table and temporary files are moved to snowMirror.attachment.rootDir. If there already exists file with same name, new file is renamed (suffix is added to filename).

Replication Using REST API

  1. SnowMirror loads attachment meta-data using REST Attachment API. It loads the meta-data in batches of serviceNow.number.of.records records (by default, this is 1,000). It always loads records which were created or updated since the last synchronization run.

  2. Based on the meta-data, SnowMirror downloads the attachments one by one using REST Attachment API.

  3. After downloading attachments their meta-data are stored to Mirror table and temporary files are moved to snowMirror.attachment.rootDir. If there already exists file with same name, new file is renamed (suffix is added to filename).

Delete strategies
Delete Strategies for Attachments

Audit

Audit deleting strategy is not available when synchronizing attachments

Diff

Diff compares sys_ids in ServiceNow table with the sys_ids in mirror DB. Sys_ids that are present in mirror table, but not ServiceNow table are deleted (with their corresponding attachment file). sys_updated_on and sys_created_on are intact.

Fast Difference

It counts records and downloads max. value of sys_updated_on column. If the values don’t match with the mirror database it splits the SysIds into smaller chunks and repeats until it gets until to 10,000 keys. Only then it downloads the keys and compares them with the mirror database. This strategy is useful when you have a large table and small amount of deletes because it reduces the number of requests to ServiceNow.

None

Does not delete any data from the mirror table or attachment directory. Insert and update tasks act as normal. Deleting data can be done manually by Clean and Synchronize button on the detail of the synchronization.

Truncate (Deprecated)

Truncate deleting strategy deletes all data from mirror table and all attachments from local directory before the insert task. Update task is skipped. Note, that sys_updated_on and sys_created_on columns are set to time of last synchronization start. It can not be used to determine the time of actual row insert or update in ServiceNow. It is deprecated - use Full Load Scheduler instead.

1.6.2. Attachments Directory

Content of snowMirror.attachment.rootDir has following structure:
<snowMirror.attachment.rootDir>/<synchronization name>/<display value> where:

  • snowMirror.attachment.rootDir - configurable directory where attachments are stored

  • synchronization name is a name of a synchronization that you filled in Synchronization settings form

  • display value is a value of a field which is marked as display value for the given table

1.6.3. Encoded Query

If you have entered Encoded query it is applied on related table (e.g. incident) and NOT on sys_attachment table.
While getting keys of created (or updated) attachments since the last synchronization another additional request is made on related table (i.e. incident, task, …​) so that SnowMirror can learn which records fit encoded query. This assures that only attachments for records which fit encoded query are downloaded.

1.7. Index Replication

SnowMirror allows you to replicate not only tables and data but also database indexes. If you define an index in your ServiceNow instance SnowMirror will pick up the change and create the index in your mirror database. That is particularly useful if you make complicated SQL queries to your mirror database and need to improve the performance of these queries.

You can enable index replication on General Settings page or per-synchronization in Advanced Settings.

1.7.1. Custom Indexes

What if you need an index in the mirror database, but you don’t want to create it in your ServerNow instance? Don’t worry, we have it covered. You can create the index manually in your database, and we will preserve it intact.

1.7.2. SysId Columns

ServiceNow defines primary keys on sys_id columns. SnowMirror ignores these keys because it always creates its own primary key on "sys_id" column.

1.7.3. Performance Implications

If have you enabled Auto Schema Update or set up Notifications on schema change SnowMirror will send one additional request per synchronization. In the case of views, it will send several requests depending on how many tables the view consists of.

The request itself is not resource expensive and time-consuming. Yet, it should be taken into consideration especially when SnowMirror is configured to run many synchronizations very often.

1.7.4. Limitations

In some cases, ServiceNow Web services return us encoded column names. That happens when a column name is too long. It also happens on few columns on "task" table and its descendants. We cannot then couple the column in mirror database with a column of the ServiceNow index. Therefore, we ignore those indexes.

1.8. Synchronization Types

This chapter describes all types of synchronizations you can encounter in SnowMirror.

1.8.1. Incremental Synchronization

SnowMirror requests and downloads only records inserted, updated or deleted since the last synchronization. This is the most commonly used type of synchronization.

  • New and updated records - incremental synchronization uses sys_updated_on column to query the data. Thanks to that column, SnowMirror is able to download just the new and updated records.

  • Deleted records - there are multiple strategies how to find deleted records.

1.8.2. Clean and Synchronize

SnowMirror cleans all data in a mirrored table and downloads all records from scratch.

After the clean, it may take a long time to synchronize the data again. The old data won’t be available during the synchronization. To tackle the issue you may want to enable "snowMirror.replication.cleanAndSyncToTmpTable" which synchronizes the data to temporary table. Only after the synchronization finishes successfully the original mirror table is removed and replaced by the temporary table.

Optionally, you can enable Clean Synchronization to temporary table. If enabled, SnowMirror downloads data during a clean synchronization to a temporary table. After the synchronization finishes, it drops the old table and replaces it with the temporary table. This feature allows you to have all your data available during a long clean synchronization. On the other hand, you will lose all your custom permissions and indexes defined on the target table.

1.8.3. Differential Synchronization

Inserts all missing and deletes all excessive records. By default, it does not update any records. Optionally, it can be configured to check for updates too.

Differential synchronizations works as follows:

  1. It requests the first 10,000 SysIds from ServiceNow ordered alphabetically.

  2. It loads the first 10,000 SysIds from the mirror database ordered alphabetically.

  3. It computes a difference to find out new and deleted SysIds. If it cannot decide if the record is new or deleted it will process it in the next batch.

  4. It repeats the first three steps until there is no records left to process.

If checking for updates is enabled, it also has to load and compare values of sys_updated_on column.

Differential synchronization is particularly useful if you need to repair your mirror table. See Consistency Check chapter for examples.

1.8.4. Synchronize From

SnowMirror downloads only records whose sys_updated_on column contains a value greater than or equal to the date you chose.

This is useful if your synchronization missed or ignored some records, you have fixed the root cause of the issue, and now you want to download the missing records.

1.8.5. Resume

Let’s say you have a long-running synchronization and halfway through the synchronization process it fails (for whatever reason). It’s inconvenient to run it again because it takes some time to do so. "Resume" functionality is here to rescue.

It allows you to resume a failed synchronization. If you resume a synchronization, SnowMirror starts to download data from the point of the failure.

SnowMirror can resume table and view synchronizations. Backups and attachments are currently not supported.

To resume a synchronization, open Synchronization Detail page and click Resume button.

1.9. Archiving Support

ServiceNow provides optional Data Archiving Plugin. It allows users to store older data into archive tables.

If you use Archiving in ServiceNow, you may want to enable support in SnowMirror too. You can do that globally in General Settings or per synchronization in Advanced Settings.

Here’s what SnowMirror does when archiving is enabled:

  • When Audit Deleting Strategy is used, SnowMirror checks sys_archive_log to find archived records. Then, it deletes the records from mirror database. We use the following columns from sys_archive_log:

    • sys_created_on - to find new records in "sys_archive_log" since the last synchronization.

    • restored - to find whether a record was restored (i.e., is not archived anymore) or not.

    • from_table - to get archived records of a specific table. When a table has child tables, we also look for records related to these child tables. Example:

      • When we look for archived records in "incident", our query contains: from_tableINincident

      • When we look for archived records in "task", our query contains even child tables: from_tableINtask,incident,etc.

  • Checks sys_archive_log table so that it can insert restored records to the mirror table.

  • Synchronization of Archive tables (those starting with "ar_") - New records are picked up using "sys_archived" column. Updated records are picked up using "sys_archive_restored". That’s different from ordinary (non "ar_") tables where we pick up inserted and updated records only using "sys_updated_on" column.

1.10. Consistency Check

In some cases SnowMirror may miss synchronizing some records.

Imagine that you imported data to your table in ServiceNow and all records had sys_updated_on set on some old date. SnowMirror already synchronized newer data hence it will miss the newly imported records. This is when Consistency check comes in handy. After every synchronization run it counts the number of records in ServiceNow and number of records in mirrored table. Then it compares the counts and if it finds out that they are not equal it will put the synchronization into Warning state and send notification email (if configured to do so).

consistency check list

When you encounter such a warning you should start looking for a reason of the inconsistency. To tackle the issue with old dates in imported records you can use differential synchronization. To diagnose any other potential issues you may want to use validate button.

consistency check activity log

See Admin Guide on how to configure consistency check.

1.11. Mirror Database Constraints And Indexes

SnowMirror creates a primary key on "sys_id" column in every mirror table. Thanks to the key, it can update records really fast.

Older SnowMirror installations used a unique key instead of a primary key. Those installations keep using unique key, even after upgrade. New (clean) installations use a primary key.

In case of Full Load Views, the records consist of several joined tables. Therefore, there is no single sys_id, and we cannot create a primary key.

You can even configure SnowMirror to use unique indexes instead of primary keys. Go to Settings → Advanced Settings page and set a property "snowMirror.replication.index.sysIdConstraintType" to "UNIQUE_INDEX".

2. Backups

SnowMirror allows you to copy and archive data from ServiceNow so that it may be restored in case of data loss. It supports backups of tables and attachments.

Data from tables can be stored to CSV or XML files. Those are valid files recognized by ServiceNow. You can use them to import your data back to ServiceNow.

2.1. Backup Strategies

SnowMirror supports three backup strategies. You can run them manually or automatically by a scheduler.

  1. Incremental - downloads changes made since the last backup (no matter if it was full or incremental).

  2. Differential - downloads changes made since the last full backup.

  3. Full - downloads all data.

2.2. Location

All backups are stored to snow-mirror/data/backups directory. It can be changed on General Settings page using "Backups" field. Please make sure to have sufficient permissions for creating directories and files in this directory.

change directory

Each backup synchronization creates its own directory with the name of the synchronization.

Sample Backup Structure:
snow-mirror/data/backups
├───incident
│       2581_FULL_2016-01-08_07h-43m-50s.zip
│       2582_INCREMENTAL_2016-01-08_07h-44m-35s.zip
│       2583_INCREMENTAL_2016-01-08_07h-46m-35s.zip
│
└───sys_user
        2581_FULL_2016-01-08_07h-43m-50s.zip
        2582_INCREMENTAL_2016-01-08_07h-44m-35s.zip
        2583_INCREMENTAL_2016-01-08_07h-46m-35s.zip

2.3. Retention Period

Every day SnowMirror runs a job which removes old backups. Each synchronization defines a "retention period". The job will remove backups older that the retention period.

2.4. Table Backup

The backup synchronization works the same as table synchronization works except a few exceptions:

  • Backups do not use the mirror database at all. They are stored exclusively on file system in files.

  • SnowMirror stores durations and timers to mirror database as either a number of milliseconds or as a date and time. Backups do not use these settings so that the fields can be imported back to ServiceNow.
    XML files store timers and durations as a date and time (timestamp). CSV files store them as a number of seconds.

  • SnowMirror allows a user to store dates and times in whatever timezone she wishes (default being UTC). Backups can only store dates and times in UTC so that the values can be imported back to ServiceNow.

  • SnowMirror can’t synchronize columns which are not marked as "active" in the Dictionary. It also can’t synchronize data from child tables - e.g. when exporting "task" (parent) table you won’t see any "incident" (child) records in the export. If you want to back up child tables you have to create a new synchronization for each child.

2.4.1. File Formats

You can choose from the following formats:

  • CSV - records are store in comma separated file format. ServiceNow can run all necessary business rules during the import.

  • XML - records are stored in XML file. Unlike CSV it can even store deleted records. ServiceNow won’t run any business rules during the import!

The CSV or XML files are always compressed into a ZIP file which highly reduces the size of the backup.

The file has the following format: [ActivityLogId_TypeOfTheSynchronization_Date_Time].

  • ActivityLogId - ID of the activity log created during the backup

  • TypeOfTheSynchronization - FULL or INCREMENTAL. See Synchronization Types chapter for details.

  • Date_Time - date and time when the synchronization started

Example:
2581_FULL_2016-01-08_07h-43m-50s.zip

2.4.2. Create and Edit

For detailed description of how to create a new synchronization please New Synchronization Wizard. Also, see Retention Period (Backups Only) and File Format (Backups Only) sections describing backup specific fields.

Here, we will just discuss several specifics related to backups.

  • CSV

    • Deleting Strategy - you cannot choose any deleting strategy, because CSV file can’t store deleted records.

    • Reference Fields - defaults to Only Display Values so that it behaves like ServiceNow does. However, you can choose whatever option you want to.

    • Columns - you can choose columns you want to synchronize.

  • XML

    • Deleting Strategy - if you choose Audit deleting strategy, and your table is audited then all deleted records will be stored in the XML file.

    • Reference Fields - you won’t have to choose any Reference Fields strategy. XML backups have to always download both value and its display value.

    • Columns - all columns are synchronized to stay compatible with ServiceNow XML files.

new sync

2.4.3. Import To ServiceNow

Backup files are valid CSV or XML files used by ServiceNow. See ServiceNow documentation for detail how to import those files.

Please take a time to study how import of CSV and XML files works and know the implications.
Importing CSV files

  1. Load Data into a new table. Open System Import Sets → Load Data page import data into a new table. Let ServiceNow create a new table with all columns for you. All columns will be of a "string" type.

    load data

  2. Create a transform map

    Open System Import Sets → Create Transform Map page and create a new transform map. Choose the new table created in the previous step as a source table and the table where you want to import the data as target table.

    create transform map

    After submitting the form, you have to map fields from the source table to fields in the target table. We recommend using "Auto Map Matching Fields" or "Mapping Assist" for that purpose. Mapping is thoroughly described in ServiceNow documentation. If you have never used transform maps, we recommend you to read ServiceNow documentation. In this tutorial we pinpoint just a few important steps:

    • Coalesce Sys ID column - configure Sys ID column in a target table to coalesce. This makes ServiceNow to match source values with values from existing records. Simply said this will allow you to update existing records with the records in your import set. See ServiceNow documentation for detailed description. Note that you will see two Sys ID columns in the source table: "Sys ID" and "sys_id". Please choose "sys_id" column - that is the one from you import set.

      coalesce sys id

    • Date/Time format - please make sure the mapping of Date and Date/Time fields uses the correct format.

      date field format

  3. Run the transformation - Open System Import Sets → Run Transform page and choose your import set and transform map. Click Transform button and ServiceNow will load the data to the target table.

Importing XML files

Please follow the documentation on ServiceNow documentation.

ServiceNow will store the data directly to the database and won’t run any business rules during the import!

The import procedure is simple. On the target table choose Import XML and then upload your XML backup file to ServiceNow. Then just click "Upload" and ServiceNow will start the import.

import xml

2.5. Attachments Backup

Attachments are stored into a directory structure which is packed into a single ZIP file. See the sample structure below.

Sample Backup Structure:
snow-mirror/data/backups/
├───incident
│       2581_FULL_2016-01-08_07h-43m-50s.zip
│      └───INC0000001
│            └───FirstAttachmentOfInc0000001.png
│            └───SecondAttachmentOfInc0000001.png
│       2582_INCREMENTAL_2016-01-08_07h-44m-35s.zip
│      └───INC0000001
│            └───ThirdAttachmentOfInc0000001.png

It is worth mentioning that such a directory structure cannot store deleted files. If you delete an attachment in ServiceNow it won’t be tracked in ZIP file. Therefore, you should perform Full backup from time to time.

2.5.1. Create and Edit

For detailed description of how to create a new synchronization please New Synchronization Wizard. Also, see Retention Period (Backups Only) section.

3. Cloud Storage

SnowMirror allows you to store data from ServiceNow as CSV or parquet files into cloud storage. We support Amazon S3, Azure Storage and Google Storage. Once you have your data stored in cloud storage, you can query it with other cloud tools. For example, Amazon Redshift can query files in S3.

SnowMirror stores data into a bucket. Every synchronization has its own folder there and is further structured into folders. Updated records are written into new files. Therefore, you end up having multiple snapshots of records in multiple files.

3.1. Bucket Structure

SnowMirror stores data into a bucket. Every synchronization has its own folder there and is further structured into folders:

  • data - contains new and updated records

  • deleted - contains sys_ids of deleted records

  • archived - contains sys_ids of deleted records

Sample File Structure in Cloud Storage:
your-bucket
├─── incident
│    ├─── data
│    │    ├─── 2022-12-31_13-00-00-100.csv.gzip
│    │    ├─── 2022-12-31_13-01-00-100.csv.gzip
│    │    └─── 2022-12-31_14-01-00-100.csv.gzip
│    ├─── deleted
│    │    ├─── 2022-12-31_13-01-01-100.csv.gzip
│    │    └─── 2022-12-31_14-01-01-100.csv.gzip
│    └─── archived
│         ├─── 2022-12-31_13-01-02-100.csv.gzip
│         └─── 2022-12-31_14-01-02-100.csv.gzip
└─── sys_user
     └─── data
          ├─── 2022-12-31_13-00-00-100.csv.gzip
          └─── 2022-12-31_14-01-00-100.csv.gzip

The structure looks the same for parquet files. Parquet are named like "2022-12-31_13-01-01-100.parquet" and are not compressed with gzip.

3.2. Updates

Updated records are written into new files. Therefore, you end up having multiple snapshots of records in multiple files. There are two options how to recognize which snapshot is newer:

  • File names contain a timestamp of when the file was created.

  • Every CSV or parquet file contains a column named "mirror_created_on". It contains a date and time when the record was written into the file.

Updated records are stored to new files and the old files are kept intact. That means, you can find historical values of your records. However, this is not a replacement for "sys_audit" table. SnowMirror does not capture every change. It captures changes at scheduled intervals. If you, for example, download data every 30 minutes, and you update your record multiple times during that timeframe, SnowMirror captures only the last value.

3.3. Deletes

SnowMirror is able to recognize deleted records only if deletes in ServiceNow table are audited in ServiceNow. That means, it supports only so-called Audit Delete Strategy.

3.4. File Format

SnowMirror stores data into CSV or parquet files. CSV files are further compressed with Gzip. CSV stands for comma-separated values. The first line in each file is a header - i.e. a list of column names. All the other lines contain your ServiceNow data.

Parquet file is a column-oriented data storage. Parquet file contains a schema so that you can see what kind of data we store there. Date/time columns are stored as a number of milliseconds from epoch. You can store date/time columns as string, see "mirror.parquet.dateTimeFormat" advanced property.

The first column is "sys_id" column. The last column is "mirror_created_on". This column does not exist in ServiceNow. SnowMirror adds it so that you can see date and time when the record was written into the file.

Sample CSV File with New and Updated Records
"sys_id","assigned_to","dv_assigned_to","number","mirror_created_on"
"46b66a40a9fe198101f243dfbc79033d","5137153cc611227c000bbd1bd8cd2007","David Loo","INC0000009","2023-02-01 12:35:23"
"46b9490da9fe1981003c938dab89bda3","9ee1b13dc6112271007f9d0efdb69cd0","Don Goodliffe","INC0000010","2023-02-01 12:35:23"

Files with deleted records contain just "sys_id" column and "mirror_created_on" column.

Sample CSV File with SysIds of Deleted Records
"sys_id","mirror_created_on"
"9b33214d1bbf6410593f42e58d4bcb28","2023-02-01 12:43:32"
"2eace1f4db1f73005c7ad92b5e9619a0","2023-02-01 12:43:32"

Files with archived records contain just "sys_id" column and "mirror_created_on" column.

Sample CSV File with SysIds of Archived Records
"sys_id","mirror_created_on"
"9b33214d1bbf6410593f42e58d4bcb28","2023-02-01 12:43:32"
"2eace1f4db1f73005c7ad92b5e9619a0","2023-02-01 12:43:32"

3.5. Consistency Check

SnowMirror accesses cloud storage using libraries provided by the cloud vendor. These libraries do not allow us to query the files. Therefore, we can run a consistency check only when running an uninterrupted clean synchronization.

3.6. Database vs. Files

Storing data into cloud storage is supported since SnowMirror 5.0.0 (February 2023). Up until then, we supported only relational databases. Therefore, it may be useful to summarize differences between synchronization to relational databases and cloud storages.

Feature Relational Database Cloud Storage

Updates

Existing records in the database are updated.

Updated records are written into new files and existing files are kept intact.

Deletes

Can utilize Audit, Differential, Fast Difference delete strategies.

Deleted records can be identified using Audit delete strategy. Differential and Fast Difference delete strategies are not supported.

Differential synchronization

Supported.

Not supported (we cannot query data in cloud storage).

Consistency Check

Executed at the end of every synchronization.

Executed only at the end of an uninterrupted clean synchronization.

Soft Delete

Optionally, records can be "soft" deleted. I.e. instead of deleting a record from database, we keep it there and mark it as deleted.

Sys_id columns of deleted records are written into new files. Existing files are kept intact. Therefore, records are "soft" deleted by default.

4. Installation

For quick installation see Installation section in Admin Guide.

5. Licensing

To be able to synchronize data, you need to enter a valid license key. Please visit our website to see license types, support levels and supported features. You can request a trial key on our website.

If you don’t enter a valid license, SnowMirror will show a warning at the top of each page.

no license entered

If your license is about to expire this message will appear in the page header:

license expiring

6. Logging in/out

Every user has a unique username for logging in and one or more user roles. User roles give him or her a permission to access application functionality. The name of the logged-in user is shown on the right side of top-horizontal menu.

If you want to log out, use the Logout link, which is placed on the right side of top menu next to username of currently logged-in user.

login top menu

User roles can be added or removed only by Super Administrators. For more information about roles and permissions see User Roles).

6.1. Permanent login

After 4 hours of inactivity, SnowMirror logs you out. In case you want to stay logged in longer (2 weeks), you can use "Remember Me" checkbox.

login remember me

When you access a login page directly (URL http://address-of-your-snowmirror:9090/login) it always shows the login page (regardless the "Remember Me" checkbox). All other pages will let you in automatically if "Remember Me" is checked.

The "Remember Me" option creates a cookie in your browser containing an access token. The cookie is removed by your browser after 2 weeks of inactivity or if you click "Logout" button. Otherwise, you stay logged in.

7. User Roles

Every SnowMirror users have to have at least one of the following roles.

Role Permissions

Synchronizations

Data Viewer since 6.0.0

User Management

Settings

REST API

Read

Run

Create

Edit

Use

Own profile

All Users

Notifications

All Other

Use

Synchronization Reader

✓

N/A

N/A

N/A

✓

✓

N/A

N/A

N/A

N/A

Synchronization Supervisor

✓

✓

N/A

N/A

✓

✓

N/A

N/A

N/A

N/A

Synchronization Administrator

✓

✓

✓

✓

✓

✓

N/A

✓

N/A

N/A

Super Administrator

✓

✓

✓

✓

✓

✓

✓

✓

✓

N/A

Data Viewer since 6.0.0

N/A

N/A

N/A

N/A

✓

✓

N/A

N/A

N/A

N/A

API Caller

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

✓

8. Dashboard

Dashboard is the main page of SnowMirror and it is divided into three parts - list of failed synchronizations, graph of migrated data volume and last / upcoming synchronizations.

8.1. Failed Synchronizations

'Failed Synchronizations' shows synchronizations that failed during the last run. Failed synchronizations shows name of synchronization and time of failure. By clicking on the synchronization name you can obtain more information about the synchronization see Synchronization Detail.

8.2. Graph of Migrated Records

Graph of migrated records shows global statistic of all transferred data per day. It counts a sum of all inserted/updated/deleted records from all synchronizations executed during a particular day. Every each color represents a kind of data operation similar to progressbar.

Much more information is available on the detail of the synchronization. It is possible to use a link on every synchronization name in the table to go to synchronization detail.

migrated records graph

8.3. Upcoming Synchronizations

Upcoming synchronizations tab shows all synchronizations which are currently running or will be executed in next 12 hours by default. Last synchronizations show all synchronization that were triggered in past 12 hours. This interval can be set in the property file.
It includes basic details like time of synchronization execution, result of the last execution and progressbar. Contents are sortable.

Last / Upcoming Synchronizations Table:

Name

Name of the synchronization. It is chosen by a user and it does not have to match the table name in mirror database or the name of the table in ServiceNow.

Next Run

Next Run is a date and time of the next scheduled run. Synchronizations with manual execution scheduler have this field left empty.

Starts In

Shows a time left until the next synchronization run starts.

Synchronization Progress

The progress bar shows a ratio of migrated records. It is continuously updated and shows the current progress of the synchronization. The progress bar is made of three colors:

green icon green - data which were inserted in mirror DB

blue icon blue - data which were updated in mirror DB

grey icon grey - records deleted from mirror DB

If the last synchronization had not migrated any records, the progress bar background is white.

8.4. Last Synchronizations

Last synchronizations tab shows all synchronizations that were triggered in past 12 hours.

It includes basic details like the start and end time of synchronization run, result of the last run and progressbar. Contents are sortable.

Last / Upcoming Synchronizations Table:

Result

An icon representing a result of the last synchronization run.

Name

Name of the synchronization. It is chosen by a user and it does not have to match the table name in mirror database or the name of the table in ServiceNow.

Start Date

Date and time when the synchronization started.

End Date

Date and time when the synchronization finished.

Duration

How had the synchronization been running.

Synchronization Progress

The progress bar shows a ratio of migrated records. It is continuously updated and shows the current progress of the synchronization. The progress bar is made of three colors:

green icon green - data which were inserted in mirror DB

blue icon blue - data which were updated in mirror DB

grey icon grey - records deleted from mirror DB

If the last synchronization had not migrated any records, the progress bar background is white.

9. Synchronization Overview

Synchronization Overview includes a list of all created synchronizations. Table includes the preview of basic information and is sortable.

sync overview table
Synchronization Overview Table

Result

An icon representing a result of the last synchronization run. SnowMirror recognizes the following results:

ok icon

success - data from ServiceNow was successfully synchronized to mirror database

warning icon

warning - data from ServiceNow was synchronized to mirror database but SnowMirror detected potential issues. The synchronization requires your attention.

error icon

fail - data could not be synchronized. An error occurred during the synchronization

running icon

running - the synchronization is currently running

canceling icon

cancelling - synchronization task is being canceled

error icon

unknown result - the synchronization was created but it was not executed yet

waiting icon

waiting - the synchronization is waiting in the queue to be executed +

disabled icon

disabled - the synchronization is deactivated and can’t be executed until you reactivate it.

warning disabled icon

disabled with warning - the last synchronization’s run ended up with a warning. The synchronization is deactivated and can’t be executed until you reactivate it.

error disabled icon

disabled with error - the last synchronization’s run ended up with error. The synchronization is deactivated and can’t be executed. It can be reactivated on Synchronization Detail Page.

Name

Name of the synchronization. It is chosen by a user, and it does not have to match mirror DB table name or the name of the table in ServiceNow.

Next Run

Next Run is a date and time of the next scheduled synchronization run in format mm/dd/yyyy hh:mm. Synchronizations with manual execution scheduler have this field left empty.

Last Run

Last Run is a date and time of the last synchronization run (no matter if it was successful or not). The format of the date ant time is mm/dd/yyyy hh:mm.

Duration

A time how long it took to finish the synchronization.

Progress

The progress bar shows a ratio of migrated records. It is continuously updated and shows the current progress of the synchronization. The progress bar is made of three colors:

green icon green - data which were inserted in mirror DB

blue icon blue - data which were updated in mirror DB

grey icon grey - records deleted from mirror DB

If the last synchronization had not migrated any records, the progress bar background is white.

9.1. Synchronization Overview Controls

  1. New Synchronization - Redirects to the New Synchronization Wizard page.

  2. Synchronize - Triggers synchronization of selected synchronizations.

  3. Drop-down menu

    1. Execute

      1. Clean and Synchronize - Deletes all mirror table data and immediately synchronizes selected synchronizations. Neither sys_updated nor sys_created on are used. That means all data are loaded from ServiceNow table.

      2. Synchronize From - Synchronize From downloads only records whose sys_updated_on column contains a value greater than or equal to the date you chose.

      3. Differential Synchronization - Inserts all missing and deletes all redundant records. It does not update any records.

      4. Resume - Resumes the last synchronization run and continues from the point of failure.

      5. Stop - Stops selected synchronizations if they are running.

    2. Manage

      1. Edit Scheduler - Redirects to edit scheduler page. Can be used to schedule multiple synchronizations (selected by checkbox).

      2. Activate - Enables deactivated synchronizations. After activation, synchronizations will be able to synchronize data from ServiceNow.

      3. Deactivate - Temporarily prevents selected synchronizations from triggering. If you disable synchronizations, you will not be able to manually trigger them.

      4. Delete - Permanently deletes selected synchronizations and (optionally) all its data from mirror database.

      5. Export - Exports definitions of selected synchronizations. See Administrator Guide for details on Import/Export.

9.2. Filtering

Synchronizations can be filtered by all columns available. To open the filtering form click the funnel icon. After filling in filtering conditions click the Run button and SnowMirror will show only synchronizations matching the conditions.

Also note that there is word "All" next to the funnel icon. This is breadcrumbs area. It shows which filter is applied. The word "All" means no filter is applied.

filtering overview filter

9.2.1. Filtering Options

Each field has a predefined set of operators which you can use to filter its values. Generally, there are four types of operators:

  1. Text - is, is not, is empty, is not empty, contains, does not contain, starts with, ends with

    Used for filtering text fields such as:

    • Name - name of the synchronization,

    • Encoded query - encoded query of the synchronization

    • Mirror table - name of the table in mirror database

    • Created by - name of who created the synchronization

    • Updated by - name of the last person who updated the synchronization

  2. Numbers - is, is not, greater than, greater than or equal to, less than, less than or equal to, between

    Used for filtering numbers in fields such as:

    • Inserted - number of records inserted during the synchronization

    • Updated - number of records updated during the synchronization

    • Deleted - number of records deleted during the synchronization

    • Total - total number of records processed during the synchronization (inserted + updated + deleted)

  3. Options - you are only allowed to choose one of the predefined values. Used on fields:

    • Synchronization Type - Table, view, attachment or backup

    • Result

      • Successful - finished its execution successfully

      • Warning - finished its execution, but a user should check inconsistencies in records count

      • Failed - finished its execution with error

      • New - synchronization has never been executed

      • Scheduled - synchronization is waiting to be executed

      • Running - synchronization is being executed

      • Finished - synchronization finished its execution (no matter if it succeeded or failed)

    • State - Active or inactive

    • Reference Fields - SysIds, Display Values, SysIds and Display Values

    • Delete Strategy - Audit, truncate, difference or none

    • Include Inherited Columns - Yes or no

    • Auto Schema Update - Yes or no

    • Synchronization Interval - Daily, weekly, periodically, cron or manually

    • Synchronization Interval (Full) - Daily, weekly, periodically, cron or manually

    • Execution Type - Type of full load scheduler: Clean and synchronize or differential synchronization

    • since 5.7.0Target database - name of the database connection that is used to store synchronization data

    • since 5.8.0Tag - tag assigned to the synchronization

  4. Date and time - today, yesterday, tomorrow, Last minute, Last 15 minutes, last 30 minutes, last hour,

    • Duration - how long it took to finish the synchronization

    • Last Run - date and time when the synchronization finished

    • Created On - date and time when the synchronization was created

    • Updated On - date and time when the synchronization was updated

9.2.2. Cancelling Filter

You can cancel the current filter by clicking the "All" breadcrumb.

9.2.3. Breadcrumbs

Breadcrumbs work the same way as they do in ServiceNow GUI:

  1. Click on one breadcrumb and only the one filter will be applied.

  2. Click the arrow separator, and the following breadcrumb will be removed from filter condition.

filtering breadcrumbs

9.3. Queue

Synchronization queue provides you a list of synchronizations which are currently running or waiting to be executed. Synchronizations are executed from the top of the list to the bottom.

queue sample

To open the queue, click on the "Show Queue" link on Dashboard.

queue link

10. New Synchronization Wizard

To create a new synchronization use the New Synchronization button on the top of Synchronization Overview (you can open it by clicking "Synchronizations" in the top menu). This button starts a three steps wizard which will guide you step-by-step through process of synchronization creation.

10.1. Table to Synchronize

First, you have to choose what kind of synchronization you want to create. You can replicate, archive or backup tables, views and attachments.

  • Workflow: Please choose use-case and a reason why you want to replicate ServiceNow data.

    • Replication: Replication is mirrors your ServiceNow data. Whatever change you make in ServiceNow, replication will make the same change too in SnowMirror database. It is most widely used for reporting.

    • Archive - Archive offloads obsolete attachments and data from your ServiceNow instance according to the rules you define. When you delete data from ServiceNow, archive synchronization will not delete the data. Archive keeps your data safe even when you decide to delete it from ServiceNow.

    • Backup: It offloads data into CSV or XML ServiceNow files and stores them into file system. If something happens with your data in ServiceNow, you still have the copy in a safe place.

  • Source: What do your want synchronize from ServiceNow.

    • Resource: ServiceNow stores the majority of data in tables. We are also able to synchronize attachments and views.

    • Table: Name of the table that you want to synchronize. The list of names is alphabetically ordered and supports quick search with autocomplete.

  • Target Choose how and where to store your data.

    • Data Format: Database table, CSV files, XML files or parquet files

    • Database/File System: Name of the database or file system where your want to store your data

Click "Next" button to move to the next step.

SnowMirror caches the list of tables/views so that it does not have to connect to ServiceNow every time you open the page. If you don’t see your table/view, please click on the "Loaded at" link on the bottom-right side of the page and SnowMirror will refresh meta-data.

new sync select table

10.1.1. Client Script For a View

Views support incremental updates only partially. To overcome the limitations on incremental synchronization you can synchronize tables involved in a view and create a view in your database manually. Before creating database view, all tables involved should exist in mirror database. If some tables are missing, please create and run synchronizations first.

Click Generate SQL button to generate the client SQL script.