1. Getting Started

SnowMirror

1.1. Introduction

SnowMirror is data replication tool for ServiceNow. The goal of SnowMirror is to replicate data from ServiceNow into a local database. Data is transferred via direct web services and saved into local database within user’s ICT infrastructure. SnowMirror enables to manage synchronized data easily and use them for other processing such as computing summary statistics, creating data analysis etc. The data is accessible anytime without frequent requests to ServiceNow, which could be time consuming and impacts its performance.

SnowMirror provides user friendly graphic interface. It is easy to set or edit which ServiceNow tables and columns will be synchronized, when and how often. A user can create, view and edit synchronization settings, runs, view synchronization details, history, synchronized data or synchronization logs directly in SnowMirror.

1.2. Architecture

SnowMirror loads data from ServiceNow via web service and saves it into mirror DB. This process is called synchronization. Configuration of synchronization is stored in configuration DB. Synchronization can load data from one ServiceNow table and save it to one mirror DB table. Multiple synchronizations can load data from same ServiceNow table, but each synchronization has its own mirror DB table. For more information about synchronizations see Migration Process and New Synchronization.

snowMirrorBasicScheme

Multiple SnowMirror instances with separate databases can be connected to single ServiceNow instance.

Synchronizations are managed via embedded web server that also provides comprehensive information about synchronization health and performance. For more information see Dashboard.

Optional SnowMirror ServiceNow Application also shows you synchronization health and performance without need to log into SnowMirror at all. It shows running state of all SnowMirrors connected to the ServiceNow instance and complex information about synchronizations. For more information see SnowMirror Monitor User Manual.

1.2.1. Database

Both configuration and mirror database can be provided by wide range of DBMS like MySQL, Postgres, SQL Server, Oracle and H2. If you are creating databases manually, use UTF-8 encoding.

Configuration database

Configuration database contains data about users and roles, synchronization logs and of course about synchronization tasks.

Mirror database

Mirrored data from ServiceNow are stored in mirror database. A structure of tables and columns is created as a copy of tables and columns from ServiceNow. You should never manipulate database structure from any other place than SnowMirror GUI. All of the data is just mirrored ServiceNow tables (or parts of tables) without any transformations. The only additional table columns are mirror_created_on and mirror_updated_on which contains date and time of row creation and last update. The "mirror_" prefix is configurable.

It is necessary to set both databases to use UTC timezone as a default timezone for all date - related records. This will help you to reduce possibility of date inconsistency because ServiceNow is using UTC too.

1.3. Migration Process

This chapter covers tasks performed during a synchronization run.

SnowMirror loads data from ServiceNow table using SOAP Web Services or REST API.

When SOAP API is used it requests a list of SysIds which were created or updated in ServiceNow. By default, it requests in a batch of 10,000 SysIds. After receiving SysIds it loads data by these SysIds. By default, data are loaded in batches of 250 records.

When REST API is used it requests a records which were created or updated in ServiceNow. By default, it requests data in a batch of 1,000 records.

SnowMirror processes multiple batches at the same time to improve the replication times. This is thoroughly explained in Performance Tuning chapter. Batch sizes are configurable on Performance Tuning page.

To ensure correct synchronization process, please make sure you have correct date and time set on your computer.

1.3.1. Transactions

Data batches are transactional but whole synchronization runs are not. This prevents SnowMirror from reverting data in case of an error.

1.3.2. Migration Steps

synchronization process
  1. Initialization: Loads data from the configuration database and decides which data it will be synchronizing. It download meta-data from ServiceNow.

  2. Logs: Logs information about your environment to the activity log. This helps us to diagnose errors.

  3. Auto Schema Update: If enabled, SnowMirror finds out if the schema has changed and makes appropriate changes in mirror database. Optionally, it sends notifications about schema changes to an email of your choice.

  4. Count: Counts records in ServiceNow so that it know how many records it will be synchronizing.

  5. Create or Alter Table: If it does not already exists, it creates a new table in mirror database. Otherwise, it just updates the table (if necessary).

  6. Replication: This is the most important step. SnowMirror synchronizes data from ServiceNow. The replication process is a bit different based on which API you use.

    SOAP API

    SnowMirror requests a list of SysIds which were created or updated in ServiceNow. By default, it requests in a batch of 10,000 SysIds. It uses getKeys method from SOAP Web Services.

    After receiving SysIds it loads data by these SysIds. By default, data are loaded in batches of 250 records. It uses getRecords method from SOAP Web Services.

    REST API

    SnowMirror requests records which were created or updated in ServiceNow. By default, it requests in a batch of 1,000 records. It useshttps://docs.servicenow.com/bundle/london-application-development/page/integrate/inbound-rest/concept/c_TableAPI.html[REST Table Api]

    Common to SOAP API and REST API

    If an encoded query is defined, SnowMirror checks for records which matched the query before but now they don’t match it anymore. Such records are deleted from mirror database. To do that, SnowMirror load SysIds of all updated records (regardless the query) and compares them with SysIds of replicated records.

    As of version 4.0.0, SnowMirror processes multiple batches at the same time to improve the replication times. This is thoroughly explained in Performance Tuning chapter. Batch sizes are configurable on Performance Tuning page.

  7. Delete: Most ServiceNow tables store deleted data in sys_audit_delete table - i.e. deletes audited. This is the preferred way of determining deleted records. However, not every table is audited.

    For such cases, SnowMirror offers multiple deleting strategies. SnowMirror cannot determine if a table is audited or not. Note, that most tables in ServiceNow can be set as audited. If you want more information about auditing tables see ServiceNow Documentation on Auditing.

    Delete Strategies

    Audit

    Audit deleting strategy gets deleted records via the sys_audit_delete table.
    Due to the data representation in this table, SnowMirror has to build child structure of the synchronized table. It can take several requests to ServiceNow data dictionary. The child structure is then cached for future synchronizations. If you want more info about cache see Cache Management.

    Diff

    Diff compares sys_ids in ServiceNow table with the sys_ids in mirror DB. Sys_ids that are present in mirror table, but not ServiceNow table are deleted. sys_updated_on and sys_created_on are intact.

    Fast Difference

    It counts records and downloads max. value of sys_updated_on column. If the values don’t match with the mirror database it splits the SysIds into smaller chunks and repeats until it gets until to 10,000 keys. Only then it download the keys and compares them with the mirror database. This strategy is useful when you have a large table and small amount of deletes because it reduces the number of requests to ServiceNow.

    None

    Does not delete any data from the mirror table. Insert and update tasks act as normal. Deleting data can be done manually by Clean and Synchronize button on detail of the synchronization (See Synchronization Detail ).

    Truncate (Deprecated)

    Truncate deleting strategy deletes all data from mirror table and downloads all records from scratch. It is deprecated - use Full Load Scheduler instead.

Default deleting strategy is Audit. If the synchronized table is not audited, consider auditing it. Use some other strategy only if the synchronized table is not audited.
When synchronizing a full load view, it is only possible to you use Truncate strategy
  1. Consistency check: Checks that number of records in ServiceNow matches the number of records in mirror database. It is explained in more detail in chapter Consistency Check.

  2. Finalize Migration: Updates the activity logs and finishes the synchronization.

1.3.3. Cache Management

For a better performance, SnowMirror caches ServiceNow metadata (data related to tables and its columns, parent-child structure, etc.). Because of this cache some changes in ServiceNow can be taken into account with a delay.

If you want to clear the cache manually go to Settings menu, choose General Settings and click on Clear All Caches button.

Cache timeout is configurable through Advanced Settings.

1.3.4. ServiceNow Performance Considerations

SnowMirror queries data from ServiceNow and extensively uses "sys_updated_on" column. For the best performance, you should have an index on "sys_updated_on" column in ServiceNow. If a table does not have "sys_updated_on" column you should have an index on "sys_created_on" column instead.

You can use validate button to verify if you need to create the index. To create an index open ServiceNow and go to System Definition → Tables page. Select your table and open Database Indexes tab. You can create the index there.

1.3.5. Activity Log

Each synchronization run has its own activity log which includes information about all activities executed during this synchronization. The activity log helps you to understand what SnowMirror does and in case of trouble provide you enough information to fix the problem. You can find the logs on synchronization detail page on "History" tab.

The logs meta-data are stored in the configuration database and the actual log data are stored in files. Default log directory is logs/activityLogs. The files are located by this pattern: <synchronizationId>/<yyyy-MM>/<logId>.log. If you find the activityLogs directory too large you can delete files or whole directories to save some disk space.

1.4. Display Values Synchronization

In ServiceNow, there is concept of display values and actual values. The actual value represents the raw value in the database, whereas the display value represents the user-understandable value that is usually shown in the UI. For some fields, there are significant differences in how these values display in the UI vs. in the database. For example:

  • Encrypted text: The database value is encrypted, while the displayed value is unencrypted based on the user’s encryption context.

  • Reference fields: The database value is sys_id, but the display value is a display field of a referenced record.

  • Date fields: The database value is in UTC format, while the display value is based on the user’s time zone.

  • Choice fields: The database value may be a number, but the display value will be more descriptive.

1.4.1. Algorithm

Let’s focus on the reference fields. How does ServiceNow evaluate which field to use as a display value? It is not so straightforward as it might seem. Imagine a table hierarchy Security Incident → Incident → Task. It is possible to specify display values in each table or just in one of them. It is possible to omit the display values at all and rely on the defaults and finally, have you heard of Dictionary Overrides?

Here is the actual evaluation algorithm ServiceNow uses:

  • Dictionary Override – Current Table – The highest priority in the evaluation algorithm has a flag called Override Display Value in the Dictionary Override (i.e. table called sys_dictionary_override). It is possible to set a different display value then the parent table is specifying. E.g. the Task table has the field number as a display value. In the child table Security Incident it is possible to set up an override configuring short_description as a display value.

  • Dictionary Override – Parent Table – same as previous. If there is no override in the referenced table then the algorithm searches for overrides in the table hierarchy from the child to the parent. E.g. override in the Incident table takes precedence over a display value field specified in the Security Incident table but not over an override in the Security Incident table.

  • Display value field – Current Table – if there are no overrides then the standard display value fields come into play. The display value on the child table overrides the display value on the parent. E.g. Security Incident can specify its own field u_security_code as a display value for security incidents which would override the number field in Task.

  • Display value field – Parent Table – same as previous. If there is no display value field in the referenced table then the algorithm searches for display fields in the table hierarchy from the child to the parent.

  • Default fields – even if there are no display values explicitly set up, ServiceNow searches for default fields to make them as display value fields. These fields are “name“, “u_name“, * “sys_name“, “number“, “u_number“. The algorithm first searches the whole table hierarchy for a field called name. Then, if not found, it searches for u_name, etc. Please note that the child-parent mechanism does not work here. Name has always precedence over Number.

  • SysId – if everything fails then ServiceNow uses sys_id as a display value.

Note that the display value field can be another reference field or can be a choice list field. Transitive relation applies here so ServiceNow is traversing through the display values to find the last in the row.

1.4.2. Synchronization Strategies

SnowMirror lets you download either sys_ids of referenced records or their display values.

  • Only sys_ids - SnowMirror stores sys_ids of the referenced records to the mirror database

  • Only Display Values - SnowMirror stores display values of referenced records to the mirror database

  • Both sys_ids and Display Values - SnowMirror stores both sys_ids and display values of referenced records to the mirror database. Display values are store to a column with prefix dv_

Note that there is a limitation imposed on display value synchronization. If display value changes and the record which references the display value does not change SnowMirror cannot discover the change. The display value is only updated when the referencing record changes too.

1.5. Views Synchronization

Migration process described in previous chapters works perfectly when synchronizing tables. Beside tables ServiceNow allows you to define database views. SnowMirror supports views synchronization with some limitations. There are several issues SnowMirror has to tackle:

  • A record in a view is a Cartesian product of records from several tables. That means there is no unique and permanent SysId identifying the record.

  • A record in a view does not have to contain "sys_created_on" and "sys_updated_on" columns.

  • There is no delete log for views in ServiceNow.

SnowMirror provides two options to synchronize a view:

  • Full Load - full load synchronization deletes data from mirror database and then downloads all data from ServiceNow. This way you are guaranteed to have the most recent and 100 percent accurate copy of your data in mirror database. Depending on the amount of your data the synchronization may take a long time to perform.

  • Incremental Load - incremental load synchronization downloads only increments, i.e. new created, updated and deleted data. To use this synchronization type you have to choose a master table. SnowMirror uses this table to find out the increments. This means if records in other tables are changed SnowMirror would not find out. Also the master table should not contain any duplicate SysId values.
    Incremental Load View synchronization requires less amount of time to perform at the cost of not having 100 percent accurate copy of your data. To tackle this issue we recommend to setup Full Load Scheduler to run once in a while because it clears data from mirror database and downloads all data from ServiceNow.

1.6. Attachments synchronization

Attachments synchronization process is very similar to table synchronization process. This section lists some differences between table synchronization and attachment synchronization.

1.6.1. Migration steps

Create directory if not exists

Besides configuration and mirror database, attachment synchronization needs an access to the filesystem so that it can store attachments there. During the synchronization, SnowMirror creates a directory for attachments (if it does not already exist).

Path to the directory is defined by snowMirror.attachment.rootDir property which can be configured in Advanced properties. Be sure to have sufficient rights for creating directories and files in this directory.

Replication Using SOAP API
  1. SnowMirror loads serviceNow.number.of.keys keys (by default, this is 10,000) which were created or updated since the last synchronization run. The keys are loaded from sys_attachment table.

  2. Attachment meta-data are loaded for these keys. Meta-data are loaded in batches by serviceNow.number.of.records records (by default this is 250).

  3. Based on the meta-data, SnowMirror start to download the attachments from sys_attachment_doc table. SnowMirror repeatedly requests data from this table because it returns the attachments divided into small chunks. SnowMirror downloads serviceNow.number.of.attachment.chunks chunks (by default this is 4,000) per one request. Then it writes the chunks into temporary files and joins them to make up the whole attachment file.

  4. After downloading attachments their meta-data are stored to Mirror table and temporary files are moved to snowMirror.attachment.rootDir. If there already exists file with same name, new file is renamed (suffix is added to filename).

Replication Using REST API
  1. SnowMirror loads attachment meta data using REST Attachment API. It loads the meta-data in batches of serviceNow.number.of.records records (by default, this is 1,000). It always load records which were created or updated since the last synchronization run.

  2. Based on the meta-data, SnowMirror downloads the attachments one by one using REST Attachment API.

  3. After downloading attachments their meta-data are stored to Mirror table and temporary files are moved to snowMirror.attachment.rootDir. If there already exists file with same name, new file is renamed (suffix is added to filename).

Delete strategies
Delete Strategies for Attachments

Audit

Audit deleting strategy is not available when synchronizing attachments

Diff

Diff compares sys_ids in ServiceNow table with the sys_ids in mirror DB. Sys_ids that are present in mirror table, but not ServiceNow table are deleted (with their corresponding attachment file). sys_updated_on and sys_created_on are intact.

Fast Difference

It counts records and downloads max. value of sys_updated_on column. If the values don’t match with the mirror database it splits the SysIds into smaller chunks and repeats until it gets until to 10,000 keys. Only then it download the keys and compares them with the mirror database. This strategy is useful when you have a large table and small amount of deletes because it reduces the number of requests to ServiceNow.

None

Does not delete any data from the mirror table or attachment directory. Insert and update tasks act as normal. Deleting data can be done manually by Clean and Synchronize button on detail of the synchronization (See Synchronization Detail ).

Truncate (Deprecated)

Truncate deleting strategy deletes all data from mirror table and all attachments from local directory before the insert task. Update task is skipped. Note, that sys_updated_on and sys_created_on columns are set to time of last synchronization start. It can not be used to determine the time of actual row insert or update in ServiceNow. It is deprecated - use Full Load Scheduler instead.

1.6.2. Attachments Directory

Content of snowMirror.attachment.rootDir has following structure:
<snowMirror.attachment.rootDir>/<synchronization name>/<display value> where:

  • snowMirror.attachment.rootDir - configurable directory where attachments are stored

  • synchronization name is a name of a synchronization that you filled in Synchronization settings form

  • display value is a value of a field which is marked as display value for the given table

1.6.3. Encoded Query

If you have entered Encoded query it is applied on related table (e.g. incident) and NOT on sys_attachment table.
While getting keys of created (or updated) attachments since the last synchronization another additional request is made on related table (i.e. incident, task, …​) so that SnowMirror can learn which records fit encoded query. This assures that only attachments for records which fit encoded query are downloaded.

1.7. Index Replication

SnowMirror allows to replicate not only tables and data but also database indexes. If you define an index in your ServiceNow instance SnowMirror will pick up the change and create the index in your mirror database. That is particularly useful if you make complicated SQL queries to your mirror database and need to improve the performance of these queries.

You can enable index replication on General Settings page.

1.7.1. Custom Indexes

What if you need an index in the mirror database but don’t want to create it in your ServerNow? Don’t worry, we have it covered. You can create the index manually and we will preserve it intact.

1.7.2. SysId Columns

ServiceNow defines primary keys on sys_id columns. SnowMirror ignores these keys because it always creates it’s own primary key on "sys_id" column.

1.7.3. Performance Implications

If have you enabled Auto Schema Update or set up Notifications on schema change SnowMirror will send one additional request per synchronization. In the case of views, it will send several requests depending on how many tables the view consists of.

The request itself is not resource expensive and time-consuming. Yet, it should be taken into consideration especially when SnowMirror is configured to run many synchronizations very often.

1.7.4. Limitations

In some cases, ServiceNow Web services return us encoded column names. That happens when a column name is too long. It also happens on few columns on "task" table and its descendants. We cannot then couple the column in mirror database with a column of the ServiceNow index. Therefore, we ignore those indexes.

1.8. Synchronization Types

This chapter describes all types of synchronizations you can encounter in SnowMirror.

1.8.1. Incremental Synchronization

SnowMirror requests and download only records inserted, updated and deleted since the last synchronization run. It is suitable to run on daily basis.

1.8.2. Clean and Synchronize

SnowMirror cleans all data in mirrored table and downloads all records from scratch.

After the clean it may take a long time to synchronize the data again. The old data won’t be available during the synchronization. To tackle the issue you may want to enable "snowMirror.replication.cleanAndSyncToTmpTable" which synchronizes the data to temporary table. Only after the synchronization finishes successfully the original mirror table is removed and replaced by the temporary table.

1.8.3. Differential Synchronization

Inserts all missing and deletes all redundant records. It does not update any records.

The algorithm works as follows:

  1. It requests the first 10,000 SysIds from ServiceNow ordered alphabetically

  2. It loads the first 10,000 SysIds from the mirror database ordered alphabetically

  3. It computes a difference to find out new and deleted SysIds. If it cannot decide if the records is new or deleted it will process it in the next batch.

  4. It repeats the first three steps until there is no records left to process.

This is particularly useful if you need to repair your mirror table. See Consistency Check chapter for examples.

1.8.4. Synchronize From

SnowMirror downloads only records whose sys_updated_on column contains a value greater than or equal to the date you chose.

This is useful if your synchronization missed or ignored some records, you have fixed the root cause of the issue and now you want to download the missing records.

1.8.5. Resume

Let’s say you have a long running synchronization and half way through the synchronization process it fails (for whatever reason). It’s inconvenient to run it again because it takes some time to do so. "Resume" functionality is here to rescue.

It allows you to resume a failed synchronization. If you resume a synchronization SnowMirror will start to download data from the point of the failure.

SnowMirror can resume table and view synchronizations. Backups and attachments are currently not supported.

1.9. Consistency Check

In some cases SnowMirror may miss synchronizing some records.

Imagine that you imported data to your table in ServiceNow and all records had sys_updated_on set on some old date. SnowMirror already synchronized newer data hence it will miss the newly imported records. This is when Consistency check comes in handy. After every synchronization run it counts the number of records in ServiceNow and number of records in mirrored table. Then it compares the counts and if it finds out that they are not equal it will put the synchronization into Warning state and send notification email (if configured to do so).

consistency check list

When you encounter such a warning you should start looking for a reason of the inconsistency. To tackle the issue with old dates in imported records you can use differential synchronization. To diagnose any other potential issues you may want to use validate button.

consistency check activity log

See Admin Guide on how to configure consistency check.

1.10. Mirror Database Constraints And Indexes

SnowMirror creates a primary key on "sys_id" column in every mirror table. Thanks to the key, it can update records really fast.

Older SnowMirror installations used an unique key instead of a primary key. Those installations keep using unique key, even after upgrade. New (clean) installations use a primary key.

In case of Full Load Views, the records consist of several joined tables. Therefore, there is no single sys_id and we cannot create a primary key.

You can even configure SnowMirror to use unique indexes instead of primary keys. Go to Settings → Advanced Settings page and set a property "snowMirror.replication.index.sysIdConstraintType" to "UNIQUE_INDEX".

2. Backups

SnowMirror allows you to copy and archive data from ServiceNow so that it may be restored in case of data loss. It support backups of tables and attachments.

Data from tables can be stored to CSV or XML files. Those are valid files recognized by ServiceNow. You can use them to import your data back to ServiceNow.

2.1. Backup Strategies

SnowMirror supports three backup strategies. You can run them manually or automatically by a scheduler.

  1. Incremental - downloads changes made since the last backup (no matter if it was full or incremental).

  2. Differential - downloads changes made since the last full backup.

  3. Full - downloads all data.

2.2. Location

All backups are stored to <SnowMirror installation folder>/snow-mirror/data/backups directory. It can be changed on General Settings page using "Backups" field. Please make sure to have sufficient permissions for creating directories and files in this directory.

change directory

Each backup synchronization creates its own directory with the name of the synchronization.

Sample Backup Structure:
<SnowMirrorDirectory>/data/backups
├───incident
│       2581_FULL_2016-43-08_07h-43m-50s.zip
│       2582_INCREMENTAL_2016-44-08_07h-44m-35s.zip
│       2583_INCREMENTAL_2016-46-08_07h-46m-35s.zip
│
└───sys_user
        2581_FULL_2016-43-08_07h-43m-50s.zip
        2582_INCREMENTAL_2016-44-08_07h-44m-35s.zip
        2583_INCREMENTAL_2016-46-08_07h-46m-35s.zip

2.3. Retention Period

Every day SnowMirror runs a job which removes old backups. Each synchronization defines a "retention period". The job will remove backups older that the retention period.

2.4. Table Backup

The backup synchronization works the same as table synchronization works except a few exceptions:

  • Backups do not use the mirror database at all. They are stored exclusively on filesystem in files.

  • SnowMirror stores durations and timers to mirror database as either a number of milliseconds or as a date and time. Backups do not use these settings so that the fields can be imported back to ServiceNow.
    XML files store timers and durations as a date and time (timestamp). CSV files store them as a number of seconds.

  • SnowMirror allows a user to store dates and times in whatever timezone she wishes (default being UTC). Backups can only store dates and times in UTC so that the values can be imported back to ServiceNow.

  • SnowMirror can’t synchronize columns which are not marked as "active" in the Dictionary. It also can’t synchronize data from child tables - e.g. when exporting "task" (parent) table you won’t see any "incident" (child) records in the export. If you want to backup child tables you have to create a new synchronization for each child.

2.4.1. File Formats

You can choose from the following formats:

  • CSV - records are store in comma separated file format. ServiceNow can run all necessary business rules during the import.

  • XML - records are stored in XML file. Unlike CSV it can even store deleted records. ServiceNow won’t run any business rules during the import!

The CSV or XML files are always compressed into a ZIP file which highly reduces the size of the backup.

The file has the following format: [ActivityLogId_TypeOfTheSynchronization_Date_Time].

  • ActivityLogId - ID of the activity log created during the backup

  • TypeOfTheSynchronization - FULL or INCREMENTAL. See Synchronization Types chapter for details.

  • Date_Time - date and time when the synchronization started

Example:
2581_FULL_2016-43-08_07h-43m-50s.zip

2.4.2. Create and Edit

For detailed description of how to create a new synchronization please New Synchronization Wizard. Also, see Retention Period (Backups Only) and File Format (Backups Only) sections describing backup specific fields.

Here, we will just discuss several specifics related to backups.

  • CSV

    • Deleting Strategy - you cannot choose any deleting strategy, because CSV file can’t store deleted records.

    • Reference Fields - defaults to Only Display Values so that it behaves like ServiceNow does. But you can choose whatever option you want to.

    • Columns - you can choose columns you want to synchronize.

  • XML

    • Deleting Strategy - if you choose Audit deleting strategy and your table is audited then all deleted records will be stored in the XML file.

    • Reference Fields - you won’t have to choose any Reference Fields strategy. XML backups have to always download both value and its display value.

    • Columns - all columns are synchronized to stay compatible with ServiceNow XML files.

new sync

2.4.3. Import To ServiceNow

Backup files are valid CSV or XML files used by ServiceNow. See ServiceNow documentation for detail how to import those files.

Please take a time to study how import of CSV and XML files works and know the implications.
Importing CSV files
  1. Load Data into a new table. Open System Import Sets → Load Data page import data into a new table. Let ServiceNow create a new table with all columns for you. All columns will be of a "string" type.

    load data
  2. Create a transform map

    Open System Import Sets → Create Transform Map page and create a new tranform map. Choose the new table created in the previous step as a source table and the table where you want to import the data as target table.

    create transform map

    After submitting the form you have to map fields from the source table to fields in the target table. We recommend using "Auto Map Matching Fields" or "Mapping Assist" for that purpose. Mapping is thoroughly described in ServiceNow documentation. If you have never used transform maps we recommend you to read ServiceNow documentation. In this tutorial we pinpoint just a few important steps:

    • Coalesce Sys ID column - configure Sys ID column in a target table to coalesce. This makes ServiceNow to match source values with values from existing records. Simply said this will allow you to update existing records with the records in your import set. See ServiceNow documentation for detailed description. Note that you will see two Sys ID columns it the source table: "Sys ID" and "sys_id". Please choose "sys_id" column - that is the one from you import set.