1. Getting Started

SnowMirror

1.1. Introduction

SnowMirror is data replication tool for Salesforce. The goal of SnowMirror is to replicate data from Salesforce into a local database. Data is transferred using Bulk API and saved into a local database within user’s ICT infrastructure. SnowMirror enables to manage synchronized data easily and use them for other processing such as computing summary statistics, creating data analysis etc. The data is accessible anytime without frequent requests to Salesforce, which could be time consuming and impacts its performance.

SnowMirror provides user a friendly graphic interface. It is easy to set or edit which Salesforce tables and columns will be synchronized, when and how often. A user can create, view and edit synchronization settings, runs, view synchronization details, history, synchronized data or synchronization logs directly in SnowMirror.

1.2. Architecture

SnowMirror loads data from Salesforce using SOAP API and Bulk API and saves it into mirror DB. This process is called synchronization. Configuration of synchronization is stored in configuration DB. Synchronization can load data from one Salesforce table and save it to one mirror DB table. Multiple synchronizations can load data from same Salesforce table, but each synchronization has its own mirror DB table. For more information about synchronizations see Migration Process and New Synchronization.

snowMirrorBasicScheme

Multiple SnowMirror instances with separate databases can be connected to single Salesforce organization.

Synchronizations are managed via embedded web server that also provides comprehensive information about synchronization health and performance. For more information see Dashboard.

1.2.1. Database

Both configuration and mirror database can be provided by wide range of DBMS like MySQL, Postgres, SQL Server, Oracle and H2. If you are creating databases manually, use UTF-8 encoding.

Configuration database

Configuration database contains data about users and roles, synchronization logs and of course about synchronization tasks.

Mirror database

Mirrored data from Salesforce are stored in mirror database. A structure of tables and columns is created as a copy of tables and columns from Salesforce. You should never manipulate database structure from any other place than SnowMirror GUI. All of the data is just mirrored Salesforce tables (or parts of tables) without any transformations. The only additional table columns are mirror_created_on and mirror_updated_on which contains date and time of row creation and last update. The "mirror_" prefix is configurable.

It is necessary to set both databases to use UTC timezone as a default timezone for all date - related records. This will help you to reduce possibility of date inconsistency because Salesforce is using UTC too.

1.3. Migration Process

This chapter covers tasks performed by SnowMirror during a synchronization itself.

SnowMirror loads data from Salesforce table using SOAP and Bulk APIs. If first finds out Ids of the records it wants to replicate and then download them using Bulk API. Salesforce prepares the data for download in CSV format. The maximum size of CSV file is 1 GB. If this size is exceeded, more CSV files are downloaded. Then we split the records into batches. Every single batch is inserted into database in a separate transaction. Batch sizes are configurable in Advanced settings of SnowMirror (property snowMirror.number.of.records).

To ensure correct synchronization process, please make sure you have correct date and time set on your computer.

1.3.1. Transactions

Data batches are transactional but whole synchronization runs are not. This prevents application from reverting data in case of an error after several minutes of synchronization run.

1.3.2. Migration Tasks

Synchronization lifecycle consists of two essential steps. First, new and updated records are synchronized. Then deleted records are synchronized. These essential steps are extended by several additional steps.

migration
Initializing Migration

First task in migration process loads all data from configuration database, which are needed to run the synchronization.

Count Records

To figure out the approximate volume of data that will be synchronized, SnowMirror sends a web service request with SoQL query. This data is useful while rendering progressbars and is stored in activitylog table in configuration database as totalToInsert, totalToDelete and totalToUpdate columns.
Requests for inserted, updated and deleted records has to be done in three separate web service requests.

Create Directory if not exists (only attachment synchronization)

If directory for saving attachments does not exists, it is created in this stage.

Create or Alter Table

If a table does not exist in mirror DB, it is created in this stage.

If synchronization configuration has changed since the last synchronization run, the table is altered.

Insert, Update

This task load Ids of records created and/or updated since the last synchronization. Then it creates a job in Salesforce and download the data in batches.

migration detailed
Delete

Delete task is different than insert and update tasks. Most Salesforce tables stores deleted data in delete log table (i.e. table is audited). Preferred way to determine which rows were deleted is to read them from the delete log. However not every table is audited. That means that deleted rows are not stored in the delete log.

For this case SnowMirror offers four deleting strategies.

Delete Strategies

Audit

Audit deleting strategy gets deleted records via the delete log table. This way is the most effective because we only get created or changed data after last synchronization run.

Truncate

Truncate deleting strategy deletes all data from mirror table before the insert task. Update task is skipped. Note, that LastModifiedDate and CreatedDate columns are set to time of last synchronization start. It can not be used to determine the time of actual row insert or update in Salesforce.

Difference

Difference compares Ids in Salesforce table with the Ids in mirror DB. Ids that are present in mirror table, but not Salesforce table are deleted. LastModifiedDate and CreatedDate are left intact.

None

Does not delete any data from the mirror table. Insert and update tasks act as normal. Deleting data can be done manually by Clean and Synchronize button on detail of the synchronization (See Synchronization Detail ).

If you choose truncate strategy as your delete strategy, there are some changes in migration process as you can see on the picture below.

synchronization process truncate
Default deleting strategy is Audit for tables that contains CreatedDate and LastModifiedDate. For history tables, default deleting strategy is None. For any other table that does not contain CreatedDate or LastModifiedDate and is not History table, default deleting strategy is Truncate.
Finalize Migration

Finalizing task updates the activity logs and finishes the synchronization.

1.3.3. Cache Management

The application uses cache for Salesforce metadata about tables and columns, parent-child structure etc. Because of this cache some changes in Salesforce can be taken into account with a delay.

If you want to clear the cache manually go to Settings menu, choose Cache management and click on Clear cache button.

Cache timeout is configurable in the configuration file.

1.3.4. Activity Log

Each synchronization run has its own activity log which includes information about all activities executed during this synchronization. They should help you understand what SnowMirror does and in case of trouble provide you enough information to fix the problem. You can find the logs on synchronization detail page, card history.

The logs meta-data are stored in the database and the actual log data are stored in files. Default log directory is logs/activityLogs. The files are located by this pattern: <synchronizationId>/<yyyy-MM>/<logId>.log. If you find the activityLogs directory too large you can delete files or whole directories to save some disk space.

1.4. Synchronization Types

In this chapter we will describe all types of synchronizations you can encounter in SnowMirror.

1.4.1. Incremental Synchronization

SnowMirror requests only inserted, updated and deleted records. It is suitable to run on daily basis.

1.4.2. Clean and Synchronize

SnowMirror cleans all data in mirrored table and downloads all records new.

After the clean it may take a long time to synchronize the data again. During the synchronization the old data won’t be available. To tackle the issue you may want to enable "snowMirror.replication.cleanAndSyncToTmpTable" which synchronizes the data to temporary table. Only after synchronization finishes successfully the original mirror table is removed and replaced with the temorary table.

1.4.3. Differential Synchronization

Inserts all missing and deletes all redundant records. It does not update any records.

This algorithm works as follows:
1. It requests the first Ids from Salesforce ordered alphabetically
2. It loads Ids from the mirror database ordered alphabetically
3. It computes a difference to find out new and deleted Ids. If it cannot decide if the records is new or deleted it will process it in the next batch.
4. It repeats the first three steps until there is no records left to process.

This is particularly useful if you need to repair your mirror table. See Consistency Check chapter for examples.

1.4.4. Synchronize From

SnowMirror downloads only records whose SystemModstamp column contains a value greater than or equal to the date you chose.

This is useful if your synchronization missed or ignored some records, you have fixed the root cause of the issue and now you want to download the missing records.

1.5. Consistency Check

In some cases SnowMirror may miss synchronizing some records.

Imagine that you imported data to your table in Salesforce and all records had SystemModstamp set on some old date. SnowMirror already synchronized newer data hence it will miss the newly imported records. This is when Consistency check comes in handy. After every synchronization run it counts the number of records in Salesforce and number of records in mirrored table. Then it compares the counts and if it finds out that they are not equal it will put the synchronization into Warning state and send notification email (if configured to do so).

consistency check list

When you encounter such a warning you should start looking for a reason of the inconsistency. To tackle the issue with old dates in imported records you can use differential synchronization. To diagnose any other potential issues you may want to use validate button.

consistency check activity log

See Admin Guide on how to configure consistency check.

1.6. Mirror Database Constraints And Indexes

SnowMirror creates unique index on "Id" column in every mirror table. Because of that, it can update a single record really fast.

You can even configure SnowMirror to create primary keys instead of unique keys. Go to Settings → Advanced Settings page and set a property "snowMirror.replication.index.idConstraintType" to "PRIMARY_KEY".

2. Backups

SnowMirror allows you to copy and archive data from Salesforce so that it may be restored in case of data loss. It supports backups of tables and attachments.

Data from tables can be stored to CSV files. This file can be imported to Salesforce using Data Loader. You can use them to import your data back to Salesforce.

2.1. Backup Strategies

SnowMirror supports three backup strategies. You can run them manually or automatically by a scheduler.

  1. Incremental - downloads changes made since the last backup (no matter if it was full or incremental).

  2. Differential - downloads changes made since the last full backup.

  3. Full - downloads all data.

2.2. Location

All backups are stored to <SnowMirror installation folder>/snow-mirror/data/backups directory. It can be changed on General Settings page using "Backups" field. Please make sure to have sufficient permissions for creating directories and files in this directory.

change directory

Each backup synchronization creates its own directory with the name of the synchronization.

Sample Backup Structure:
<SnowMirrorDirectory>/data/backups
├───Account
│       2581_FULL_2016-43-08_07h-43m-50s.zip
│       2582_INCREMENTAL_2016-44-08_07h-44m-35s.zip
│       2583_INCREMENTAL_2016-46-08_07h-46m-35s.zip
│
└───Contact
        2581_FULL_2016-43-08_07h-43m-50s.zip
        2582_INCREMENTAL_2016-44-08_07h-44m-35s.zip
        2583_INCREMENTAL_2016-46-08_07h-46m-35s.zip

2.3. Retention Period

Every day SnowMirror runs a job which removes old backups. Each synchronization defines a "retention period". The job will remove backups older that the retention period.

2.4. Table Backup

The backup synchronization works the same as table synchronization works except a few exceptions:

  • Backups do not use the mirror database at all. They are stored exclusively on filesystem in files.

  • SnowMirror allows a user to store dates and times in whatever timezone she wishes (default being UTC). Backups can only store dates and times in UTC so that the values can be imported back to Salesforce.

2.4.1. CSV File Formats

Data is stored into a file in CSV (comma separated values) format. The CSV file is always compressed into a ZIP file which highly reduces the size of the backup.

The file has the following format: [ActivityLogId_TypeOfTheSynchronization_Date_Time].

  • ActivityLogId - ID of the activity log created during the backup

  • TypeOfTheSynchronization - FULL or INCREMENTAL. See Synchronization Types chapter for details.

  • Date_Time - date and time when the synchronization started

Example:
2581_FULL_2016-43-08_07h-43m-50s.zip

2.4.2. Create and Edit

For detailed description of how to create a new synchronization please New Synchronization Wizard. Also, see Retention Period (Backups Only) section describing backup specific fields.

Here, we will just discuss several specifics related to backups.

  • Deleting Strategy - you cannot choose any deleting strategy, because CSV file can’t store deleted records.

  • Columns - you can choose columns you want to synchronize.

new sync

2.4.3. Import To Salesforce

You can use Salesforce Data Loader to import the data back to Salesforce.

You have to locate your backup ZIP file, unpack it and import the contained CSV file using Data Loader. See Data Loader documentation for details how to import CSV files.

2.5. Attachments Backup

Attachments are stored into a directory structure which is packed into a single ZIP file. See the sample structure below.

Sample Backup Structure:
<SnowMirrorDirectory>/data/backups/
├───Account
│       2581_FULL_2016-43-08_07h-43m-50s.zip
│      └───INC0000001
│            └───FirstAttachmentOfInc0000001.png
│            └───SecondAttachmentOfInc0000001.png
│       2582_INCREMENTAL_2016-44-08_07h-44m-35s.zip
│      └───INC0000001
│            └───ThirdAttachmentOfInc0000001.png

It is worth mentioning that such a directory structure cannot store deleted files. If you delete an attachment in Salesforce it won’t be tracked in ZIP file. Therefore, you should perform Full backup from time to time.

2.5.1. Create and Edit

For detailed description of how to create a new synchronization please New Synchronization Wizard. Also, see Retention Period (Backups Only) section.

3. Installation

For quick installation see Installation section in Admin Guide.

4. Licensing

You need to enter a valid license key for SnowMirror to be able to synchronize data.

There are three types of licenses: Lite, Enterprise and Trial. Enterprise and Trial editions contain all features and support all databases. Lite edition contains limited number of features. See comparison on our website.

If you don’t enter a valid license SnowMirror will show a warning at the top of each page. To retrieve a valid key visit our website.

no license entered

If your license is about to expire this message will appear in the page header:

license expiring

5. Logging in/out

Every user has a unique username for logging in and one or more user roles, which gives him permission to access application functionality. The name of logged in user is shown on the right side of top-horizontal menu. If you want to log out, use the Logout link, which is placed on the right side of top menu next to username of currently logged in user. User roles are added or removed only by Super Administrator. For more information about roles and permissions see User Roles).

5.1. Permanent login

On login page you can choose to stay logged in indefinitely by checking Remember Me checkbox. This option creates a cookie in your browser containing access token. The cookie is removed only after clicking "Logout". Otherwise you stay logged in indefinitely.

6. User Roles

Users of SnowMirror has to have at least one of the following roles.

Synchronization Reader

Synchronization Reader can view synchronizations, its data and trigger a synchronization.

  • read synchronization details

  • read synchronization logs

  • read migrated data

  • manually trigger synchronizations

  • stop currently running synchronizations

  • change his or hers own time zone

Synchronization Administrator

Synchronization Administrator has permission to read and manage synchronizations. It includes all permissions of 'Synchronization reader' plus:

  • create, edit and delete synchronizations

  • activate and deactivate synchronizations

  • create, edit and delete email notifications

Super Administrator

Super Administrator has beside all permissions of 'Synchronization administrator' access to user management, which includes:

  • create, edit and delete users

  • edit permissions of any user

  • change password of any user

  • activate and deactivate any user

API Caller

API Caller has permission to call REST API

  • authenticate against REST API and call operations on its resources

7. Dashboard

Dashboard is the main page of SnowMirror and it is divided into three parts - list of failed synchronizations, graph of migrated data volume and last / upcoming synchronizations.

7.1. Failed Synchronizations

'Failed Synchronizations' shows synchronizations that failed during the last run. Failed synchronizations shows name of synchronization and time of failure. By clicking on the synchronization name you can obtain more information about the synchronization see Synchronization Detail.

7.2. Graph of Migrated Records

Graph of migrated records shows global statistic of all transferred data per day. It counts a sum of all inserted/updated/deleted records from all synchronizations executed during a particular day. Every each color represents a kind of data operation similar to progressbar.

Much more information is available at the the detail of synchronization. It is possible to use a link on every synchronization name in the table to go to synchronization detail.

migrated records graph

7.3. Last / Upcoming Synchronizations

Upcoming synchronizations shows all synchronizations which are currently running or will be executed in next 12 hours by default. Last synchronizations shows all synchronization that were triggered in past 12 hours. This interval can be set in the property file.
It includes basic details like time of synchronization execution, result of the last execution and progressbar. Contents are sortable.

Last / Upcoming Synchronizations Table:

Name

Name of the synchronization. It is chosen by a user and it does not have to match the table name in mirror database or the name of the table in Salesforce.

Next Run

Next Run is a datum of next scheduled run in format mm/dd/yyyy hh:mm. Synchronization with manual execution scheduler has this field empty.

Duration

Is the time from start to finish of the last synchronization run.

Synchronization Progress

Synchronization progress bar shows ratio of migrated records. It is continuously updated and shows actual progress of the running synchronization. Progress bar has three colors, each of them represents a kind of data operation:

green icon green for data, which were inserted into mirror DB
blue icon blue for updated data
grey icon grey for records deleted from mirror DB

8. Synchronization Overview

Synchronization Overview includes a list of all created synchronizations. Table includes the preview of basic information and is sortable.

sync overview table
Synchronization Owerview Table

Result

Result icon shows icon, which represents result of the last synchronization. There are following possibilities:

ok icon success - data in mirror DB table was successfully synchronized with data from Salesforce
warning icon warning - data in mirror DB table was synchronized but something SnowMirror detected potential issues. The synchronization requires your attention.
error icon fail - data could not be synchronized or an error occurred during synchronization
running icon running - synchronization task is currently running
canceling icon canceling - synchronization task is going to be canceled
error icon unknown result - the synchronization was created but it was not executed yet
waiting icon waiting - the synchronization is waiting in the queue to be executed
disabled icon disabled - the synchronization is deactivated and can’t be executed until you reactivate it.
warning disabled icon disabled with warning - the last synchronization’s run ended up with a warning. The synchronization is deactivated and can’t be executed until you reactivate it.
error disabled icon disabled with error - the last synchronization’s run ended up with error. The synchronization is deactivated and can’t be executed until you reactivate it.

Name

Name of synchronization. It is chosen by user and it may not match with mirror DB table name or name of table from Salesforce.

Next Run

Next Run is a datum of next scheduled run in format mm/dd/yyyy hh:mm. Synchronization with manual execution scheduler has this field empty.

Last Run

Last Run is a datum of last synchronization execution no matter the result. The format of a datum is mm/dd/yyyy hh:mm.

Duration

Is the time from start to finish of the last synchronization run.

Progress

Synchronization progress bar shows ratio of migrated records. It is continuously updated and shows actual progress of the running synchronization. Progress bar has three colors, each of them represents a kind of data operation:

green icon green for data, which were inserted into mirror DB
blue icon blue for updated data
grey icon grey for records deleted from mirror DB

If the last synchronization had not migrated any rows, progressbar background is white.

8.1. Synchronization Overview Controls

  1. New Synchronization
    Redirects to new synchronization wizard. See New Synchronization Wizard.

  2. Differential Synchronization
    Inserts all missing and deletes all redundant records. It does not update any records.

  3. Synchronize
    Triggers synchronization of checked synchronizations.

  4. Clean and Synchronize
    Deletes all mirror table data and immediately synchronizes selected synchronizations. Neither SystemModstamp or CreatedDate are used. That means all data are loaded from Salesforce table.

  5. Edit Scheduler
    Redirects to edit scheduler page. Can be used to schedule multiple synchronizations (selected by checkbox).

  6. Activate
    Enables deactivated synchronizations. After activation synchronizations will be able to synchronize data from Salesforce.

  7. Deactivate
    Temporarily prevents selected synchronizations from triggering. If you disable synchronizations you will not be able to manually trigger them.

  8. Delete
    Permanently deletes selected synchronizations and all its data from mirror database.

  9. Stop
    Stops selected synchronizations if they are running.

  10. Export
    Exports definitions of selected synchronizations. See Administrator Guide for details on Import/Export.

8.2. Filtering

Synchronizations can be filtered by all columns available. To open the filtering form click the funnel icon. After filling in filtering conditions click the Run button and SnowMirror will show only synchronizations matching the conditions.

Also note that there is word "All" next to the funnel icon. This is breadcrumbs area. It shows which filter is applied. The word "All" means no filter is applied.

filtering overview filter

8.2.1. Filtering Options

Each field has a predefined set of operators which you can use to filter its values. Generally, there are four types of operators:

  1. Text - is, is not, is empty, is not empty, contains, does not contain, starts with, ends with

    Used for filtering text fields such as:

    • Name - name of the synchronization,

    • Encoded query - encoded query of the synchronization

    • Mirror table - name of the table in mirror database

    • Created by - name of who created the synchronization

    • Updated by - name of the last person who updated the synchronization

  2. Numbers - is, is not, greater than, greater than or equal to, less than, less than or equal to, between

    Used for filtering numbers in fields such as:

    • Inserted - number of records inserted during the synchronization

    • Updated - number of records updated during the synchronization

    • Deleted - number of records deleted during the synchronization

    • Total - total number of records processed during the synchronization (inserted + updated + deleted)

  3. Options - you are only allowed to choose one of the predefined values. Used on fields:

    • Synchronization Type - Table, view, attachment or backup

    • Result

      • Successful - finished its execution successfully

      • Warning - finished its execution but a user should check inconsistencies in records count

      • Failed - finished its execution with error

      • New - synchronization has never been executed

      • Scheduled - synchronization is waiting to be executed

      • Running - synchronization is being execuded

      • Finished - synchronization finished its execution (no matter if it succeeded or failed)

    • State - Active or inactive

    • Delete Strategy - Audit, truncate, difference or none

    • Include Inherited Columns - Yes or no

    • Auto Schema Update - Yes or no

    • Synchronization Interval - Daily, weekly, periodically, cron or manually

    • Synchronization Interval (Full) - Daily, weekly, periodically, cron or manually

    • Execution Type - Type of full load scheduler: Clean and synchronize or differential synchronization

  4. Date and time - today, yesterday, tomorrow, Last minute, Last 15 minutes, last 30 minutes, last hour,

    • Duration - how long it took to finish the synchronization

    • Last Run - date and time when the synchronization finished

    • Created On - date and time when the synchronization was created

    • Updated On - date and time when the synchronization was updated

8.2.2. Cancelling Filter

You can cancel the current filter by clicking the "All" breadcrumb.

8.2.3. Breadcrumbs

Breadcrumbs work as follows:

  1. Click on one breadcrumb and only the one filter will be applied.

  2. Click the arrow separator and the following breadcrumb will be removed from filter condition.

filtering breadcrumbs

8.3. Queue

Synchronization queue provides you a list of synchronizations which are currently running or waiting to be executed. Synchronizations are executed from the top of the list to the bottom.

queue sample

To open the queue, click on the "Show Queue" link on Dashboard.

queue link

9. New Synchronization Wizard

To create or add a new synchronization use the New synchronization button in the bottom of synchronization overview (go to: top menu / Synchronizations). This button starts a three steps wizard, which will lead you step by step through complete settings of the new synchronization.

9.1. Table to Synchronize

At first you have to choose if you wish to synchronize table, attachments or create a backup. Then you can choose it from multiselect, which contains names of all available tables in Salesforce. This multiselect is alphabetically ordered and supports quick search with autocomplete. Save your choice to move to step two.

new sync select table
Figure 1.

9.2. Synchronization Settings

Second step of a wizard allows you to set synchronization parameters and records of table which would be synchronized.

9.2.1. Synchronization Name

Name of synchronization is chosen by user and it may not match with mirror DB table name or name of table from Salesforce. It should be any word name or phrase, which characterizes the synchronization (Eg.: Account_nightly, Account-basic info,…​).

9.2.2. Name of the Table in Salesforce

This field contains actual name of Salesforce table chosen in step one. Its editing is disabled.

9.2.3. Table Name in Mirror DB

This is the name of table in your local mirror DB. The value matches the Salesforce table name by default, but there is a possibility to choose different name. The name can be 255 chard long and it must contain only letters without diacritic, numbers, underscores and dashes.

9.2.4. Salesforce Object Query Language

An SoQL query string represents a complex filter on a list of records using one string. In user queries could be used operators like and, or, not, greater than, lower than, etc. For more information about SoQL query look at Salesforce wiki Salesforce reference/SoQL WHERE clause.

Be sure to fill in valid query. Query will not be validated.
After editing SoQL query all synchronized data from corresponding table will be deleted.
When you are creating attachment synchronization please note that SoQL query is applied on target table (e.g. Account) and not on Attachment table.
new sync user query
Figure 2.

9.2.5. Delete Strategy

This parameter specifies, how will be data deleted from mirror database, based on deletes in Salesforce. Following strategies are available:

Audit

Audit deleting strategy gets deleted records via the delete log table. This way is the most effective because we only get created or changed data after last synchronization run.

Truncate

Truncate deleting strategy deletes all data from mirror table before the insert task. Update task is skipped. Note, that SystemModstamp and CreatedDate columns are set to time of last synchronization start. It can not be used to determine the time of actual row insert or update in Salesforce.

Difference

Difference compares Ids in Salesforce table with the Ids in mirror DB. Ids that are present in mirror table, but not Salesforce table are deleted. SystemModstamp and CreatedDate are left intact.

None

Does not delete any data from the mirror table. Insert and update tasks act as normal. Deleting data can be done manually by Clean and Synchronize button on detail of the synchronization (See Synchronization Detail ).

For more information about delete strategies see Delete Migration Task
When synchronizing materialized view, it is only possible to you use Truncate strategy

9.2.6. Auto Schema Update (only table synchronization)

If you don’t want to choose columns to synchronize you can check Auto Schema Update checkbox. If it is enabled then every time the synchronization runs it automatically updates its columns meta-data. It means it will automatically add, update (in case data type or its maximum length has changed) and remove columns. Basically changes made to a table Salesforce are automatically reflected in SnowMirror.

new sync auto schema update
Figure 3.

9.2.7. Columns to Synchronize (only table synchronization)

You don’t need to synchronize whole table, because there is a possibility to select table columns to synchronize separately in two sided multi select. Column 'Id' is selected by default and is mandatory. You can add all other columns with >> button or select columns separately. Only columns on the right side of select will be synchronized.

new sync settings
Figure 4.

9.2.8. Retention Period (Backups Only)

Old backups are periodically deleted. By setting "Retention Period" you define how long they will be kept on the disc.

settings backup retention period
Figure 5.

9.2.9. Count Records (Count Attachments)

At the very bottom of the Synchronization Settings form there is a button called Count Records (for attachments there is Count Attachments). It calls Salesforce to find out how many records are there in the table you are synchronizing. It also takes into account the encoded query. Only if the encoded query is invalid you are notified of the error. Otherwise a message with a number of records pops up.

new sync records count
new sync attachments count

9.2.10. Validate Button

At the very bottom of the Synchronization Settings form there is a button called Validate. It helps you to discover potential issues which may occur during synchronization.

  1. Form validation - validates values provided in the form. Checks if all required values were provided. Checks that synchronization name is unique in SnowMirror and that mirror table with the same name does not already exists.

  2. Replication columns - checks if columns used during synchronization are present in your table. Checks if "SystemModstamp" column exists. If not it checks the presence of "CreatedDate" column.

  3. Count records - count number of records in the Salesforce table

  4. Diff deleting strategy - checks the size of the mirrored table and warns you if the table is too big to be synchronized using Diff deleting strategy.

  5. Table size limit - Lite edition of SnowMirror is limited to 100,000 synchronized records per table. It warns you if you exceed the limit. Not performed in Enterprise edition.

validate button dialog

9.3. Schedule

Every synchronization can be triggered manually anytime, but it has also mandatory schedulers, which define execution plan.

There are two types of schedulers:

  • Incremental Load Scheduler - When it starts for the first time it downloads all available data. Next synchronizations only download increments (new records, updated records and removes deleted records). It should be typically executed more often than Full Load Scheduler. It is also less time consuming.

  • Full Load Scheduler - This scheduler makes sure all data is synchronized regardless its created and updated dates. You can choose between to types of full load scheduler. Full Load Scheduler should run less often (once a week, once a month, …​) and requires more time to execute than incremental load scheduler.

If Incremental Load Scheduler is running and Full Load Scheduler kicks in then the Full Load Scheduler is postponed and executed after the Incremental Load Scheduler finishes. SnowMirror expects that the Full Load Scheduler runs less often (one a week, one a month, …​) hence it has to wait to execute.
On the other hand if Full Load Scheduler is running and Incremental Scheduler kicks in then the Incremental Load won’t be executed - it is not necessary because the Full Load Scheduler downloads all data anyway.

There are five approaches how to schedule a synchronization - daily, weekly, periodically, manually and cron. See details below.

9.3.1. Daily

Choose this option when you want to execute your synchronization every day exactly at the same time.

  • Active Since - The day, when the synchronization will get active.

  • Time - The time, when the synchronization will be triggered every day.

new sync scheduler

9.3.2. Weekly

This option allows you to run synchronization every week at selected days and time.

  • Active Since - The day, when the synchronization will get active.

  • Days of Week - You have to select one or more days, when the synchronization will be triggered.

  • Time - The time, when the synchronization will be triggered every selected day.

9.3.3. Periodically

  • Active Since - The day, when the synchronization will get active.

  • Time - The time, when the synchronization will get active.

  • Interval - Time interval in minutes between synchronization runs. Available values are natural numbers in the range of 1 to 1440 (i.e. one day).

9.3.4. Cron

A cron expression is a string consisting of six or seven fields that describe individual details of the schedule. These fields, separated by white space, can contain any of the allowed values, shown in the expected order in the table below, with various combinations of the special characters described in the cron manual.

Cron expression have to be compatible with format of Quartz Scheduler cron expressions. There are some valid CRON extensions which are not supported by Quartz Scheduler cron expressions. See Official Quartz documentation for details.
Field Allowed Values

Seconds

0-59

Minutes

0-59

Hours

0-23

Day of month

1-31

Month

0-11 or JAN-DEC

Day of week

1-7 or SUN-SAT

Examples:

0 0 12 * * ?

Fire at 12:00 PM (noon) every day.

0 0/5 14 * * ?

Fire every 5 minutes starting at 2:00 PM and ending at 2:55 PM, every day.

0 15 10 ? * MON-FRI

Fire at 10:15 AM every Monday, Tuesday, Wednesday, Thursday and Friday.

Pay attention to the effects of '?' and '*' in the day-of-week and day-of-month fields!

9.3.5. Manually

Last approach of scheduling is the manual execution. There is no automatic plan, you can just run the synchronization whenever it is necessary.

9.4. Scheduler Priority

Each synchronization has a certain priority (the default value being "normal"). If several synchronizations are waiting to be executed then synchronization with a higher priority will execute first.

There are five priorities: Highest, high, normal, low, lowest

The settings can be found on Scheduler page under Advanced Settings section.

settings schedule priority

9.5. Run Immediately

Use this option in case you want to run the new synchronization.

9.6. Finalization

When synchronization configuration is done then synchronization overview is showed. Created synchronization has question mark in the result column because it has not been triggered yet. You can check synchronization settings by clicking on its name.

new sync overview

10. New Bulk Synchronizations

SnowMirror offers a way to create multiple new synchronizations at a time. Pages and forms are very similar to the ones for creating a single synchronization. To create or add new synchronizations use the New synchronization button in the bottom of synchronization overview and then click Bulk Create button.

bulk sync create new

10.1. Tables to Synchronize

On this page you can choose multiple tables for which you want to create a synchronization for.

When creating Bulk synchronization you can create either Attachment, Table or Backup synchronization. You cannot create more types in one wizard.
bulk sync create choose

10.2. Synchronizations Settings

The form is similar to the form for creating a single synchronization but some of the fields are missing. SnowMirror assigns a value to these fields automatically:

  • Synchronization name will match the name of the table from Salesforce (for attachment synchronization Synchronization name will match the name of the table from Salesforce with _attachment suffix)

  • Mirror database table name will match the name of the table from Salesforce (for attachment synchronization Mirror database table name will match the name of the table from Salesforce with _attachment suffix)

  • All columns will be added synchronized by default. For attachment synchronization no columns will be added.

bulk sync create settings

In case a synchronization for a table already exists you will be prompted to resolve the conflict.

  • Ignore - a synchronization for with the duplicate name won’t be created

  • Rename - a new synchronization will be created with a new automatically assigned name

bulk sync create resolve duplicates

10.3. Schedule

The form is exactly the same as the one for creating a single synchronization. See Schedule for details.

10.4. Run Immediately

Use this option in case you want to run the new synchronization.

11. Synchronization Detail

The following specifications are available for all synchronizations on their detail. Detail of a synchronization is accessible by clicking on its name on dashboard or list of synchronizations.

11.1. Controls

Synchronization Detail Table:

Synchronize Now

Manually triggers synchronization. No planned synchronizations are affected.

Clean and synchronize

Delete all mirror table data and immediately synchronizes. Neither SystemModstamp or CreatedDate are used. That means all data are loaded from Salesforce table.
In the activity log will be two rows. One containing data about the deleting data and second about loading new data.

Differential Synchronization

Inserts all missing and deletes all redundant records.

Synchronize From

SnowMirror downloads only records whose SystemModstamp column contains a value greater than or equal to the date you chose.

Deactivate

You can temporarily prevent synchronization from triggering. If you disable a synchronization you will not be able to manually trigger it.

Edit Settings

Edit synchronization name, name of table in mirror DB, delete strategy, encoded query or synchronized columns. See Synchronization Settings.
Settings are not editable during running synchronization.

Edit Mappings

Allows you to explore and edit data types to which the columns will be mapped.

Edit Scheduler

Edit the interval of triggering of the synchronization.
Scheduler is not editable during running synchronization.

Delete

Deletes synchronization. Deletes rows from configuration DB and drops table from mirror DB. All data in the mirror DB table will be lost.

11.2. Basic Information

Basic information about the synchronized table.

sync detal basic info

Basic information about the synchronized attachments.

sync detal basic info attachment
Basic Information Table:

Salesforce Table Name

Name of synchronized table in Salesforce.

Salesforce User Query

SoQL query used to restrict synchronized rows. Useful when you want to synchronize just active Accounts. If empty, no query is applied.

Table Name in Mirror DB

Name of target table in mirror DB.

Local system directory

This is the directory where your synchronized attachments will be saved. Displayed only in attachments synchronization.

Created by

User who created this synchronization.

Created on

Time of synchronization creation.

Updated by

The last user who updated synchronization scheduler or synchronization settings.

Updated On

Time of last synchronization update.

Delete strategy

Strategy to determine which rows has been deleted. For more information see Delete Migration Strategy.

11.3. Migrated Records per Synchronization

Chart shows volume of migrated data in each synchronization.
By default shows 20 latest synchronization runs, but this is configurable in property file.

migrated records per sync graph

11.4. Synchronization Duration

Chart shows time that each synchronization run has taken. Shows same number of synchronization runs as the chart above.
Number of shown columns is configurable by the same property as the Migrated Records per Synchronization.

sync duration

11.5. Schedule

Contains information about Incremental Load scheduling of the synchronization. See Scheduler Settings.

Schedule Table:

Synchronization Interval

Scheduling type. See Scheduler Settings.

Time

Displayed in case of daily or weekly synchronization interval. Shows the time that triggers synchronization on selected days.

Day of Week

Displayed solely in case of weekly synchronization interval. Shows days on which the synchronization is triggered.

Cron

Displayed in case of cron scheduler type. Shows cron expression used to schedule the synchronization. For more info about cron expression see Quartz scheduler.

Next Run

Shows date and time when synchronization will be triggered next time.

sync detail schedule

11.6. Full Load Scheduler

Contains information about Full Load scheduling of the synchronization. Contains the same fields as Incremental Load scheduling. See previous chapter for details.

11.7. Last Run

Shows comprehensive information about last or currently running synchronization.

sync detail last run
Last Run Table:

State

Result and progress of last run.

Synchronization Log

Link to log of last synchronization.

Started by

Shows who triggered last run. In case of manual synchronization shows name of user who started it. If run was triggered automatically by scheduler shows value <SCHEDULER> regardless who set up the scheduler.

Started on

Time of last synchronization start.

Duration

Time that synchronization run has taken.

Inserted Records

Number of new records inserted into mirror DB.

Updated Records

Number of records updated in mirror DB.

Deleted Records

Number of records deleted from mirror DB.

Total Processed Records

Sum of inserted, updated and deleted records.

11.8. Synchronized Columns

Shows columns that are synchronized and their data type and length in Salesforce.

sync detail sync columns

11.9. Activate/Deactivate

You can suspend launching synchronization by clicking to Deactivate synchronization button. After that you can still change synchronization or scheduler settings. After set up you can reactivate the synchronization by clicking on Activate Synchronization button.

11.10. Edit Mappings

Edit Mappings page allows you to explore and edit data types to which the columns will be mapped. For each column the table shows data type from Salesforce and the data type we will use in mirror database.

Note that if you change the data type SnowMirror immediately changes the type in mirror database.

sync detail edit mappings

The rows you have changed are painted in bold. If you click on a name of a column you will be redirected to the page where you can change the data type of that column.

sync detail edit mappings detail

11.10.1. Snowflake

Here is how we map Salesforce data types to data types in Snowflake database.

Salesforce Snowflake Snowflake in SnowMirror <= 2.5.1

id

VARCHAR(18)

VARCHAR

int

NUMBER(38,0)

VARCHAR

long

NUMBER(38,0)

VARCHAR

double

NUMBER(p, s)

VARCHAR

string(n)

VARCHAR(n)

VARCHAR

boolean

BOOLEAN

BOOLEAN

currency

NUMBER(p, s)

NUMBER(32,2)

percent

NUMBER(p, s)

VARCHAR

date

DATE

DATE

datetime

TIMESTAMP_NTZ

TIMESTAMP_NTZ

time

TIME

TIME

the rest

VARCHAR

VARCHAR

11.11. History

All synchronization activities left a log about their running. These logs are listed in history card on synchronization detail.

History Table

Result

Result icon shows icon, which represents result of the last synchronization. There are following possibilities:


ok icon success - data in mirror DB table was successfully synchronized with data from Salesforce
warning icon warning - data in mirror DB table was synchronized but something SnowMirror detected potential issues. The synchronization requires your attention.
error icon fail - data could not be synchronized or an error occurred during synchronization
running icon running - synchronization task is currently running
canceling icon canceled - synchronization task is going to be canceled
error icon unknown result - the synchronization was created but it was not executed yet
waiting icon waiting - the synchronization is waiting in the queue to be executed
disabled icon disabled - the synchronization is deactivated and can’t be executed until you reactivate it.
warning disabled icon disabled with warning - the last synchronization’s run ended up with a warning. The synchronization is deactivated and can’t be executed until you reactivate it.
error disabled icon disabled with error - the last synchronization’s run ended up with error. Now the synchronization is deactivated and can’t be executed. It can be reactivated on Synchronization Detail Page.

Start

Date and time when the synchronization was launched.

End

Date and time when the synchronization was stopped (by any cause - finished, canceled etc.)

Started by

Shows who triggered last run. In case of manual synchronization shows name of user who started it. If run was triggered automatically by scheduler shows value <SCHEDULER> regardless who set up the scheduler.

Duration

Is the time from start to finish of the last synchronization run.

Inserted

Number of new records inserted into mirror DB.

Updated

Number of records updated in mirror DB.

Deleted

Number of records deleted from mirror DB

Total

Sum of inserted, updated and deleted records.

11.11.1. Filters and Sorting

To get a sense of history data you can filter activity logs by its results. Just click the funnel icon to see the filtering form. See more on Filtering.

history filters
Filtering Options
  1. Result

    1. Successful - finished its execution successfully

    2. Failed - finished its execution with error

  2. Name - show only synchronizations whose name contains the entered value

  3. Duration - shows only synchronizations whose execution took longer than or equal to the entered value

  4. Next Run - Sets From and To fields based on the selected value. You can also fill in From and To manually

  5. Last Run - Sets From and To fields based on the selected value. You can also fill in From and To manually

  6. Inserted Records - shows synchronizations which inserted at least entered number of records to mirror database

  7. Updated Records - shows synchronizations which updated at least entered number of records in mirror database

  8. Deleted Records - shows synchronizations which deleted at least entered number of records from mirror database

  9. Records Processed - shows synchronizations which processed at least entered number of records

11.11.2. Synchronization Logs

Each synchronization run has its own activity log which includes all activities executed during this synchronization. They should help you to understand what SnowMirror does and in case of trouble provide you enough information to fix the problem. You can find the logs in History tab of Synchronization Detail page, card History.

log error example

11.12. Data

Card data on synchronization detail page provides a simple data explorer of the synchronization’s mirror table.

12. Settings

12.1. Notification Management

To track work of a synchronization, you can add a notification. SnowMirror provides notification for user-specified synchronizations when their run fails or when they succeeds after the previous fail. Each notification can have its own set of tracked synchronizations and recipient email address.

12.1.1. Create a New Notification

To create a new notification you need to input an e-mail address. You can also choose specific synchronization you want to be informed about. If you want to receive information about all synchronization leave the Synchronization column on default All Synchronization option.

notifications

12.1.2. Delete Notification

All added notifications are in a notification table. To cancel sending of notifications you can delete them by selecting a checkbox next to desired email address and clicking on Delete button.

notification list

12.1.3. Notification Types

Let’s look at the events you may be notified about.

  1. Every state change - you will be notified whenever a state of a synchronization changes.

  2. All failed, all warnings, first successful - whenever a synchronization fails or finishes with a warning you will be notified. When you fix it we will send another notification.

  3. All failed, first warning, first successful - whenever a synchronization fails you will be notified. If it finishes with a warning we will notify you only once…​ why would we send it multiple times, you already know about the warning. When you fix it we will send another notification.

  4. Every synchronization run - You will be notified after every synchronization run no matter if it succeeded or failed.

  5. Schema change - if you add this type of notification SnowMirror will check if a schema of a table (new columns, deleted column, …​) changed. If so you will get a notification. SnowMirror won’t update the schema in mirror database unless you enable Auto Schema Update.