1. Getting Started

SnowMirror

1.1. Introduction

SnowMirror is data replication tool for Salesforce. The goal of SnowMirror is to replicate data from Salesforce into a local database. Data is transferred via direct web services and saved into local database within user’s ICT infrastructure. SnowMirror enables to manage synchronized data easily and use them for other processing such as computing summary statistics, creating data analysis etc. The data is accessible anytime without frequent requests to Salesforce, which could be time consuming and impacts its performance.

SnowMirror provides user friendly graphic interface. It is easy to set or edit which Salesforce tables and columns will be synchronized, when and how often. A user can create, view and edit synchronization settings, runs, view synchronization details, history, synchronized data or synchronization logs directly in SnowMirror.

1.2. Architecture

SnowMirror loads data from Salesforce using SOAP API and Bulk API and saves it into mirror DB. This process is called synchronization. Configuration of synchronization is stored in configuration DB. Synchronization can load data from one Salesforce table and save it to one mirror DB table. Multiple synchronizations can load data from same Salesforce table, but each synchronization has its own mirror DB table. For more information about synchronizations see Migration Process and New Synchronization.

snowMirrorBasicScheme

Multiple SnowMirror instances with separate databases can be connected to single Salesforce organization.

Synchronizations are managed via embedded web server that also provides comprehensive information about synchronization health and performance. For more information see Dashboard.

1.2.1. Database

Both configuration and mirror database can be provided by wide range of DBMS like MySQL, Postgres, SQL Server, Oracle and H2. If you are creating databases manually, use UTF-8 encoding.

Configuration database

Configuration database contains data about users and roles, synchronization logs and of course about synchronization tasks.

Mirror database

Mirrored data from Salesforce are stored in mirror database. A structure of tables and columns is created as a copy of tables and columns from Salesforce. You should never manipulate database structure from any other place than SnowMirror GUI. All of the data is just mirrored Salesforce tables (or parts of tables) without any transformations. The only additional table columns are mirror_created_on and mirror_updated_on which contains date and time of row creation and last update. The "mirror_" prefix is configurable.

It is necessary to set both databases to use UTC timezone as a default timezone for all date - related records. This will help you to reduce possibility of date inconsistency because Salesforce is using UTC too.

1.3. Migration Process

This chapter covers tasks performed by SnowMirror during a synchronization itself.

SnowMirror loads data from Salesforce table using SOAP and Bulk APIs. If first finds out Ids of the records it wants to replicate and then download them using Bulk API. Salesforce prepares the data for download in CSV format. The maximum size of CSV file is 1 GB. If this size is exceeded, more CSV files are downloaded. Then we split the records into batches. Every single batch is inserted into database in a separate transaction. Batch sizes are configurable in Advanced settings of SnowMirror (property snowMirror.number.of.records).

To ensure correct synchronization process, please make sure you have correct date and time set on your computer.

1.3.1. Transactions

Data batches are transactional but whole synchronization runs are not. This prevents application from reverting data in case of an error after several minutes of synchronization run.

1.3.2. Migration Tasks

Synchronization lifecycle consists of two essential steps. First, new and updated records are synchronized. Then deleted records are synchronized. These essential steps are extended by several additional steps.

migration
Initializing Migration

First task in migration process loads all data from configuration database, which are needed to run the synchronization.

Count Records

To figure out the approximate volume of data that will be synchronized, SnowMirror sends a web service request with SoQL query. This data is useful while rendering progressbars and is stored in activitylog table in configuration database as totalToInsert, totalToDelete and totalToUpdate columns.
Requests for inserted, updated and deleted records has to be done in three separate web service requests.

Create Directory if not exists (only attachment synchronization)

If directory for saving attachments does not exists, it is created in this stage.

Create or Alter Table

If a table does not exist in mirror DB, it is created in this stage.

If synchronization configuration has changed since the last synchronization run, the table is altered.

Insert, Update

This task load Ids of records created and/or updated since the last synchronization. Then it creates a job in Salesforce and download the data in batches.

migration detailed
Delete

Delete task is different than insert and update tasks. Most Salesforce tables stores deleted data in delete log table (i.e. table is audited). Preferred way to determine which rows were deleted is to read them from the delete log. However not every table is audited. That means that deleted rows are not stored in the delete log.

For this case SnowMirror offers four deleting strategies.

Delete Strategies

Audit

Audit deleting strategy gets deleted records via the delete log table. This way is the most effective because we only get created or changed data after last synchronization run.

Truncate

Truncate deleting strategy deletes all data from mirror table before the insert task. Update task is skipped. Note, that LastModifiedDate and CreatedDate columns are set to time of last synchronization start. It can not be used to determine the time of actual row insert or update in Salesforce.

Difference

Difference compares Ids in Salesforce table with the Ids in mirror DB. Ids that are present in mirror table, but not Salesforce table are deleted. LastModifiedDate and CreatedDate are left intact.

None

Does not delete any data from the mirror table. Insert and update tasks act as normal. Deleting data can be done manually by Clean and Synchronize button on detail of the synchronization (See Synchronization Detail ).

If you choose truncate strategy as your delete strategy, there are some changes in migration process as you can see on the picture below.

synchronization process truncate
Default deleting strategy is Audit for tables that contains CreatedDate and LastModifiedDate. For history tables, default deleting strategy is None. For any other table that does not contain CreatedDate or LastModifiedDate and is not History table, default deleting strategy is Truncate.
Finalize Migration

Finalizing task updates the activity logs and finishes the synchronization.

1.3.3. Cache Management

The application uses cache for Salesforce metadata about tables and columns, parent-child structure etc. Because of this cache some changes in Salesforce can be taken into account with a delay.

If you want to clear the cache manually go to Settings menu, choose Cache management and click on Clear cache button.

Cache timeout is configurable in the configuration file.

1.3.4. Activity Log

Each synchronization run has its own activity log which includes information about all activities executed during this synchronization. They should help you understand what SnowMirror does and in case of trouble provide you enough information to fix the problem. You can find the logs on synchronization detail page, card history.

The logs meta-data are stored in the database and the actual log data are stored in files. Default log directory is logs/activityLogs. The files are located by this pattern: <synchronizationId>/<yyyy-MM>/<logId>.log. If you find the activityLogs directory too large you can delete files or whole directories to save some disk space.

1.4. Synchronization Types

In this chapter we will describe all types of synchronizations you can encounter in SnowMirror.

1.4.1. Incremental Synchronization

SnowMirror requests only inserted, updated and deleted records. It is suitable to run on daily basis.

1.4.2. Clean and Synchronize

SnowMirror cleans all data in mirrored table and downloads all records new.

After the clean it may take a long time to synchronize the data again. During the synchronization the old data won’t be available. To tackle the issue you may want to enable "snowMirror.replication.cleanAndSyncToTmpTable" which synchronizes the data to temporary table. Only after synchronization finishes successfully the original mirror table is removed and replaced with the temorary table.

1.4.3. Differential Synchronization

Inserts all missing and deletes all redundant records. It does not update any records.

This algorithm works as follows:
1. It requests the first Ids from Salesforce ordered alphabetically
2. It loads Ids from the mirror database ordered alphabetically
3. It computes a difference to find out new and deleted Ids. If it cannot decide if the records is new or deleted it will process it in the next batch.
4. It repeats the first three steps until there is no records left to process.

This is particularly useful if you need to repair your mirror table. See Consistency Check chapter for examples.

1.4.4. Synchronize From

SnowMirror downloads only records whose SystemModstamp column contains a value greater than or equal to the date you chose.

This is useful if your synchronization missed or ignored some records, you have fixed the root cause of the issue and now you want to download the missing records.

1.5. Consistency Check

In some cases SnowMirror may miss synchronizing some records.

Imagine that you imported data to your table in Salesforce and all records had SystemModstamp set on some old date. SnowMirror already synchronized newer data hence it will miss the newly imported records. This is when Consistency check comes in handy. After every synchronization run it counts the number of records in Salesforce and number of records in mirrored table. Then it compares the counts and if it finds out that they are not equal it will put the synchronization into Warning state and send notification email (if configured to do so).

consistency check list

When you encounter such a warning you should start looking for a reason of the inconsistency. To tackle the issue with old dates in imported records you can use differential synchronization. To diagnose any other potential issues you may want to use validate button.

consistency check activity log

See Admin Guide on how to configure consistency check.

1.6. Mirror Database Constraints And Indexes

SnowMirror creates unique index on "Id" column in every mirror table. Because of that, it can update a single record really fast.

You can even configure SnowMirror to create primary keys instead of unique keys. Go to Settings → Advanced Settings page and set a property "snowMirror.replication.index.idConstraintType" to "PRIMARY_KEY".

2. Backups

SnowMirror allows you to copy and archive data from Salesforce so that it may be restored in case of data loss. It supports backups of tables and attachments.

Data from tables can be stored to CSV files. This file can be imported to Salesforce using Data Loader. You can use them to import your data back to Salesforce.

2.1. Backup Strategies

SnowMirror supports three backup strategies. You can run them manually or automatically by a scheduler.

  1. Incremental - downloads changes made since the last backup (no matter if it was full or incremental).

  2. Differential - downloads changes made since the last full backup.

  3. Full - downloads all data.

2.2. Location

All backups are stored to <SnowMirror installation folder>/snow-mirror/data/backups directory. It can be changed on General Settings page using "Backups" field. Please make sure to have sufficient permissions for creating directories and files in this directory.

change directory

Each backup synchronization creates its own directory with the name of the synchronization.

Sample Backup Structure:
<SnowMirrorDirectory>/data/backups
├───Account
│       2581_FULL_2016-43-08_07h-43m-50s.zip
│       2582_INCREMENTAL_2016-44-08_07h-44m-35s.zip
│       2583_INCREMENTAL_2016-46-08_07h-46m-35s.zip
│
└───Contact
        2581_FULL_2016-43-08_07h-43m-50s.zip
        2582_INCREMENTAL_2016-44-08_07h-44m-35s.zip
        2583_INCREMENTAL_2016-46-08_07h-46m-35s.zip

2.3. Retention Period

Every day SnowMirror runs a job which removes old backups. Each synchronization defines a "retention period". The job will remove backups older that the retention period.

2.4. Table Backup

The backup synchronization works the same as table synchronization works except a few exceptions:

  • Backups do not use the mirror database at all. They are stored exclusively on filesystem in files.

  • SnowMirror allows a user to store dates and times in whatever timezone she wishes (default being UTC). Backups can only store dates and times in UTC so that the values can be imported back to Salesforce.

2.4.1. CSV File Formats

Data is stored into a file in CSV (comma separated values) format. The CSV file is always compressed into a ZIP file which highly reduces the size of the backup.

The file has the following format: [ActivityLogId_TypeOfTheSynchronization_Date_Time].

  • ActivityLogId - ID of the activity log created during the backup

  • TypeOfTheSynchronization - FULL or INCREMENTAL. See Synchronization Types chapter for details.

  • Date_Time - date and time when the synchronization started

Example:
2581_FULL_2016-43-08_07h-43m-50s.zip

2.4.2. Create and Edit

For detailed description of how to create a new synchronization please New Synchronization Wizard. Also, see Retention Period (Backups Only) section describing backup specific fields.

Here, we will just discuss several specifics related to backups.

  • Deleting Strategy - you cannot choose any deleting strategy, because CSV file can’t store deleted records.

  • Columns - you can choose columns you want to synchronize.

new sync

2.4.3. Import To Salesforce

You can use Salesforce Data Loader to import the data back to Salesforce.

You have to locate your backup ZIP file, unpack it and import the contained CSV file using Data Loader. See Data Loader documentation for details how to import CSV files.

2.5. Attachments Backup

Attachments are stored into a directory structure which is packed into a single ZIP file. See the sample structure below.

Sample Backup Structure:
<SnowMirrorDirectory>/data/backups/
├───Account
│       2581_FULL_2016-43-08_07h-43m-50s.zip
│      └───INC0000001
│            └───FirstAttachmentOfInc0000001.png
│            └───SecondAttachmentOfInc0000001.png
│       2582_INCREMENTAL_2016-44-08_07h-44m-35s.zip
│      └───INC0000001
│            └───ThirdAttachmentOfInc0000001.png

It is worth mentioning that such a directory structure cannot store deleted files. If you delete an attachment in Salesforce it won’t be tracked in ZIP file. Therefore, you should perform Full backup from time to time.

2.5.1. Create and Edit

For detailed description of how to create a new synchronization please New Synchronization Wizard. Also, see Retention Period (Backups Only) section.

3. Installation

For quick installation see Installation section in Admin Guide.

4. Licensing

You need to enter a valid license key for SnowMirror to be able to synchronize data.

There are three types of licenses: Lite, Enterprise and Trial. Enterprise and Trial editions contain all features and support all databases. Lite edition contains limited number of features. See comparison on our website.

If you don’t enter a valid license SnowMirror will show a warning at the top of each page. To retrieve a valid key visit our website.

no license entered

If your license is about to expire this message will appear in the page header:

license expiring

5. Logging in/out

Every user has a unique username for logging in and one or more user roles, which gives him permission to access application functionality. The name of logged in user is shown on the right side of top-horizontal menu. If you want to log out, use the Logout link, which is placed on the right side of top menu next to username of currently logged in user. User roles are added or removed only by Super Administrator. For more information about roles and permissions see User Roles).

5.1. Permanent login

On login page you can choose to stay logged in indefinitely by checking Remember Me checkbox. This option creates a cookie in your browser containing access token. The cookie is removed only after clicking "Logout". Otherwise you stay logged in indefinitely.

6. User Roles

Users of SnowMirror has to have at least one of the following roles.

Synchronization Reader

Synchronization Reader can view synchronizations, its data and trigger a synchronization.

  • read synchronization details

  • read synchronization logs

  • read migrated data

  • manually trigger synchronizations

  • stop currently running synchronizations

  • change his or hers own time zone

Synchronization Administrator

Synchronization Administrator has permission to read and manage synchronizations. It includes all permissions of 'Synchronization reader' plus:

  • create, edit and delete synchronizations

  • activate and deactivate synchronizations

  • create, edit and delete email notifications

Super Administrator

Super Administrator has beside all permissions of 'Synchronization administrator' access to user management, which includes:

  • create, edit and delete users

  • edit permissions of any user

  • change password of any user

  • activate and deactivate any user

API Caller

API Caller has permission to call REST API

  • authenticate against REST API and call operations on its resources

7. Dashboard

Dashboard is the main page of SnowMirror and it is divided into three parts - list of failed synchronizations, graph of migrated data volume and last / upcoming synchronizations.

7.1. Failed Synchronizations

'Failed Synchronizations' shows synchronizations that failed during the last run. Failed synchronizations shows name of synchronization and time of failure. By clicking on the synchronization name you can obtain more information about the synchronization see Synchronization Detail.

7.2. Graph of Migrated Records

Graph of migrated records shows global statistic of all transferred data per day. It counts a sum of all inserted/updated/deleted records from all synchronizations executed during a particular day. Every each color represents a kind of data operation similar to progressbar.

Much more information is available at the the detail of synchronization. It is possible to use a link on every synchronization name in the table to go to synchronization detail.

migrated records graph

7.3. Last / Upcoming Synchronizations

Upcoming synchronizations shows all synchronizations which are currently running or will be executed in next 12 hours by default. Last synchronizations shows all synchronization that were triggered in past 12 hours. This interval can be set in the property file.
It includes basic details like time of synchronization execution, result of the last execution and progressbar. Contents are sortable.

Last / Upcoming Synchronizations Table:

Name

Name of the synchronization. It is chosen by a user and it does not have to match the table name in mirror database or the name of the table in Salesforce.

Next Run

Next Run is a datum of next scheduled run in format mm/dd/yyyy hh:mm. Synchronization with manual execution scheduler has this field empty.

Duration

Is the time from start to finish of the last synchronization run.

Synchronization Progress

Synchronization progress bar shows ratio of migrated records. It is continuously updated and shows actual progress of the running synchronization. Progress bar has three colors, each of them represents a kind of data operation:

green icon green for data, which were inserted into mirror DB
blue icon blue for updated data
grey icon grey for records deleted from mirror DB

8. Synchronization Overview

Synchronization Overview includes a list of all created synchronizations. Table includes the preview of basic information and is sortable.