Session 09: Synchronization

At the end of this session participants will be able to:

Deploy applications to mobile devices over the Internet
Use the synchronization options dialog to add synchronization to an application
Create advanced synchronizations using CSPro logic to synchronize data files and update applications in the field
Create peer to peer synchronizations between devices for situations where there is no Internet access

Synchronization in CSPro

After collecting data in the field, you need to get the data from the interviewer's devices back to headquarters to create a combined data file. While this can be done manually by connecting each device to a laptop, copying the individual data files and combining them using the Concatenate Data tool, this is rather inconvenient for large surveys. Instead, it is possible to send the data from each device over the Internet to a central server that will combine the data into a single data file to be used for further processing. In CSPro this is called synchronization.

The Sync Server

In order to use Internet synchronization in CSPro you need a server accessible over a network. CSPro supports three types of Synchronization servers:

CSWeb: A free web application that can run in a local data center or hosted on a server in the cloud. Best for large surveys. Requires skills in setting up and administering a website and SQL database.
Dropbox: A free service for synchronization with servers in the cloud using an account created at www.dropbox.com. Data is stored on the servers managed by Dropbox in the United States. Best for small surveys when the skills and infrastructure for setting up CSWeb are not available.
FTP: CSPro can synchronize with any FTP (File Transfer Protocol) server accessible over the network. Best for small surveys when the skills and infrastructure for setting up CSWeb are not available and you don't want your data on the Dropbox servers.

For our examples in this workshop we will use a CSWeb server, however using Dropbox or FTP is very similar. Installing the CSWeb server is beyond the scope of this training so we will work with a server that is already installed and running. The process for setting up a CSWeb server is well documented in the CSWeb User's Guide.

Deploying Applications

Instead of copying our application to a mobile device using a USB connection we can download it from a server. Before we do that we need to use the Deploy Application tool to upload the application to the server. We saw in session 8 how to use the Deploy Application tool to deploy to a local folder. To deploy to a server simply change the "Deploy To" to the type of server to upload to. You can deploy to a CSWeb server, Dropbox or FTP. The deploy tool will upload all of the files that you have dropped onto the files list to the server. It will also upload some or all of the dictionaries in your application to the server. Uploading a dictionary to the server is required before you can synchronize the data file associated with that dictionary.

Once you have uploaded the application to the server you can download to a mobile device by choosing "Add Application" from the menu in the applications listing screen. This will ask you for the server details and then display a list of applications available on the server. Once you download an application it is copied to the CSEntry directory of your device.

You can also deploy updated versions of your application to the server and use the same procedure to download the updated application to your device.

Adding Synchronization to the Data Entry Application

In order to use synchronization, you must first add the server information to the application to enable synchronization. In CSPro choose "Synchronization" from the Options menu. Choose the server type (CSWeb, Dropbox, or FTP). For CSWeb or FTP, enter the server URL (for Dropbox no URL is required). You can use the "Test Connection" button to verify that the URL is correct. For CSWeb, the URL to use for synchronization in CSPro will usually end with "/api" while the URL to use to access the web interface will usually end in "/ui". The former is for CSPro itself to connect to the server while the latter is for use only in the web browser.

In the dropdown, you have three choices for how to synchronize the main data file:

Upload changes to server: all new cases and any cases modified locally will be uploaded to the server.
Download changes from server: all new cases and cases modified on server will be downloaded from server.
Sync local and remote changes: combines the first two options by sending updates to the server and downloading updates from the server.

In most cases you will want to choose either the first or third option. Use the third option if you want to share cases between multiple devices in the field as this will download every case on the server onto the local device. For a survey operation with a large number of interviewers you may want to avoid this as it will cause you to download a lot of data. For our test, we will use the first option.

Once the sync settings are complete, regenerate the pen file. The sync settings are stored in the pen file so if it is not updated you won't be able to sync. Copy the pen and pff file into the Popstan2020Census directory under the files directory on the server so that it will be downloaded to the local device during sync.

Running Sync

To synchronize the application with the server, run the application in CSPro and from the case listing screen choose "Synchronize…" from the file menu on the PC or tap the synchronize icon ( sync icon

) on mobile.

Downloading the Combined Data File

Once you have used synchronize to upload data to the server you can download the data file from the server using the Data Manager tool. This will download a single data file that combines all the data uploaded from all the devices that have synchronized with the server. Open Data Manager, choose the server type, enter the same URL you used in the synchronization options dialog, click "Show Available Data" to download the list of data files available on the server, choose the data file, choose a place to save the downloaded file and click "Download".

Using the CSPro Web Server interface

The CSWeb server has a web interface that you can access through a browser. The URL for the web interface is similar to the one used in CSPro except that the "api" at the end is replaced by "ui". The web interface allows you to manage deployed applications, data dictionaries and user accounts.

If you are using CSWeb, you can avoid filling out the download dialog by clicking on the "Download" icon in the CSWeb interface in the browser. This will download a pff file. When you open the downloaded pff file, it will automatically open Data Manager and begin downloading the combined data. Once you have downloaded the data file the first time, you can use the "synchronize" function in Data Manager to only download the newly updated data rather downloading the entire file a second time. If you are using Dropbox or FTP, CSPro will automatically create a pff file named DownloadData-<your dictionary name>.pff in your Dropbox or on your FTP server that will launch Data Manager to download the data.

Group exercise – Sync Race

Split into groups of 3-4 people and create a small CSPro application for the following questionnaire:

Each team should use their team name as the name for their application's dictionary. If two teams use the same dictionary name they will both sync to the same data file on the server and we would like to avoid that.
Deploy your application to the following server:

URL: http://csweb.teleyah.com/api
Admin username: test
Admin password: password

Add synchronization to your application using the above server details.

Choose to upload data to the server.
Do NOT sync the application files (pen and pff).

Run your application, complete a case and synchronize to send your data to the server.
The first team to get their data uploaded to the server is the winner.

Synchronization from Logic

Sometimes we need more control over the synchronization than the sync options dialog provides. For example, if there are external dictionaries in an application, if you want to sync with a universe or if there are multiple applications to sync at once. CSPro provides logic functions to run synchronizations from within an application that allow more complex synchronizations.

Let's add synchronization to our menu application that will upload the main data file and download the latest version of the pen and pff files for the menu application and for the listing and household questionnaires. We will add a new entry in the supervisor main menu called "Synchronize with Headquarters".

The first step in running sync from logic is to call the command syncconnect() and pass it the server information. If this function succeeds then we are connected to the server and we can make calls to the commands syncdata() and syncfile() to synchronize data files and non-data files respectively. Finally we call syncdisconnect() to end the session.

The commands syncdata() and syncfile() operate differently. syncdata() may only be used with data files in CSPro DB format. With syncdata(), CSPro keeps track of which cases in the file have already been uploaded to the server so that it avoids transferring cases that have already been synced. This reduces the amount of data transferred and keeps data costs and transfer times to a minimum. It also avoids one interviewer overriding changes made by another interviewer when they are working on different cases in the same file. syncfile() may be used with any type of file including text files, images and application files (pen and pff). It does not look at the file contents so it simply uploads or downloads the entire file, overwriting any existing version. When syncing data files, you should always use syncdata().

syncdata() takes two arguments: the direction and the dictionary name. The direction can be PUT (upload data to server), GET (download data from server) or BOTH (upload and download). These options correspond to the drop down in the sync options dialog. The second argument is the name of a dictionary in the application that corresponds to the data file to synchronize. Note that this dictionary must be added to the application as an external dictionary.

In our application, we want to synchronize the household questionnaire dictionary so we need to first add it to the menu program as an external dictionary. Now we can use it in the call to syncdata().

We will also add calls to syncfile() to download the latest versions of the application programs from the server. This function takes the direction (PUT or GET), the "from" directory and the "to" directory. In the case of GET the "from" directory is the path on the server and the "to" directory is the local directory on the device.

// Upload cases to web server at headquarters
// and download latest version of application files
function syncWithHeadquarters()

// Connect to the webserver
if syncconnect(CSWeb, "http://csweb.teleyah.com/api") then

// Sync cases from household data file with server
syncdata(BOTH, POPSTAN2020_DICT);

// Download latest version of menu application
syncfile(GET, "/Popstan2020/Menu/Menu.pen", "./Menu.pen");
syncfile(GET, "/Popstan2020/Menu/Menu.pff", "./Menu.pen");

// Get latest household data entry program
syncfile(GET, "/Popstan2020/Household/Popstan2020.pen",
"../Household/Popstan2020.pen");

// Disconnect from the server
syncdisconnect();

endif;
end;

Now that we have added the synchronization logic to the menu we rebuild the pen files and copy them to the files directory on the server. This will allow us to run the menu program and test our synchronization.

Group exercise – Sync Race II

In your teams modify the sync exercise questionnaire to take a photo of the team and save it to a file.

Add two questions, C and D, to your application. For question C, take a photo if the interviewer answers "yes". For question D, add logic to the postproc of the question so that it synchronizes with the server if the user selects "yes". Use syncfile() with PUT to upload the photo to the directory /photos/ on the server.
Run your application, enter a case and synchronize to send your photo to the server.
The first team to get their photo uploaded to the server is the winner.

Security Considerations

It is important to ensure the security of your server and data. Computer and network security is a challenging problem and well beyond the scope of this training. Here a few security considerations to think about. For any large survey or census operation you should consult with a security expert to ensure that your data is safe.

Use HTTPS on the web server for data transfer

This encrypts the data transmission that goes over the Internet. It requires purchasing and installing an SSL certificate on the server. Without SSL, server passwords are sent unencrypted and are vulnerable to hackers.

Use a private network

For censuses and large surveys you may be able to work with the telecom provider to create a private network that only your devices can access. This avoids putting your server on the public Internet where it could be vulnerable to attacks. This also makes it possible to limit your devices to only be able to access your server and not other websites that you don't want your interviewers to use.

Use device encryption on Android devices

By setting a PIN code on your Android tablet or phone you enable encryption of files on the device using strong hardware encryption. This makes it very difficult for anyone without the PIN code to retrieve data on the device.

Consider how to manage passwords

It is more convenient to have a single password that is shared by all devices and is hardcoded in the pen file however it is more secure to use a different username/password for each interviewer and have them enter it for every sync. For a large number of interviewers managing passwords and resetting passwords for those who have forgotten their passwords could be a significant management burden. You will need to find the right balance of security and convenience. If you want to hardcode the password in your pen file you can pass the username and password in the call to syncconnect(). If you do not provide a username and password in this call, the interviewer is prompted to enter the password the first time they sync with the device. After that the username and password are saved on the device so they do not need to be entered for synchronizations. If you don't want the password saved on the device, you can use the prompt() function to have the interviewer enter the password before each sync and then pass the password received from the call to prompt() to syncconnect().

Synchronization with a Universe

The syncdata() command has an optional third parameter, the universe. This is a string that CSPro tries to match to the concatenated case ID items to limit the data transferred. If a universe is provided, then only cases who case IDs match the universe are synchronized. For example, to limit synchronization to only province 1, district 2 we would call:

syncdata(BOTH, HOUSEHOLDQUESTIONNAIRE_DICT, "102");

This will only synchronize cases whose case IDs start with "102". This limits the sync to cases in province 1 and district 2. Similarly, if we used "1" as the universe all cases in province 1 would be synced.

This is useful when there are a large number of interviewers and you don't want every interviewer to download the data from every other interviewer. You can specify the geographic area assigned to the interviewer as the universe so that only cases in the interviewer's area are synced.

Peer to Peer Synchronization Using Bluetooth

In some places, interviewers do not have reliable Internet access and cannot synchronize directly with the server. In this case, you can use peer to peer to synchronization. Interviewers synchronize with a supervisor's tablet or laptop over Bluetooth. Later, the supervisor goes to a location where they can connect to the Internet and they synchronize with the central server.

In order to implement Bluetooth synchronization, we need to add logic on both the supervisor device and on the interviewer device. One device acts like the web server, processing synchronization requests from the client device, and the other device, the client, sends syncfile and syncdata commands the same way that it would to a web server. For our example let's make the supervisor device be the server and the interviewer device be the client. This is arbitrary, we could easily reverse the roles.

To extend the menu program to support peer to peer synchronization we will add a new menu option to the supervisor main menu "sync with interviewer" and a new menu option to the interviewer main menu "sync with supervisor".

The logic on the interviewer's tablet is nearly identical to the logic used for web synchronization. The only differences are the arguments to syncconnect(). When using Bluetooth, syncconnect() does not need a URL, username or password.

syncconnect(bluetooth)

syncconnect() will scan all nearby Bluetooth devices and present a list to the interviewer who can choose which device to connect to. Alternatively, if you know ahead of time the name of the device to connect to you may specify it as a second parameter. In this case syncconnect() will try to connect directly to the named device.

syncconnect(Bluetooth, "Supervisor03")

The rest of the logic is the same as for web synchronization except that the "from" paths change to match the paths on the supervisor device.

function syncWithSupervisor()
// Connect to the supervisor device
// We do not specify the device name to connect to which allows the interviewer
// to pick the device from a list of nearby devices.
if syncconnect(bluetooth) then

// Sync main data file.
syncdata(BOTH, POPSTAN2020_DICT);

// Download latest application files from the supervisor. The
// root file on the supervisor tablet is in the Menu folder.
syncfile(GET, "./Menu.pen", "./Menu.pen");
syncfile(GET, "./Menu.pff", "./Menu.pff");

// Since the current application is in the Menu folder we need to use
// "../Household" to go up one level and back down into the Household
// folder for the household application files.
syncfile(GET, "../Household/Popstan2020.pen",
"../Household/Popstan2020.pen");

syncdisconnect();
endif;
end;

The logic for the supervisor is even simpler. We simply call the command syncserver() which runs the Bluetooth server and waits for connections.

function syncWithInterviewer()

// Run the Bluetooth server to receive data from interviewer.
syncserver(Bluetooth);
end;

By default, the root directory is the application directory on the server.

Group exercise – Sync Race III

In your teams modify the sync exercise application to support Bluetooth sync.

Modify question D so that you have the option of synchronizing via Bluetooth.
Run your application on two devices, complete a case on one device and synchronize to send your photo to the other device via Bluetooth. Use the second device to synchronize to send your photo the web server.
The first team to get their photo uploaded to the server is the winner.

Exercises

Modify the syncWithHeadquarters() and syncWithSupervisor() functions in the menu program to also sync the listing data file and the staff lookup file. The listing file should be synced using BOTH and the staff file should be downloaded from the server using syncdata with GET.
Modify the syncWithHeadquarters() and syncWithSupervisor() functions so that only data in the geographic area assigned to the interviewer/supervisor are synced. For the interviewer this means that only households in the assigned EA are synced and for the supervisor only households in the assigned district are synced.