Using only CSPro there is no simple way to split a data file into several parts. Someone asked me: "How would I split a file with 300 cases into six files, each with 50 cases?" It is possible to do this by writing a recursive batch program. This is not a particularly efficient way to split a file into parts, but it works fine for data files that are not so large. This code is probably not worth using if your data file contains more than a million cases.

What I do here is use the skip case statement to selectively write out cases. The first run of the program, I do nothing but create a PFF that calls the program again with the starting position. Then that program runs, writing out certain cases and skipping others, and then calls the program again, with a new starting position. This continues until the whole file has been processed. In the above example, the program would be run seven times, once to initialize the PFF, and then six times for each block of 50 cases. See the code below:

You can use this code almost exactly as is, with the following modifications:

1. Modify the numeric numCasesPerFile from 50 to your liking.
2. Replace "CEN2000" with the name of your dictionary. (There are two places where this appears.)
3. Replace "DICTIONARY_FF" with the name of your top-level batch PROC. (It will end with _FF.)
4. Replace "QUEST" with the name of your dictionary's first level.

See here for an example of this application using the Popstan dictionary.

Recently I put up two new tools that may be useful to people using CSPro to edit data.

The first tool, Listing File Comparer, provides a way of quickly looking at the error percentages across a group of listing files. This is useful for people who process data, typically census data, on files split by geography.

The second tool, Save Array Viewer, is a program that visually displays the contents of save array files. This program is especially useful if you use DeckArrays for hotdeck imputation.

Also newly posted on the site is a tutorial about creating CAPI applications written by Anne Abelsæth of Statistics Norway: "Development of Data Entry and CAPI Applications in CSPro"

Unfortunately, CSPro does not have a way, within a data entry application, to get a listing of the IDs of the other cases that have been entered to the primary data file. If your application is somewhat simple, you can write a two-dictionary application to facilitate a basic version of case management.

In this example, the main dictionary of the application is a junk dictionary. Any data entered to this dictionary will be ignored. We only use this dictionary to provide the framework for the main data entry application and as a way to enter a menu selection.

The external dictionary and form is actually where data is entered for this application. By using loadcase and writecase statements, it is possible to add and modify cases.

Whenever the data entry application is started, an array is populated with information about all of the cases in the data file. In this example, the program is hardcoded to expect that information about households 1-8 will eventually be added to the file. The program reports on what has been entered and what cases are remaining.

This list is created by using the find statement to check on all of the expected IDs in the external file (which really is the main data file for this application). Then the setvalueset and setcapturetype functions display the results on the screen.

Download the example here, or view the code below.

Using checkboxes for categorical variables is a common way for censuses and surveys to quickly get data onto a paper form and scanned into a data file. Enumerators do not have to translate responses into numbers, and scanners can more easily recognize checked boxes than handwritten characters. These checkboxes need to be converted into single variables with single responses. But what do you do when there is more than one checked box, either due to enumerators not understanding how to complete the question or due to scanners reading stray marks as checked boxes?

When converting checkbox variables into a single variable, there are two basic methods for correcting multiple checks: hotdeck between two marked boxes or determine an order of likelihood. If three or more boxes are marked, the edits become more complicated and, due to the small probability of such occurrences, one can either write complicated edits or the converted variable can be set to blank and edited later using more detailed content edits.

Before converting multiple checkbox variables into a single variable, one must first check the number of boxes marked. This can be done by counting the number of individual checkbox variables. Then the conversion coding will be done based on the number of checked boxes. If only one box is checked, the new variable can easily be created without any special methods to determine the value.

Assuming that the checkbox variables have a value of 1 if checked and 0 if not checked, one can count the number of marked boxes like so:

Using a Hotdeck

Hotdecking is a simple way to estimate the value of a variable based on the checked boxes and on related variables. In the example below, in cases when two boxes are checked, the selection of which of the two values will be used for the new variable is determined from the roofing material. The hotdeck array has two dimensions: the first for each combination of the two checked boxes, and the other for the roofing material. When there are no checked boxes, then the new variable is left blank, to be handled by later edits. When one box is checked, the new variable is coded according to the checked box and the hotdeck is updated based on each combination of two checked boxes and on the roofing material. If two boxes are checked, then the new variable is coded based on the hotdeck using the given combination of checked boxes and the roof type.

Question: Type of Dwelling?
Answers: Separate, Apartment, Joint/Barrack House
Conversion Method: Hotdeck based on roofing material

Order of Likelihood

In a series of responses, it may be decided that certain responses are more likely than others. When multiple boxes are checked, the predetermined more likely response will be recoded as the actual response. In the example below, enumerators may mistakenly select more than one source of drinking water if the household has more than one source. However, the enumerators were instructed to select only the most "modern" water source, so the responses were prioritized in the order that they appear, with the exception of tubewell. Thus, if tubewell and any other response were selected, the new variable is set to tubewell, and this recoding priority continues with tap, then well, and so on.

Question: Main Source of Drinking Water?
Answers: Tap, Tubewell/Deep Tubewell, Well, Pond, River/Ditch/Canal, Others
Conversion Method: Recode with priority (tubewell, tap, well, etc.)

If conversions like this are often being made, the code in the numChecks = 1 and numChecks = 2 sections can be combined in a reusable function.

As mentioned in my previous post, Unicode support (and thus internationalization) will be a great addition to CSPro, coming out in the next half year. After that, the development team plans to focus on the CAPI (computer assisted personal interviewing) world. CSPro currently supports a very basic version of CAPI, but only for Windows platforms. With the proliferation of Android devices, as well as the upcoming Windows 8 tablets, CSPro must adapt to this new world of enumeration.

The world of small-scale surveys may not change dramatically, but the impact of technology on censuses is huge. This is a photo from an East African country that recently conducted a census. The warehouse in this photo stores all of the census forms and requires many workers to operate:

What if an EA were not in the final set ... how easy would it be to find in a mountain of forms? Imagine a world in which all data collection is conducted on a phone or a tablet and immediately sent to the operation headquarters. Data editing would be minimized and the time from collection to publication could effectively be cut to almost zero. Such a world will be an exciting one.