Append Data in CSPro 6.3

[khurshid.arshad]

Dear Team

We are going to conduct HH survey and listing on the Tablet. Before starting the HH interview, we will do listing in each specific area, which is called “block”. Each block will be covered by 4 enumerators. The listing software has two records other than IDS0 - “Geographical information” and “Household information”. The max is “1” against each HH.

Now my question is that in the end, we have 4 “dat” files with different number of cases. I want to append all these cases in one case to run sampling as “dat” file. Example:
Enum 1 entered 20 cases
Enum 2 entered 25 cases
Enum 3 entered 32 cases
Enum 4 entered 20 cases
The total cases are 97. I want 97 cases in one case with 97 occurrence.


Thanks.
A.

[josh]

If you are doing this on Windows on a PC you can concatenate the data files together using the concatenate data tool, do a record sort using the sort data tool and then use a different dictionary for the combined file where the max records is now 97 instead of 1 (or whatever the max you expect to have in your largest block). To understand how this works think about the layout of the records in the original data files:

enumerator1 file:
--------------------
household record for hh1
geo record for hh1
household record for hh2
geo record for hh2
household record for hh3
geo record for hh3

enumerator2 file:
--------------------
household record for hh4
geo record for hh4
household record for hh5
geo record for hh5
household record for hh6
geo record for hh6

Now when you concatenate them you have one file with:

household record for hh1
geo record for hh1
household record for hh2
geo record for hh2
household record for hh3
geo record for hh3
household record for hh4
geo record for hh4
household record for hh5
geo record for hh5
household record for hh6
geo record for hh6

After you do the record sort:

household record for hh1
household record for hh2
household record for hh3
household record for hh4
household record for hh5
household record for hh6
geo record for hh1
geo record for hh2
geo record for hh3
geo record for hh4
geo record for hh5
geo record for hh6

which is how the data file needs to be formatted for a single case with multiple occurring records.

In the combined data file you may need to reduce the number of id-items as well. For example if in the original dictionary you have id-items region, block, household number then in the combined dictionary you would have just region and block as id-items and you would move household number to be variable in both of the records in the same start position that it is in in the original dictionary. CSPro only considers consecutive records in the data file to be in the same case if all the id-items match the previous record so if you have the household number in the id-items it won't treat the whole file as a single case. Since the concatenate and sort won't actually change the contents of the records, the household number variable will still be in each line in the data file so in the dictionary you need to move it from the id-items to be part of the records.

If you are doing this on an Android tablet you won't have access to the sort or concatenate tools so you would have to do this using logic. That is a bit more complex. You could still use the same two dictionaries: original and combined. Create a separate data entry program (normally you would use batch for this but batch is not supported on Android) that has a dummy main dictionary and the two other dictionaries as external dictionaries. In loop use setfile to open each of the enumerator data files with the original dictionary (possibly use dirlist to get their names). For each data file use locate and loadcase in a loop to loop through each case in the data file and copy each record to the combined dictionary. Unfortunately there is no simple record copy function in CSPro so you will copy variable by variable. Keep a counter of the record number in the output file that will go from 1 to 97 and use the as the occurrence number to copy into. Increment it after each household. The logic might look something like this:

// Get a list of all enumerator data files in directory
dirlist(fileListing,pathToEnumeratorDataFiles,"*.dat");

numeric iCase = 1;

// Loop through all the enumerator data files
do iFile= 1 while iFile <= length(fileListing)
    // open the file
    if setfile(ORIGINAL_DICT, fileListing(iFile) = 0 then
        errmsg("Failed to open file %s", fileListing(iFile);
    endif;

    // go to first case in file
    if locate(ORIGINAL_DICT, >=, "") =0 then
        errmsg("Failed to read file %s", fileListing(iFile);
    endif;

    // loop through all cases in enumerator data file
    while loadcase(ORIGINAL_DICT) do

        // Copy the id-items from original to combined
        COMBINED_DICT.REGION(iCase) = ORIGINAL_DICT.REGION;
        COMBINED_DICT.BLOCK(iCase) = ORIGINAL_DICT.BLOCK;

        // Insert new record occurrences
        insert(COMBINED_DICT.HOUSEHOLD_REC(iCase);
        insert(COMBINED_DICT.GEOGRAPHIC_REC(iCase);

        // Copy the variables in household info  from original to combined
        COMBINED_DICT.HOUSEHOLD_NUMBER_HH(iCase) = ORIGINAL_DICT.HOUSEHOLD_NUMBER;
        COMBINED_DICT.HEAD(iCase) = ORIGINAL_DICT.HEAD;
        COMBINED_DICT.MALES(iCase) = ORIGINAL_DICT.MALES;
        COMBINED_DICT.FEMALES(iCase) = ORIGINAL_DICT.FEMALES;

        // Copy the variables in geographic info  from original to combined
        COMBINED_DICT.HOUSEHOLD_NUMBER_GEO(iCase) = ORIGINAL_DICT.HOUSEHOLD_NUMBER;
        COMBINED_DICT.LATITUDE(iCase) = ORIGINAL_DICT.LATITUDE;
        COMBINED_DICT.LONGITUDE(iCase) = ORIGINAL_DICT.LONGITUDE;

        // Increment case number
        iCase = iCase + 1;
    enddo;
enddo;

// Write out the combined case
if writecase( COMBINED_DICT) = 0 then
   errmsg("Failed to write combined data file");
endi

This is totally untested so it might need some debugging.

[khurshid.arshad]

Dear Josh
Thank you for prompt reply. I will try on Andriod and share the result with you.

Regards.
a.

[khurshid.arshad]

Dear Josh;

Thank for the logic. Based on this logic, I am sharing program for others.

Please follow the following steps:

In this program I have three "ent" files, Listing, Dummy and Combine.

1. Run listing.ent and enter data by different file name with ".DAT" extension;
2. Run Dummy.ent file to append data in one file. "You can see logic in "PROC LISTING_QUEST"
3. Run Combine.ent program to view all the data as one case in one file.

a.