records are disappearing

[AriSilva]

I know this is very tricky because I cannot reproduce it, but same cases are having problems with missing records.
The situation is the following:
1. we have a program that creates the cases (we call it the menu program) and then this program passes the control to the interview program.
2. the interview program does all the interview with all the forms.
3. the interviewer can save partially the interview and come back later.
4. there is a single record (the person record) that is fueled by several forms (we call them a questionnaire block)
5. the menu program also sends the questionnaires to the server.
6. by looking at the server, a certain questionnaire, in a certain day, has all the person records.
7. later, we do not know how, the interviewer does something wrong (we do not know what) and when it sends the questionnaire to the server the person records are not there anymore.
8. we do not have any delete command, in the menu program or the interview program.
9. of course this does not happen all the time, most of the cases are correct, but we could not pinpoint the cause.
10. we were guessing about using the left arrow, the case tree, but the interviewers we talked with swear they did not use them.
Have you ever heard about something like that?
What might be going wrong?
Is there a matter of duplicating records and the system are doing some crazy "merge".
We are really lost
by the way, we are still with the 7.1

[josh]

If you use syncdata(), the syncing is all done at the case (questionnaire) level. There is no merging inside the questionnaire of what is on the server and what is sent from the tablet. Whatever records that exist in the questionnaire on the tablet are sent to the server and they replace whatever was on the server before for that questionnaire.

When you see that there are missing records on the server if you look at the tablet are they missing there too?

[AriSilva]

I´m sorry that I did not mention that it does not have anything with the syncdata functions.
The missing records are missing in the tablet also.
It happened between partial savings.
Some people thought that it happened after implementing a routine to copy the csdb file to an external card, but I do not agree with them. I cannot see why copying a file to the card would provoke missing records in a specific questionnaire.

[josh]

Ok. Thanks for the clarification.

When copying the csdb file you should be careful not to do so while the file is open by the data entry application. This is because the SQLite database system that is used for the csdb file creates some temporary files on the disk that store pending transactions. If you copy the csdb while those temporary files are in use you may not get the pending changes. So if you are using filecopy() to copy the csdb file you should always call close() first on the csdb and then call open() or setfile() to reopen the file afterwards. I'm not sure that would be the problem as I would expect that this would only cause the records to be lost if you had just entered them right before calling filecopy().

We have not seen any records lost with a partialsave(). If you are able to figure out anything more about the circumstances please let us know.

[AriSilva]

The filecopy for the backup on the memory card is done only in the menu program, and the datafile is not open at that point. The menu program reads the data file to set some internal tables at the very beginning (using a forcase loop), and close it afterwards. By the way, the forcase loop automatically closes the file at the end? Anyway, now I´m closing it directly with the close function, just in case.
The interview program does nothing as far as handling the datafile, it only reads (and writes) the specific case it opened when starting the program.
It is called by the menu program who generates a pff file with the startmode as follows:
filewrite(pffFile, "StartMode=modify;%s", idQuest);

I was wondering that one of the ways that might delete one record is to blank all the variables in it, and then, CsPro would not write it when saving the case. But we never blank the person records (it is a huge record with more than 2800 bytes), nor we delete it by mistake, there is no delete instructions in the programs.

We thought also that the deletion might be connected with the inadvertent usage of either the left arrow and/or the case tree, but in all the tests we have made internally so far we were not able to reproduce the problems.
Does it have anything to do with having several forms connected with a single physical record? (we already used this strategy several times before and it really worked well, but...).

One thing that occurred to us is that, after being in the field for a week, we changed the dictionary, inserting a variable at the end of the person record, but you (or Greg, I do not remember), said that we could do that safely.

[josh]

Forcase does not close the file automatically so adding the close is a good idea just in case.

If you were to set all the variables in a record to blank and that record was the last record in the roster/group then it would not get written to the file.

We have used several forms that share a single record quite a bit and have not seen any issues related to it.

Inserting a variable at the end of the record is absolutely fine as it does not change any of the existing data. The only dictionary change that I could see causing records to get deleted would be if you decreased the max for a record or if you changed the record type. Inserting a variable in the middle of the record or changing the size of the variable will also mess up your existing data but it will just shift data from one variable to another, not cause an entire record to be empty.

Are the missing records always the last ones in the case or are they in the middle of the case?

Do you have any endlevel or endgroup calls in the logic that could get triggered? If you call endgroup or endlevel while in the middle of a roster or repeating form that could cause the record occurrences that follow to be deleted.

[AriSilva]

It is not one record that is missing. The whole record type (person) is not written in the file. That is, the questionnaire has 2 persons (and we know that because we keep registers of the number of persons in another record type, as well as some basic characteristics as name, sex and age), and next time we open it there is no records for the persons.

We are not using any endlevel or endgroup calls. However, we do have some heavy coding in the onstop() function, but it worked well so far, for this survey (and all the other surveys we had so far) .

It is really a mistery.

[nlxlee]

Did you manage to resolve this? I had a similar issue. After entry, the program would lose all data I entered. I later noticed that the close function was closing the current dictionary instead of the external dictionary that I was using as the reference file. The only issue I'm facing now is, I always have to refresh whenever I start a new interview. The program doesn't automatically refresh the reference file despite having the open and close functions before and after the loadcase function

[justinlakier]

Hello,

When you are calling close, are you specifying the dictionary?
If you attach your zipped application files, or send them to cspro@lists.census.gov, along with instructions on how to reproduce the issue, we will be able to test the issue.

Thank you,
Justin