Error when trying to download cases to a tablet

Discussions about syncing data via Bluetooth, Dropbox, FTP, and using CSWeb
Forum rules
New release: CSPro 8.0
Post Reply
Don
Posts: 55
Joined: February 15th, 2013, 1:28 pm
Location: Barbados

Error when trying to download cases to a tablet

Post by Don »

We are using Filezilla FTP server to synchronise data from the tablets. Our supervisors need to download data to their tablets that the enumerators have uploaded to the servers. However, whenever a supervisor tries to download data, it would give an error message after about 20 minutes of downloading. Stuff like
Error connection to sync server: it.sauronsoftware.ftp4j.FTPDataTransferException: I/O error in data transfer
or
Error connecting to sync server: java.net.SocketTimeoutException
Looking at the FTP server, it was successfully transferring data the entire time until it simply says "Can't open data connection for transfer of ..."

Is there a reason for this? I'm running CSEntry 7.1.3 on a Samsung Galaxy Note 10.1 2014 Edition with Android 5.1.1

My code for the download function is below:

Code: Select all

//downloads Data based on ED
function downloadData(string rnd_edno)
	errflag = 0;
	errmsg("Attempting to download all data from Round :%s, ED :%s",rnd_edno[1:2],rnd_edno[3:3]);
	//errmsg("Data dictionary value Round :%V, ED :%V", RNDNO,EDNO);
	if syncconnect(FTP,ftpip1,ftpun,ftppw) then		//connect to FTP when outside of the government WAN
		if not syncdata(GET,LABOURFORCE_DICT,maketext("%5s",rnd_edno)) then		//get labourforce data for round and ED
			inc(errflag);
		endif;
		if not syncdata(GET,LISTING_DICT,maketext("%5s",rnd_edno)) then			//get control sheet data for round and ED
			inc(errflag);
		endif;
		if not syncdata(get,ENUMERATIONREPORT_DICT,maketext("%v",ME_ED)) then	//get enumeration report for ED
			inc(errflag);
		endif;
		if not syncfile(GET,"/CSPro/Update/Questionnaire/Labourforce.pen","../Questionnaire/Labourforce.pen") then		//update questionnaire
			inc(errflag);
		endif;
		if not syncfile(GET,"/CSPro/Update/ControlSheets/Listing.pen","../ControlSheets/Listing.pen") then		//update control sheet
			inc(errflag);
		endif;
		if not syncfile(GET,"/CSPro/Update/Menu/Menu.pen","../Menu/Menu.pen") then		//update Menu
			inc(errflag);
		endif;
		if not syncfile(GET,"/CSPro/Update/Reports/EnumerationReport.pen","../Reports/EnumerationReport.pen") then		//update Enumeration Report
			inc(errflag);
		endif;
		if not syncfile(GET,"/CSPro/Update/Reports/enumeratorrpt.html","../Reports/enumeratorrpt.html") then		//report template
			inc(errflag);
		endif;
		syncdisconnect();		//disconnect
	else
		inc(errflag);
	endif;
	if errflag >= 1 then
		errmsg("Download of round %s, ED %s failed, Please try again. %d errors were found",rnd_edno[1:2],rnd_edno[3:3],errflag);
	elseif errflag = 0 then
		errmsg("Download of round %s, ED %s was successful",rnd_edno[1:2],rnd_edno[3:3]);
		savesetting("Updated",maketext("%d",sysdate("YYYYMMDD")));
	endif;
end;
Here's the end of my anonymized FTP log:

Code: Select all

(009562)1/31/2019 15:40:59 PM - redacted (redacted)> CWD /CSPro/DataSync/LABOURFORCE_DICT/data/
(009562)1/31/2019 15:40:59 PM - redacted (redacted)> 250 CWD successful. "/CSPro/DataSync/LABOURFORCE_DICT/data" is current directory.
(009562)1/31/2019 15:40:59 PM - redacted (redacted)> TYPE I
(009562)1/31/2019 15:40:59 PM - redacted (redacted)> 200 Type set to I
(009562)1/31/2019 15:40:59 PM - redacted (redacted)> SIZE f81d2d9022e51dcd$f50f5680-d306-4280-a5b3-9f65d3b735ae
(009562)1/31/2019 15:40:59 PM - redacted (redacted)> 213 11519
(009562)1/31/2019 15:40:59 PM - redacted (redacted)> TYPE I
(009562)1/31/2019 15:40:59 PM - redacted (redacted)> 200 Type set to I
(009562)1/31/2019 15:40:59 PM - redacted (redacted)> PASV
(009562)1/31/2019 15:40:59 PM - redacted (redacted)> 227 Entering Passive Mode (redacted)
(009562)1/31/2019 15:40:59 PM - redacted (redacted)> REST 0
(009562)1/31/2019 15:40:59 PM - redacted (redacted)> 350 Rest supported. Restarting at 0
(009562)1/31/2019 15:41:00 PM - redacted (redacted)> RETR f81d2d9022e51dcd$f50f5680-d306-4280-a5b3-9f65d3b735ae
(009562)1/31/2019 15:41:00 PM - redacted (redacted)> 150 Opening data channel for file download from server of "/CSPro/DataSync/LABOURFORCE_DICT/data/f81d2d9022e51dcd$f50f5680-d306-4280-a5b3-9f65d3b735ae"
(009562)1/31/2019 15:41:00 PM - redacted (redacted)> 226 Successfully transferred "/CSPro/DataSync/LABOURFORCE_DICT/data/f81d2d9022e51dcd$f50f5680-d306-4280-a5b3-9f65d3b735ae", unfortunately compression did increase the transfer size by 11 bytes to 11530 bytes (0.09%)
(009562)1/31/2019 15:41:00 PM - redacted (redacted)> CWD /CSPro/DataSync/LABOURFORCE_DICT/data/
(009562)1/31/2019 15:41:00 PM - redacted (redacted)> 250 CWD successful. "/CSPro/DataSync/LABOURFORCE_DICT/data" is current directory.
(009562)1/31/2019 15:41:00 PM - redacted (redacted)> TYPE I
(009562)1/31/2019 15:41:00 PM - redacted (redacted)> 200 Type set to I
(009562)1/31/2019 15:41:00 PM - redacted (redacted)> SIZE f8b156c92236$713ab9f2-8b7a-4f33-81c8-1467c0f520e8
(009562)1/31/2019 15:41:00 PM - redacted (redacted)> 213 87759
(009562)1/31/2019 15:41:00 PM - redacted (redacted)> TYPE I
(009562)1/31/2019 15:41:00 PM - redacted (redacted)> 200 Type set to I
(009562)1/31/2019 15:41:01 PM - redacted (redacted)> PASV
(009562)1/31/2019 15:41:01 PM - redacted (redacted)> 227 Entering Passive Mode (redacted)
(009562)1/31/2019 15:41:01 PM - redacted (redacted)> REST 0
(009562)1/31/2019 15:41:01 PM - redacted (redacted)> 350 Rest supported. Restarting at 0
(009562)1/31/2019 15:41:01 PM - redacted (redacted)> RETR f8b156c92236$713ab9f2-8b7a-4f33-81c8-1467c0f520e8
(009562)1/31/2019 15:41:11 PM - redacted (redacted)> 425 Can't open data connection for transfer of "/CSPro/DataSync/LABOURFORCE_DICT/data/f8b156c92236$713ab9f2-8b7a-4f33-81c8-1467c0f520e8"
josh
Posts: 2399
Joined: May 5th, 2014, 12:49 pm
Location: Washington DC

Re: Error when trying to download cases to a tablet

Post by josh »

I don't see anything in your log file that indicates what the issue could be. Perhaps there is a setting on the server that limits the total connection time. If you can find the server log for your FTP server perhaps there would be some error message that would shed some light on the situation.

If you have enough data on the server that it is taking 20 minutes you should probably use CSWeb instead of FTP. The FTP based sync doesn't scale well when you have a lot of data and it isn't smart enough to limit the downloads to the universe in syncdata(). It ends up downloading everything from the server and then discarding the cases that are outside the universe after downloading everything. With CSWeb the server has a database instead of just files so it can limit the download to only cases that are in the universe. This should make your downloads much much faster and less prone to errors.
Post Reply