FTPS file transfer

How can I transfer file from 4D with FTPS protocol?
Should log in with username and password.

I do it with the curl v2 plugin from Magic Miyako

https://github.com/miyako/4d-plugin-curl-v2

<code 4D>
If (Test path name($path)=Is a document)
vr_DocSize:=Get document size($path)

Progress QUIT (0)
$progressId:=Progress New 
Progress SET PROGRESS ($progressId;-1)
Progress SET BUTTON ENABLED ($progressId;False)
Progress SET TITLE ($progressId;FP_FileName ($path))
OB SET($options;\
"URL";$vt_remote;\
"UPLOAD";1;\
"USERNAME";vt_userName;\
"PRIVATE";String($progressId);\
"PASSWORD";vt_password)

$vr_filesize:=(vr_DocSize/1000)/1000
Case of 
	: ($vr_filesize>10)
		OB SET($options;"UPLOAD_BUFFERSIZE";2000000)
	: ($vr_filesize>5)
		OB SET($options;"UPLOAD_BUFFERSIZE";1000000)
	Else 
		OB SET($options;"UPLOAD_BUFFERSIZE";500000)
End case 

$cacert:=FP_CheckPath (FP_CheckPath (Get 4D folder(Current resources folder))+"XSL")+"cacert.pem"
OB SET($options;"CAINFO";$cacert)  //need this for ftps
OB SET($options;"SSL_VERIFYPEER";0)
OB SET($options;"SSL_VERIFYHOST";0)
OB SET($options;"USE_SSL";"USESSL_TRY")

SET BLOB SIZE($Request;0)
SET BLOB SIZE($response;0)
If (Test path name($path)=Is a document)
	DOCUMENT TO BLOB($path;$Request)
	$error:=cURL (JSON Stringify($options);$Request;$response;"FTPSend_PROGRESS";$transferInfo;$headerInfo)
	If (BLOB size($response)>0)
		$returntxt:=Convert to text($response;"utf-8")
	End if 
End if 
Progress QUIT ($progressId)

End if
</code 4D>

Hi Armin,

I notice in your code you set the UPLOAD_BUFFERSIZE. Would you mind explaining what this does and the effect changing it has for you?

Thanks much.

Thanks.
I will try it.
If I go to your link I find: “cURL.dmg v 3.5” in a folder. Is this the one I should download?

[]33565309;“I get a file like this”[/]
Filename: “cURL.bundle”

Should I put it in the “Plugins”-folder?

: Cannon SMITH

Hi Armin,

I notice in your code you set the UPLOAD_BUFFERSIZE. Would you mind
explaining what this does and the effect changing it has for you?

Thanks much.

More Upload Speed for large files

Thanks, Armin. Do you find that only the size of the file you are sending matters? How about the upload bandwidth?

You can also use https://github.com/miyako/4d-plugin-curl-ftpcurl-ftp>, maybe easier because it mimics FTP commands from 4DIC plugin

: Cannon SMITH

Thanks, Armin. Do you find that only the size of the file you are
sending matters? How about the upload bandwidth?

While testing the FTP transfer with curl I noticed that SFTP or FTPS are noticeably slower than FTP. I guess that the encryption costs some bandwidth.

I did the test with 30 transmissions each and took the average value.

By increasing the UPLOAD_BUFFERSIZE I was able to shorten the required transfer time.
But if the transfer files are not that big, then a bigger buffer does not show the curl callback progress thermometer.

On macOS the increase has only a minor advantage, because on macOS CURL via SFTP or FTPS is faster than under windows.

Armin

Hi Armin,

Thank you for the details. That helps me know what kind of testing I can try.

Thanks!

Armin,
I am trying your solution, but there is some procedures/commands I can not find.
FP_FileName($path)
FP_CheckPath()

Can you please help me :?:

FP_FileName($path) return the file name of the given path
FP_CheckPath($path) insures that the correct folder separator for a directory is at the last position

You can change the line to:

$cacert:=Get 4D folder(Current resources folder)+“cacert.pem”
If you copy the cacert.pem inside your resources folder.

You can find it inside the curl demo:
[]33618473;“Your comment here…”[/]

And You have to replace my progress message “FTPSend_PROGRESS” to “”

Miyako updates yesterday version 3.7 of the curl plugin to github.

Armin

Hi again,

A small question, I am still struggling.

How does your URL looks like? ($vt_remote)
The folder I am sending to is the root folder.

“URL”;$vt_remote;

Per

Product :4D - 4D Server
OS : Mac OS X

: Per Ivar SKAANEVIK

Hi again,

A small question, I am still struggling.

How does your URL looks like? ($vt_remote)
The folder I am sending to is the root folder.

“URL”;$vt_remote;

Per

Product :4D - 4D Server
OS : Mac OS X

use curl with ftps:

<code 4D>
$vt_remote:=“ftp://ftp.mydomain.com/myfile.zip
vr_DocSize:=Get document size($path)
OB SET($options;
“URL”;$vt_remote;
“UPLOAD”;1;
“USERNAME”;vt_userName;
“PASSWORD”;vt_password)

$vr_filesize:=(vr_DocSize/1000)/1000
Case of 
	: ($vr_filesize>10)
		OB SET($options;"UPLOAD_BUFFERSIZE";2000000)
	: ($vr_filesize>5)
		OB SET($options;"UPLOAD_BUFFERSIZE";1000000)
	Else 
		OB SET($options;"UPLOAD_BUFFERSIZE";500000)
End case 

$cacert:=Get 4D folder(Current resources folder)+"cacert.pem"
OB SET($options;"CAINFO";$cacert)  //need this for ftps
OB SET($options;"SSL_VERIFYPEER";0)
OB SET($options;"SSL_VERIFYHOST";0)
OB SET($options;"USE_SSL";"USESSL_TRY")
SET BLOB SIZE($Request;0)
SET BLOB SIZE($response;0)
DOCUMENT TO BLOB($path;$Request)
$error:=cURL (JSON Stringify($options);$Request;$response;"";$transferInfo;$headerInfo)
If (BLOB size($response)>0)
	$returntxt:=Convert to text($response;"utf-8")
End if 

</code 4D>

And a tip for use curl with sftp:

<code 4D>
$vt_remote:=“sftp://ftp.mydomain.com/myfile.zip”
vr_DocSize:=Get document size($path)
OB SET($options;
“URL”;$vt_remote;
“UPLOAD”;1;
“USERNAME”;vt_userName;
“PASSWORD”;vt_password)
OB SET($options;“SSL_VERIFYPEER”;0)
OB SET($options;“SSL_VERIFYHOST”;0)
$vr_filesize:=(vr_DocSize/1000)/1000
Case of
: ($vr_filesize>10)
OB SET($options;“UPLOAD_BUFFERSIZE”;2000000)
: ($vr_filesize>5)
OB SET($options;“UPLOAD_BUFFERSIZE”;1000000)
Else
OB SET($options;“UPLOAD_BUFFERSIZE”;500000)
End case
SET BLOB SIZE($Request;0)
SET BLOB SIZE($response;0)
DOCUMENT TO BLOB($path;$Request)
$error:=cURL (JSON Stringify($options);$Request;$response;"";$transferInfo;$headerInfo)
If (BLOB size($response)>0)
$returntxt:=Convert to text($response;“utf-8”)
End if

</code 4D>

And a tip for use curl with ftp:

<code 4D>
$vt_remote:=“ftp://ftp.mydomain.com/myfile.zip
vr_DocSize:=Get document size($path)
OB SET($options;
“URL”;$vt_remote;
“UPLOAD”;1;
“USERNAME”;vt_userName;
“PASSWORD”;vt_password)

$vr_filesize:=(vr_DocSize/1000)/1000
Case of 
	: ($vr_filesize>10)
		OB SET($options;"UPLOAD_BUFFERSIZE";2000000)
	: ($vr_filesize>5)
		OB SET($options;"UPLOAD_BUFFERSIZE";1000000)
	Else 
		OB SET($options;"UPLOAD_BUFFERSIZE";500000)
End case 
OB SET($options;"SSL_VERIFYPEER";0)
OB SET($options;"SSL_VERIFYHOST";0)
SET BLOB SIZE($Request;0)
SET BLOB SIZE($response;0)
DOCUMENT TO BLOB($path;$Request)
$error:=cURL (JSON Stringify($options);$Request;$response;"";$transferInfo;$headerInfo)
If (BLOB size($response)>0)
	$returntxt:=Convert to text($response;"utf-8")
End if 

</code 4D>

And a method to change the curl error code in readable text, some lines are in german.

<code 4D>
C_LONGINT($1)
C_TEXT($0)
Case of
: ($1=1)
$0:=“UNSUPPORTED_PROTOCOL (1) Probieren Sie sFTP oder FTPs in den Portaleinstellungen”
: ($1=2)
$0:=“FAILED_INIT (2)”
: ($1=3)
$0:=“URL_MALFORMAT (3) The URL was not properly formatted”
: ($1=4)
$0:=“NOT_BUILT_IN (4) A requested feature, protocol or option was not found built-in in this libcurl due to a build-time decision. This means that a feature or option was not enabled or explicitly disabled when libcurl was built and in order to get it to f”+“unction you have to get a rebuilt libcurl”
: ($1=5)
$0:=“COULDNT_RESOLVE_PROXY (5) Couldn’t resolve proxy. The given proxy host could not be resolved”
: ($1=6)
$0:=“COULDNT_RESOLVE_HOST (6) Couldn’t resolve host. The given remote host was not resolved”
: ($1=7)
$0:=“COULDNT_CONNECT (7) Tipp Probieren Sie bei den Portaleinstellungen in den Einstellungen SFTP anzuhaken (Häufiges Problem bei Immowelt)”
: ($1=8)
$0:=“FTP_WEIRD_SERVER_REPLY (8)”
: ($1=9)
$0:=“REMOTE_ACCESS_DENIED (9)”
: ($1=10)
$0:=“FTP_ACCEPT_FAILED (10)”
: ($1=11)
$0:=“FTP_WEIRD_PASS_REPLY (11) After having sent the FTP password to the server, libcurl expects a proper reply. This error code indicates that an unexpected code was returned”
: ($1=12)
$0:=“FTP_ACCEPT_TIMEOUT (12) During an active FTP session while waiting for the server to connect, the CURLOPT_ACCEPTTIMEOUT_MS (or the internal default) timeout expired”
: ($1=13)
$0:=“FTP_WEIRD_PASV_REPLY (13) libcurl failed to get a sensible result back from the server as a response to either a PASV or a EPSV command. The server is flawed”
: ($1=14)
$0:=“FTP_WEIRD_227_FORMAT (14) FTP servers return a 227-line as a response to a PASV command. If libcurl fails to parse that line, this return code is passed back”
: ($1=15)
$0:=“FTP_CANT_GET_HOST (15) An internal failure to lookup the host used for the new connection”
: ($1=16)
$0:="HTTP2 (16) problem was detected in the HTTP2 framing layer. This is somewhat generic and can be one out of several problems, see the error buffer for details. "
: ($1=17)
$0:=“FTP_COULDNT_SET_TYPE (17) Received an error when trying to set the transfer mode to binary or ASCII”
: ($1=18)
$0:=“PARTIAL_FILE (18) A file transfer was shorter or larger than expected. This happens when the server first reports an expected transfer size, and then delivers data that doesn’t match the previously given size”
: ($1=19)
$0:=“FTP_COULDNT_RETR_FILE (19) This was either a weird reply to a ‘RETR’ command or a zero byte transfer complete”

: ($1=21)
	$0:="QUOTE_ERROR (21) When sending custom Quote commands to the remote server, one of the commands returned an error code that was 400 or higher (for FTP) or otherwise indicated unsuccessful completion of the command. "
: ($1=22)
	$0:="HTTP_RETURNED_ERROR (22) This is returned if CURLOPT_FAILONERROR is set TRUE and the HTTP server returns an error code that is >= 400"
: ($1=23)
	$0:="WRITE_ERROR (23) An error occurred when writing received data to a local file, or an error was returned to libcurl from a write callback"
: ($1=24)
	$0:=""
: ($1=25)
	$0:="UPLOAD_FAILED (25) Failed starting the upload. For FTP, the server typically denied the STOR command. The error buffer usually contains the server's explanation for this"
: ($1=26)
	$0:="READ_ERROR (26) There was a problem reading a local file or an error returned by the read callback"
: ($1=27)
	$0:="OUT_OF_MEMORY (27) A memory allocation request failed. This is serious badness and things are severely screwed up if this ever occurs"
: ($1=28)
	$0:="OPERATION_TIMEDOUT (28) Operation timeout. The specified time-out period was reached according to the conditions"
	
: ($1=30)
	$0:="FTP_PORT_FAILED (30) The FTP PORT command returned error. This mostly happens when you haven't specified a good enough address for libcurl to use. See CURLOPT_FTPPORT"
: ($1=31)
	$0:="FTP_COULDNT_USE_REST (31) The FTP REST command returned error. This should never happen if the server is sane"
	
: ($1=34)
	$0:="HTTP_POST_ERROR (34) This is an odd error that mainly occurs due to internal confusion"
: ($1=35)
	$0:="SSL_CONNECT_ERROR (35) A problem occurred somewhere in the SSL/TLS handshake. You really want the error buffer and read the message there as it pinpoints the problem slightly more. Could be certificates (file formats, paths, permissions), passwords, a"+"nd others"
: ($1=36)
	$0:="BAD_DOWNLOAD_RESUME (36) The download could not be resumed because the specified offset was out of the file boundary"
: ($1=37)
	$0:="FILE_COULDNT_READ_FILE (37) A file given with FILE:// couldn't be opened. Most likely because the file path doesn't identify an existing file. Did you check file permissions"
: ($1=45)
	$0:="INTERFACE_FAILED (45) Interface error. A specified outgoing interface could not be used. Set which interface to use for outgoing connections' source IP address with CURLOPT_INTERFACE"
: ($1=47)
	$0:="TOO_MANY_REDIRECTS (47) Too many redirects. When following redirects, libcurl hit the maximum amount. Set your limit with CURLOPT_MAXREDIRS"
: ($1=53)
	$0:="SSL_ENGINE_NOTFOUND (53) The specified crypto engine wasn't found"
: ($1=54)
	$0:="SSL_ENGINE_SETFAILED (54) Failed setting the selected SSL crypto engine as default"
: ($1=55)
	$0:="SEND_ERROR (55) Failed sending network data"
: ($1=56)
	$0:="RECV_ERROR (56) Failure with receiving network data"
: ($1=58)
	$0:="SSL_CERTPROBLEM (58) problem with the local client certificate"
: ($1=59)
	$0:="SSL_CIPHER (59) Couldn't use specified cipher"
: ($1=60)
	$0:="PEER_FAILED_VERIFICATION (60) The remote server's SSL certificate or SSH md5 fingerprint was deemed not OK. This error code has been unified with CURLE_SSL_CACERT since 7.62.0. Its previous value was 51"
: ($1=61)
	$0:="BAD_CONTENT_ENCODING (61) Unrecognized transfer encoding. "
: ($1=62)
	$0:="LDAP_INVALID_URL (62) "
: ($1=63)
	$0:="FILESIZE_EXCEEDED (63) Maximum file size exceeded."
: ($1=64)
	$0:="USE_SSL_FAILED (64) Requested FTP SSL level failed"
: ($1=65)
	$0:="SEND_FAIL_REWIND (65) When doing a send operation curl had to rewind the data to retransmit, but the rewinding operation failed"
: ($1=66)
	$0:="SSL_ENGINE_INITFAILED (66) Initiating the SSL Engine failed"
: ($1=67)
	$0:="LOGIN_DENIED (67) The remote server denied curl to login"
: ($1=73)
	$0:="REMOTE_FILE_EXISTS (73) File already exists and will not be overwritten"
: ($1=75)
	$0:="CONV_FAILED (75) Character conversion failed"
: ($1=77)
	$0:="SSL_CACERT_BADFILE (77) Problem with reading the SSL CA cert (path? access rights"
: ($1=78)
	$0:="REMOTE_FILE_NOT_FOUND (78) The resource referenced in the URL does not exist"
: ($1=79)
	$0:="SSH (79) An unspecified error occurred during the SSH session"
: ($1=80)
	$0:="SSL_SHUTDOWN_FAILED (80) Failed to shut down the SSL connection. "
: ($1=81)
	$0:="AGAIN (81) Socket is not ready for send/recv wait till it's ready and try again. This return code is only returned from curl_easy_recv and curl_easy_send"
: ($1=82)
	$0:="SSL_CRL_BADFILE (82) Failed to load CRL file"
: ($1=83)
	$0:="SSL_ISSUER_ERROR (83) Issuer check failed"
: ($1=84)
	$0:="FTP_PRET_FAILED (84) The FTP server does not understand the PRET command at all or does not support the given argument. Be careful when using CURLOPT_CUSTOMREQUEST, a custom LIST command will be sent with PRET CMD before PASV as well"
	
Else 
	$0:=String($1)

End case
</code 4D>

Armin

Armin, thanks for the method with errors in English, I was looking for that all over internet in the form I can convert to 4D without retyping them.

If you are going to Summit in Paris, look me up, I owe you a beer for this. :sunglasses:

Hallo,

da hat Miyako natürlich ein Plugin für geschrieben :slight_smile:

Ich hoffe das hilft weiter.

Gruß,
Christian Görgen

ja hier https://github.com/miyako/4d-plugin-curl-ftp

Super Danke!

wenn wir MiyakoSan nicht hätten…

…müsste 4D doch glatt die Internet Commands überarbeiten, was für diesen Fall vielleicht gar kein Fehler wäre.

habe das plugin heruntergeladen, finde aber kein Beispiel mit ftps

der Versuch mit

C_LONGINT($error)
C_TEXT(list)
C_OBJECT($options)

OB SET($options;“URL”;“ftps://ftps.xyserver.de/”;“USERNAME”;“abc”;“PASSWORD”;“1234”)

$error:=cURL_FTP_GetDirList (JSON Stringify($options);list)

ergibt

$error = 7
CURLE_COULDNT_CONNECT (7)
Failed to connect() to host or proxy.

Fehlen da noch Options oder wo könnte ich suchen?

Danke für die Hilfe

noch ein Hinweis der vielleicht weiter hilft.

der FTP Client Cyberduck auf auf win10 verwendet folgenden Verbindung.

[]30039720;“Your comment here…”[/]

So machen wir es bei uns:

<code 4D>

APPEND TO ARRAY($tNomOption_AL;CURLOPT_USERNAME)
APPEND TO ARRAY($tValOption_AT;$Account_T)
APPEND TO ARRAY($tNomOption_AL;CURLOPT_PASSWORD)
APPEND TO ARRAY($tValOption_AT;$Password_T)
APPEND TO ARRAY($tNomOption_AL;CURLOPT_SSL_VERIFYHOST)
APPEND TO ARRAY($tValOption_AT;“0”)
APPEND TO ARRAY($tNomOption_AL;CURLOPT_SSL_VERIFYPEER)
APPEND TO ARRAY($tValOption_AT;“0”)
APPEND TO ARRAY($tNomOption_AL;CURLOPT_UPLOAD)
APPEND TO ARRAY($tValOption_AT;“1”)

$FileName_T:=“ftps://”+$URL_T+$ServerPfad_T+$FileName_T
$err_L:=cURL ($FileName_T;$tNomOption_AL;$tValOption_AT;$CalendarBlob_X;$out_X)
$ErrTxt_T:=BLOB to text($out_X;UTF8 text without length)

</code 4D>