cUrl Plugin - smtp does not work in some environments

Hi,

we experienced huge problems with the internet commands for sending e-mails

  • smtp_send - produces mail bodies that have an incorrect structure -> some e-mail servers do not accept the e-mails and attatchments are not always shown on iOS mail clients)
  • smtp_quick_send - no support for cc and bcc => not an option for business mails).
    Those are handled in taow cases, but there is not short or midterm solution in sight.

Hence we were looking for another way to send e-mails and found the curl plugin.

Thank you for that plugin, it’s easy to use and does the job most of the time.

During our pilot shipment of that new way to send e-mails we got problems.
Depending on the environment (?) (virus scanner, fire wall, etc. (we are not sure who interferes)) or smtp server the curl plugin produces an error and is not sending the e-mail.

Using the exact same parameters for the plugin in our environment (same smtp server with same account) might work or not, depending on the scenario. (We have currently 2 scenarios).

We tried sending the e-mail using curl in a command line.
Using the curl version of the plugin leads to the same problem. (1 year old curl version).
Using an up to date curl version sends the e-mail in any case.

We are no experts on writing plugins (win and mac) and have not the knowledge to build the plugin with the latest curl version.
Neither can we be sure if this would solve all problems.

Here are 2 problems we found so fare:

Scenario 1: Does not work in the partner environment but does work in our environment:

The debug output of the plugin produces:

info: Rebuilt URL to: smtp://smtp.1und1.de:25/
info: Trying 212.227.15.167…
info: Connected to smtp.1und1.de (212.227.15.167) port 25 (#0)
info: Marked for [keep alive]: SMTP default
headerIn: 220 kundenserver.de (mreue009) Nemesis ESMTP Service ready
headerOut: EHLO DESKTOP-T639N2I
headerIn: 250-kundenserver.de Hello DESKTOP-T639N2I [84.177.244.186]
headerIn: 250-8BITMIME
headerIn: 250-AUTH LOGIN PLAIN
headerIn: 250-SIZE 69920427
headerIn: 250 STARTTLS
headerOut: STARTTLS
headerIn: 220 OK
info: TLSv1.2, TLS handshake, Client hello (1):
:

Scenario 2: Does not work in either environment:

info: Rebuilt URL to: smtp://ms11smtp.webland.ch:465/
info: Trying 92.43.217.111…
info: Connected to ms11smtp.webland.ch (92.43.217.111) port 465 (#0)
info: Marked for [keep alive]: SMTP default
info: response reading failed
info: Marked for [closure]: SMTP done with bad status
info: Closing connection 0

Same configuration in outlook works fine.

We would be very grateful to get some advice, as it’s vital for our business to have a robust way to send e-mails (with attachments, cc and bcc) in a well-formed (correct structure of the mail body) way.

Best regards
Arnd Graf

IMHO scenario 1 does not work for your client because port 25 is not open. Nowadays SSL port is usually used instead.
And scenario 2 does not work due to old SSL used by the plugin - that’s why latest command line curl works for you. I assume you are using it on Windows (statically linked ‘some’ version of SSL), because on Mac it uses current version found in the system.

Thank you for the hint, that helped solving scenario 2.
We learned that ‘smtps’ is not the same as ‘smtp + ssl’
=> We adjusted our code to take care of that and have the protocols and ports now adjustable to the needs of the smtp server.

Scenario 1 works in our network but not in the network of our partner. The call with an up to date curl and commandline works on both sides. Incompatible ssl libs might not be the reason as it works on our side with the same settings, except there is a man in the middle at our partner who is breaking it?!

Anyone having the curl plugin build with a newer version of the curl lib?

I had that ‘man in the middle’ in the mind - port 25 disabled by internal firewall at client, allowed on your network…

Kindly ask Miyako for help - to get newer build.

Hi,

use port 587, most modern mailserver use this port instead of 25. SSL v1-v3 is disabled, just TLS, that’s OK.

4D Internet Commands tend to produce ‘bad headers’ (1x CRLF too much) and mails will go to the spam collection (best case).

Use a gmail account for testing. If it passes Google’s wait open eyes, it’s fine.

No need to build your own plugin, it can be done with ‘NTK’, did this in the dark ages when 4D didn’t support SSL mails.

NTK is by far the best choice.

I did not really expect cURL to be used for SMTP that much,
although I like the design that it focuses on the network protocol and delegates MIME to the caller.

in fact, creating bad MIME is mostly where Internet Commands fails you, as Peter rightly explains.

for sending emails there is also etpan

https://github.com/miyako/4d-plugin-etpan

you could also create your own MIME with GMIME

https://github.com/miyako/4d-plugin-gmime

and send it with Internet Commands’ QuickSend with option 8 or 9 (send raw MIME).

I am not that keen on updating the plugin anymore, as it uses the ‘single’ API and not very efficient with concurrent calls. I also have to tweak libcurl every time I update so that it does not block 4D in long calls.

I like to focus on the HTTP and FTP editions which uses the ‘multi’ API (no need to tweak except for unicode path support on windows) and written to support preemptive mode.

Thank you for your feedback.

NTK - I downloaded the demo and documentation. Looks like a very low level api. I assume you need to learn a lot about the smtp protocol in order to use the NTK for it. We are no low level communication experts …

The mime is not the problem. We already create our own body mime and are now able to have attachments inline and as separate documents in one e-mail and stil get all the content displayed on iOS Mail Clients. (That is not possible with smtp_send due to the incorrect body …). Switching again to a framework that creates the mime is a risk, it might not consider every case and we are locked in again…

SMTP_Quick_Send is not an option, it does not support cc and bcc. (At least this was the feedback from the 4D hotline).

The etpan plugin was not updated for 2 years. If it’s a problem with the ssl / tsl area, we might run in the same issue. I’m not sure, if it supports a ‘raw’ mode, where we can put our own body mime in.

cUrl is so easy to use, it’s well maintained, gets lots of fixes and enhancements and it’s used all over the world.

Our software runs at more than 600 customers = 600 different system settings. If a software can handle that, than it’s cUrl.

Using cUrl via command line has 2 disadvantages:

  1. How to pass the password in a secure way. You can activate a logger for the command line stuff and you get the password (we have tried it). That’s a show stopper.
  2. On larger customer sites (especially the banking area) .exe files are perceived as a security issue and monitored. Especially if the are in %appdata% synced like in the client case.

By the way: Why does the cUrl plugin contains a ‘curl.exe’ file besides the dll? At least smtp seems to work even after removing it from the plugin. (The IT of one of our banking customers noticed it on the day that we introduced the curl plugin in our deployment (not yet active for all customers, but deployed)).

As issues with the Internet Commands and smtp rises (more and more mail servers rejection those e-mails completely), I expect more 4D developers will have the need for a simple and robust way to send e-mails.

An update on the plugin would be very welcomed.

If I am going to update the plugin, I would also like to do some refactoring (no arrays, use JSON instead) to make it future proof.

So the syntax will have to change.

It should be straight forward to make the transition, but existing code will not work ‘as is’.

Would that be an acceptable condition for you?

The exe is there to compare the behaviour of plugin vs CLI.

I guess I will remove it in the next round since nobody seems to be using it.

Perfect!
The changes to our code are straigth forward and we are looking forward to doing them.

Thank you very much for your assistance.

here is the repository

with Mac 32-bit

https://github.com/miyako/4d-plugin-curl-v2/tree/carbon

without Mac 32-bit

https://github.com/miyako/4d-plugin-curl-v2/tree/master

Thank you for helping us so quickly, we appreciate it very much.

We don’t want to push you, just for our internal planning:
Can you give us a rough estimation on when the plugin will be testable? Days, weeks or months?

Thank you very much again
Arnd Graf

the plugin is complete and ready.
otherwise I would not have published it on GitHub.

OK, then I assume that the build of the plugin was not successful.

There is no curl.dll in any of the plugin folders and executing Method2 of the test database returns $error=1 and an empty response.

Maybe it’s just some manufacturing problem (4d-plugin-curl-v2-carbon\curl\cURL.4dbase\Plugins\cURL.bundle\Contents\Windows64 etc. do not contain the dlls).

Thank you for your assistance.

you can check out the commits list

https://github.com/miyako/4d-plugin-curl-v2/commits/master

windows support happened yesterday.

maybe you are using a download from the day before.

https://github.com/miyako/4d-plugin-curl-v2/releases

cURL.4DX is the plugin. there is no DLL.

I think I found a problem (but not the same as what you reported).

the libcurl inside the plugin has only basic features (no SSL, noHTTP/2, etc).

We got the plugin sending mails in standalone mode and on the server.
But we get a strange message running the exact same code on a 4D client.

$User:=‘test_4d@bankettprofi.de’
$From:=‘test_4d@bankettprofi.de’
$To:=‘test_mail_1@bankettprofi.de’
$Passwort:=
$Server:=‘smtp.1und1.de:25

$Body:=‘To: ‘+$To+’\r\n’+
‘From: ‘+$From+’\r\n’+
‘Message-ID: <’+Generate UUID+’@bp-event-software.com>\r\n’+
‘Subject: Test Curl 1und1\r\n’+
‘\r\n’+
‘The body of the message.\r\n’

If ($Server#‘smtps://@’)
$Server:=‘smtp://’+$Server
End if

C_OBJECT($Options)
C_BLOB($Request;$Response)
C_TEXT($TransferInfo;$HeaderInfo)
OB SET($Options;‘URL’;$Server)
OB SET($Options;‘MAIL_FROM’;$From)
ARRAY TEXT($Receptions;0)
APPEND TO ARRAY($Receptions;$To)
OB SET ARRAY($Options;‘MAIL_RCPT’;$Receptions)

OB SET($Options;‘PASSWORD’;$Passwort)
OB SET($Options;‘USERNAME’;$User)
OB SET($Options;‘SSL_VERIFYPEER’;0)
OB SET($Options;‘SSL_VERIFYHOST’;0)
OB SET($Options;‘USE_SSL’;‘USESSL_TRY’)
OB SET($Options;‘UPLOAD’;1)
TEXT TO BLOB($Body;$Request;UTF8 text without length)

$err:=cURL (
JSON Stringify($Options);
$Request;
$Response;
‘’;
$TransferInfo;
$HeaderInfo)

ALERT($HeaderInfo)
ALERT(’$err:=’+String($err))

Running this code on a 4D client we get error 55 and
220 kundenserver.de (mreue107) Nemesis ESMTP Service ready
250-kundenserver.de Hello WS042 [24.134.53.25]
250-8BITMIME
250-AUTH LOGIN PLAIN
250-SIZE 69920427
250 STARTTLS
220 OK
250-kundenserver.de Hello WS042 [24.134.53.25]
250-8BITMIME
250-AUTH LOGIN PLAIN
250 SIZE 69920427
334
235 Authentication succeeded
552 Requested mail action aborted: exceeded storage allocation
221 kundenserver.de Service closing transmission channel
==> Mail is not sent

Activating the method to be executed on the server or running it in standalone mode we get error 0 and
220 kundenserver.de (mreue012) Nemesis ESMTP Service ready
250-kundenserver.de Hello VM006 [24.134.53.25]
250-8BITMIME
250-AUTH LOGIN PLAIN
250-SIZE 69920427
250 STARTTLS
220 OK
250-kundenserver.de Hello VM006 [24.134.53.25]
250-8BITMIME
250-AUTH LOGIN PLAIN
250 SIZE 69920427
334
235 Authentication succeeded
250 Requested mail action okay, completed
250 OK
354 Start mail input; end with .
250 Requested mail action okay, completed: id=1MSqbe-1fxln81oqM-00UFgO
==> Mail is sent.

The mail box is empty, no problem with quota.
Any idea what to do about
552 Requested mail action aborted: exceeded storage allocation
on the 4D Client mode?

Just an idea … attachement exceeds allowed size?

honestly I have no idea.

I tried your code,

I generally get

235 2.7.0 Authentication successful
250 2.1.0 Sender OK
250 2.1.5 Recipient OK
354 Start mail input; end with .
250 2.6.0 F8F39285168D224FA62834379300CF96@4d.com [InternalId=80298708566028, Hostname=EXCH16.ad2.4d.com] 1368 bytes in 0.715, 1,866 KB/sec Queued mail for delivery

but the same code sometimes return

235 2.7.0 Authentication successful
552 5.3.4 Message size exceeds fixed maximum message size
221 2.0.0 Service closing transmission channel

running on the client.

it seems random.

do you constantly reproduce the problem?