Indy SMTP and Exchange Server - delphi

I'm having a very strange problem sending emails via indy through two different mechanisms in my program. This problem is similar to [this one][1], but not exactly the same. I'm using Indy 10.5 with the latest OpenSSL libraries and Delphi XE3.
The first code snippet, which works, is a simple SMTP client I wrote. Here's an example of how I set it up. This isn't the exact code but it should give you an idea.
FIndySMTP.Intercept := FIndyLogFile;
FIndySMTP.IOHandler := FIndySSLHandler;
FIndyMessage.From.Address := FEmailAddress;
FIndySSLHandler.Destination := FSMTPAddress + ':' + IntToStr(FSMTPPort);
FIndySSLHandler.Host := FSMTPAddress;
FIndySSLHandler.Port := FSMTPPort;
FIndySMTP.Host := FSMTPAddress;
FIndySMTP.Port := FSMTPPort;
FIndySMTP.Username := FAccountName;
FIndySMTP.Password := FAccountPass;
FIndySMTP.AuthType := satDefault;
FIndySMTP.UseEhlo := True;
FIndySMTP.UseTLS := utUseExplicitTLS
FIndySMTP.Connect;
try
if FIndySMTP.Connected = True then
FIndySMTP.Send(FIndyMessage);
finally
FIndySMTP.Disconnect;
end;
This generates a successful email with the log:
Recv 10/9/2015 10:41:02 AM: 220 server.server1.local Microsoft ESMTP MAIL Service ready at Fri, 9 Oct 2015 13:41:01 -0400<EOL>
Sent 10/9/2015 10:41:02 AM: EHLO DEIMOS<EOL>
Recv 10/9/2015 10:41:02 AM: 250-server.server1.local Hello [68.14.239.173]<EOL>250-SIZE 36700160<EOL>250-PIPELINING<EOL>250-DSN<EOL>250-ENHANCEDSTATUSCODES<EOL>250-AUTH<EOL>250-8BITMIME<EOL>250-BINARYMIME<EOL>250 CHUNKING<EOL>
Sent 10/9/2015 10:41:02 AM: RSET<EOL>
Recv 10/9/2015 10:41:02 AM: 250 2.0.0 Resetting<EOL>
....(The rest snipped)
Now the second method, which uses Report Builder built in Email components to send a Report via email (again, some code is snipped for brevity, the real clue is in the logs):
lIOHandler.Destination := Host + ':' + IntToStr(Port);
lIOHandler.Host := Host;
lIOHandler.Port := Port;
TheReport.EmailSettings.HostAddress := Host;
TheReport.EmailSettings.UserName := UserName;
TheReport.EmailSettings.Password := Password;
lEmail.SMTP.OnEmailError := FMain.EmailErrorEvent;
TppSMTPIndy(lEmail.SMTP).IndySMTP.OnFailedRecipient := FMain.idSMTPFailedRecipient;
TppSMTPIndy(lEmail.SMTP).IndySMTP.Intercept := FMain.IdLogFile1;
TppSMTPIndy(lEmail.SMTP).IndySMTP.Port := Port;
TppSMTPIndy(lEmail.SMTP).IndySMTP.IOHandler := lIOHandler;
TppSMTPIndy(lEmail.SMTP).IndySMTP.UseTLS := utUseExplicitTLS;
TppSMTPIndy(lEmail.SMTP).IndySMTP.AuthType := satDefault;
TppSMTPIndy(lEmail.SMTP).IndySMTP.UseEhlo := True;
TheReport.SendMail;
You can see that with the exception of using the Report Builder TppSMTPIndy, every setting is the same. Yet the email does not get sent, and the log looks like this:
Stat Connected.
Recv 10/9/2015 10:44:31 AM: 220 server.server1.local Microsoft ESMTP MAIL Service ready at Fri, 9 Oct 2015 13:44:28 -0400<EOL>
Sent 10/9/2015 10:44:31 AM: EHLO DEIMOS<EOL>
Recv 10/9/2015 10:44:31 AM: 250-server.server1.local Hello [68.14.239.173]<EOL>250-SIZE 36700160<EOL>250-PIPELINING<EOL>250-DSN<EOL>250-ENHANCEDSTATUSCODES<EOL>250-AUTH<EOL>250-8BITMIME<EOL>250-BINARYMIME<EOL>250 CHUNKING<EOL>
Sent 10/9/2015 10:44:31 AM: QUIT<EOL>
Recv 10/9/2015 10:44:31 AM: 221 2.0.0 Service closing transmission channel<EOL>
Stat Disconnected.
You can see that a QUIT is sent immediately after receiving HELLO. This is why my question is different than the link above. That person was at least receiving a STARTTLS request.
What could be causing Indy to send a QUIT immediately after receiving a HELLO? I am not getting any errors. It just silently fails and the program moves on.
Now here's a kicker hint. If I set the AuthType to satNone in the Report Builder example it works. While in my first example I can set the AuthType to satNone and satDefault and both work.
Any ideas?
Thanks a lot for your time.

What could be causing Indy to send a QUIT immediately after receiving a HELLO?
The only time QUIT is sent is when TIdSMTP.Disconnect() is called.
The only times that TIdSMTP itself calls Disconnect() are when:
an exception is raised inside of TIdSMTP.Connect(), such as if the server's greeting has an error code, or there is a unexpected problem parsing the server's greeting or EHLO response.
an exception is raised inside of TIdSMTP.StartTLS(), which is called by TIdSMTP.Authenticate() (which is called by TIdSMTP.Send() if not already called beforehand). However, since you have set UseTLS=utUseExplicitTLS and the server's EHLO response does not advertise support for STARTTLS, TIdSMTP.StartTLS() is effectively a no-op on this server.
I am not getting any errors. It just silently fails and the program moves on.
Unless Report Builder is catching exceptions internally and not passing them into your code, then the most likely scenario is that Report Builder itself is calling TIdSMTP.Disconnect() without calling TIdSMTP.Send() first. The RSET command shown in your log is sent by TIdSMTP.Send() at the beginning of the email (BTW, more recent versions of Indy no longer send RSET unless the email fails with an SMTP error code). Report Builder is likely just skipping Send(), and I can think of one possible reason why it might do that.
AuthType=satDefault uses the AUTH LOGIN command (which is not a secure command), but your server's EHLO response is not advertising support for LOGIN authentication (in fact, it is not advertising support for any authentications at all). As such, TIdSMTP.Authenticate() will skip authentication on this server by default and return False, and also set the TIdSMTP.DidAuthenticate property to False. Maybe Report Builder is calling TIdSMTP.Authenticate() directly and checking the result before calling TIdSMTP.Send(). Your non-ReportBuilder example does not do that validation. Setting AuthType=satNone will cause TIdSMTP.Authenticate() to return True and set the TIdSMTP.DidAuthenticate property to True.
If the server happens to support LOGIN authentication (some servers do support it without advertising it), you can set the TIdSMTP.ValidateAuthLoginCapability property to False (it is True by default) to make satDefault attempt authentication as long as the TIdSMTP.Username property has been assigned a non-blank string.

Related

POST with Indy + SSL + Proxy

I'm trying to do a POST request through proxy using https.
Code looks like:
FHttp := TIdHttp.Create(nil);
FHttp.ProxyParams.ProxyServer := Host;
FHttp.ProxyParams.ProxyPort := Port;
FHttp.ProxyParams.ProxyUsername := User;
FHttp.ProxyParams.ProxyPassword := Password;
FHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
FHandler.SSLOptions.Method := sslvTLSv1_2;
FHandler.PassThrough := true;
FHttp.IOHandler := FHandler;
FHttp.HandleRedirects := true;
FHttp.Request.ContentType := 'application/x-www-form-urlencoded';
FHttp.Request.Connection := 'keep-alive';
FHttp.Request.ProxyConnection := 'keep-alive';
...
FParams.Add('username=user');
FParams.Add('password=pwd');
FHttp.Post('https://my.service/login', FParams);
Proxy server is Squid.
Code generates error "Socket Error # 10054 Connection reset by peer."
Now, the interesting part comes:
If not using proxy at all (i.e. not setting FHttp.ProxyParams settings) - everything is OK.
If not setting any POST parameters (i.e. empty FParams), but still using proxy - everything is OK.
The most strange one: If I'm debugging the Indy code step by step (TIdCustomHTTP.DoRequest method) - everything is OK with above example (proxy settings + parameters).
POST parameters are not sent properly for some reason?
And why step 3 is happening?
Indy is up to date, just pulled from repository
UPDATE
After intercepting TIdHTTP calls (thanks Remy) there is a little bit more clarity. (failing log, working log).
Short version: when doing debug, Indy does 3 CONNECT + POST + DISCONNECT requests (because there are redirection on the service I believe) and it works.
When running test without debug - CONNECT + DISCONNECT + POST - and it fails obviously (i.e. POST is executed without CONNECT in front).
See attached log files for details.
You have found some logic bugs in TIdHTTP that need to be fixed. I have opened a new ticket for that:
#315: Bugs in TIdHTTP proxy handling
Here is what I see happening in your "failing" scenario:
TIdHTTP connects to the proxy, sends a CONNECT request that successfully connects to my.service.com:443, then sends a POST request (using HTTP 1.0 rather than HTTP 1.1 a).
a) to send a POST request with HTTP 1.1, you have to set the TIdHTTP.ProtocolVersion property to pv1_1, AND enable the hoKeepOrigProtocol flag in the TIdHTTP.HTTPOptions property. Otherwise, TIdHTTP.Post() forces the ProtocolVersion to pv1_0.
The HTTP server replies with a 302 Found response redirecting to a different URL, including a Keep-Alive header indicating the server will close the connection if a new request is not sent in the next 5 seconds.
When TIdHTTP is done processing the POST response, it knows it is going to re-send the same request to a new URL. On the next loop iteration, it sees that the target server is the same, and the proxy is still connected, and so the connection is not closed, and the code that would have sent a new CONNECT request is skipped.
Just before the POST request is sent, the Response.KeepAlive property is checked to know whether or not to close the socket connection anyway. The KeepAlive property getter sees the ProtocolVersion property is pv1_0 and that there is no Proxy-Connection: keep-alive header present in the response (even though there is a Connection: keep-alive header), so it returns False, and then the socket connection is closed.
TIdHTTP then re-connects to the proxy again, but does not send a new CONNECT request before sending the POST request. The proxy does not know what to do with the POST, so it fails the request with a 400 Bad Request response.
Here is what I see happening in your "working" scenario:
Everything is the same as above, up to the point where the 1st POST request is processed. Then there is a delay of roughly 16 seconds (likely since you are stepping through code) - more than the 5-second Keep-Alive delay allows - so the HTTP server closes its connection with the proxy, which then closes its connection to TIdHTTP.
By the time TIdHTTP is ready to send the 2nd POST request, it knows it has been disconnected from the proxy, so it re-connects to the proxy, sends a new CONNECT request, and then sends the POST request.
Until I can fix the bugs properly, try the following:
enable the hoKeepOrigProtocol flag in the TIdHTTP.HTTPOptions property to allow TIdHTTP.Post() to use HTTP 1.1. That in itself may fix the issue with the connection being closed unnecessarily before sending the 2nd POST request to the redirected URL.
if that doesn't solve the issue, try editing IdHTTP.pas yourself and recompile Indy, to update the TIdCustomHTTP.ConnectToHost() method to force a Disconnect() if the Response.KeepAlive property is False BEFORE the local LUseConnectVerb variable is set to not Connected in the case where ARequest.UseProxy is ctSSLProxy (and ctProxy, too). That way, the 2nd POST request will disconnect from the proxy and re-connect with a new CONNECT request.

Delphi soap client - no response from server

My application consumes some web service.
Segment of calling is below.
try
intf := getInterface(urlString);
request := TransaccionWS1.VentaDispositivoConvenioWSINTO.Create;
// here request field are initiated
//
//
AddToLogFile(format('Transaction Send: tid=%s authid=%s prod=%s',
[request.idExterno, request.nroReserva, elem.codProducto]), log_debug);
rs := intf.ventaDispositivoConvenio(request);
AddToLogFile(format('Transaction Recv: OK=%d result=%s auth_result=%s ' +
'descr=%s ticketnr=%s amount=%s text=%s',
[ord(rs.OK), rs.codigoRetorno, rs.codigoAutorizacion, rs.descripcionRetorno,
rs.nroTicket, rs.importeTicket, rs.textoLegal]), log_debug);
except
on e:exception do begin
AddToLogFile(format('**** Transaction: %s', [e.message]), log_exceptions);
raise;
end
end;
In general it is working ok
But sometimes i have a problem
I can see in log 'Transaction Send' but cannot see no 'Transaction Recv' and no exception.
Another thing i cannot explain, sometimes i see in log
03/01/2014 14:32:35.453 Transaction Send: tid=266996 authid=0000001958472 prod=86
03/01/2014 14:36:09.046 **** Transaction: The connection with the server was reset
Here exception raised after 4 minutes, while client use default timeouts values 30 seconds
May be someone has direction where to search?
You are assuming that your code is wrong, but maybe it's environmental? Consume the WSDL with SoapUI and run some tests to see if the webservice is stable and reachable.

Why it returns 403 when TIdHTTPProxyServer tries to forward the request to most of proxy servers?

the question is extended from the question which I asked before.
Currently I can let my proxy server with TIdHTTPProxyServer forward the request to another proxy server which also using TIdHTTPProxyServer. But when I tried to let it forward the request to other proxy servers from the free proxy servers list on web. Most servers which can be used through IE proxy setting return 403 error message to my proxy server. In fact I have tried about 20 valid proxy servers, but only two can accept the requests from my proxy server. I can not figure out why it happens.
Below is the code I use in HTTPBeforeCommand of TIdHTTPProxyServer.
TIdIOHandlerStack* tempIO = new TIdIOHandlerStack(AContext->OutboundClient);
TIdConnectThroughHttpProxy* tempProxy = new TIdConnectThroughHttpProxy(AContext->OutboundClient);
tempProxy->Enabled = true;
tempProxy->Host = ProxyServerAddress;
tempProxy->Port = ProxyServerPort ;
tempIO->TransparentProxy = tempProxy;
AContext->OutboundClient->IOHandler = tempIO;
After monitoring the behaviors of TIdHTTPProxyServer using wireshark, I found TIdHTTPProxyServer always send a CONNECT request to other proxy servers at first berfore foward the real requests(the browser does not do that).
And then receive 403 response for most proxy servers. But still do not know how to make it works.
Updated on 2012/08/07
Hi, I am not really familiar with those HTTP stuffs, so I just record what I saw in wireshark here. It seems IE uses GET/POST commands for HTTP requests and CONNECT command for HTTPS requests. And most proxy servers block Connect commands when they are not HTTPS request (For example, CONNECT www.google.com.tw:80 HTTP/1.0). That is why TIdConnectThroughHttpProxy always does not work.
Below is my workaround, I made a little bit changes in IdHTTPProxyServer.pas. Hope it is useful for someone else who meets the same problems.
For CONNECT commands, still use TIdConnectThroughHttpProxy
Within TIdHTTPProxyServer.CommandCONNECT
if UseProxy = True then
begin
tempIO := TIdIOHandlerStack.Create(LContext.FOutboundClient);
tempProxy := TIdConnectThroughHttpProxy.Create(LContext.FOutboundClient);
tempProxy.Enabled := True;
tempProxy.Host := UseProxyAddr;
tempProxy.Port := UseProxyPort ;
tempIO.TransparentProxy := tempProxy;
LContext.FOutboundClient.IOHandler := tempIO;
end;
For GET/POST commands, I should directly send GET www.google.com.tw:80 HTTP/1.0 to other proxy servers instead of sending CONNECT www.google.com.tw:80 HTTP/1.0 at first.
Within TIdHTTPProxyServer.CommandPassThrough
if UseProxy = True then
begin
TIdTCPClient(LContext.FOutboundClient).Host := UseProxyAddr;
TIdTCPClient(LContext.FOutboundClient).Port := UseProxyPort;
end else
begin
TIdTCPClient(LContext.FOutboundClient).Host := LURI.Host;
TIdTCPClient(LContext.FOutboundClient).Port := IndyStrToInt(LURI.Port, 80);
end;
Also within TIdHTTPProxyServer.CommandPassThrough, I should let header Proxy-Connection = close
LContext.Connection.IOHandler.Capture(LContext.Headers, '', False);
LContext.Headers.Values['Proxy-Connection'] := 'close';
Finally within TIdHTTPProxyServer.TransferData
if AContext.TransferSource = tsClient then
begin
if UseProxy = True then
begin
ADest.IOHandler.WriteLn(AContext.Command + ' ' + AContext.Target + ' HTTP/1.0');
end else
begin
ADest.IOHandler.WriteLn(AContext.Command + ' ' + AContext.Document + ' HTTP/1.0');
end;
end;
It is my first time to implement delphi, and hope there are better solutions.
I believe you should not use pin-holing - http://www.indyproject.org/docsite/html/TIdConnectThroughHttpProxy.html
CONNECT command is not how WWW works. No browser use it. It is how non-WWW programs try to break-through the firewalls and open direct access to all areas of internet beyond WWW.
Don't use "transparent proxy" classes.
Use regular HTTP proxy, like in How to download a file over HTTPS using Indy 10 and OpenSSL?
BTW, there is no such event handler as u name "HTTPBeforeCommand"
http://www.indyproject.org/docsite/html/!!MEMBEROVERVIEW_TIdHTTPProxyServer.html

Get extended error information when an error 11004 occurs using TIdSmtp.Connect() method?

I have a Delphi 6 application that sends E-mails with attachments. When I first tested it I got an 11004 error (policy violation) when I called TIdSmtp.Connect(). It turned out my E-mail server SMTP setting were wrong and now it works fine. But know my users are going to run into trouble and I'd like to know if there is a way to get more extensive error information back from the SMTP server via the Indy components to help with my debugging efforts. I am using Indy 9 with Delphi 6.
Is there a way to get a much richer report and find out what the probable cause for the error is? (e.g. - SSL required, From field not filled in, rejected domain, etc). Also, same question but when an error occurs on the TIdSmtp.Send() method?
EIdSMTPReplyError is raised in response to an error message from the SMTP server at the SMTP protocol layer. 11004 is a socket/DNS error, not an SMTP error. There would be no additional error information available other than the basic OS error message (which you can get from SysErrorMessage()):
The requested name is valid and was found in the database, but it does not have the correct associated data being resolved for.
Regarding TIdSMTP.Send(), you can look for EIdSMTPReplyError exceptions, eg:
try
IdSMTP1.Connect;
try
IdSMTP1.Send(IdMessage1);
finally
IdSMTP1.Disconnect;
end;
except
on E: EIdSMTPReplyError do
Application.MessageBox(
PChar(
'Error message: ' + E.Message + sLineBreak +
'Error code: ' + IntToStr(E.ErrorCode) + sLineBreak +
'Error reply: ' + E.EnhancedCode.ReplyAsStr
),
'SMTP Error...', MB_OK + MB_ICONSTOP + MB_TOPMOST);
end;
That will help some, and if the server supports Extended Error Codes then you might get a little bit closer. But more times then not, you will not likely be able to distinquish EXACTLY what went wrong without looking at the actual SMTP command/response traffic over the wire.
Regarding SSL/TLS errors in particular, one of several different things may happen, depending on your TIdSMTP configuration:
1) you might get a general EIdSocketError exception raised, like with your 11004 error.
2) You might get an EIdSMTPReplyError exception with a malformed error code if you connect to a server port that requires Implicit SSL/TLS but you do not have TIdSMTP.UseTLS set to utUseImplicitTLS.
2) You might get an EIdOSSLConnectError or other OpenSSL-related exception if you have UseTLS=utUseImplicitTLS assigned but the server is not expecting Implicit SSL/TLS.
3) You have UseTLS set to either utUseExplicitTLS or utRequiresTLS and the server supports Explicit TLS, but the SSL/TLS handshake fails for whatever reason, you will get a TIdSMTP.OnTLSHandShakeFailed event triggered only for utUseExplicitTLS. If the event handler does not set VContinue=True, or you used utRequiresTLS, you will get an EIdTLSClientTLSHandShakeFailed exception raised.
4) You have UseTLS set to either utUseExplicitTLS or utRequiresTLS and the server does not support Explicit TLS, you will get a TIdSMTP.OnTLSNotAvailable event triggered for utUseExplicitTLS only. If the event handler does not set VContinue=True, or you used utRequiresTLS, you will then get an EIdTLSClientTLSNotAvailable exception raised.

Indy idftp client "Invalid argument to time encode"

Using Delphi 2010 and Indy 10.5.8.0.
Against the server Titan FTP I'm getting all the time the exception "Invalid argument to time encode" (EConvertError) when connecting.
The server log tells me:
FEAT<EOL>
211-Extensions Supported<EOL> COMB<EOL> MLST type*;size*;modify*;create*;perm*;<EOL> SIZE<EOL> MDTM<EOL> XCRC<EOL> REST STREAM<EOL> AUTH SSL<EOL> AUTH TLS<EOL> CCC<EOL> PBSZ<EOL> PROT<EOL> EPRT<EOL> EPSV<EOL> DQTA<EOL>211 End<EOL>
TYPE A<EOL>
200 Type set to A.<EOL>
The user "*****" has initiated a session on "217.********:21"
SYST<EOL>
215 UNIX Type: L8<EOL>
SITE ZONE<EOL>
210 UTC-2147483647<EOL>
QUIT<EOL>
221 Session Ended. Downloaded 0KB, Uploaded 0KB. Goodbye *** from 130.******.<EOL>
Any ideas?
The server is sending a faulty UTC offset in response to the SITE ZONE command. That is a bug in Titan. When Indy tries to parse the value for use in later TDateTime operations, the parse fails. Contact the Titan devs and let them know about the bug. In the meantime, I will look into updating TIdFTP to handle that error in the future.

Resources