I'm currently using BizTalk Server 2013 R2 to exchange EDI as well as non-EDI documents using AS2 with a number of different trading partners. I recently added a new trading partner and after receiving a number of documents successfully I started seeing this error occur every now and then:
An output message of the component "Microsoft.BizTalk.EdiInt.PipelineComponents" in receive pipeline "Microsoft.BizTalk.EdiInt.DefaultPipelines.AS2Receive, Microsoft.BizTalk.Edi.EdiIntPipelines, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" is suspended due to the following error: The content transfer encoding quoted-printable is not supported..
The sequence number of the suspended message is 2.
After some investigation I found that the AS2 platform of the trading partner in question will sometimes set the Content-Transfer-Encoding of the MIME body part to quoted-printable when the enclosed XML payload contains non-ASCII characters. When this happens the message is suspended (non-resumable) with the error above.
Messages received from this trading partner are encrypted and signed, but not compressed - and received using a HTTP request-response (two-way) port configured with the out-of-the-box AS2Receive pipeline. I've tried using a custom pipeline with the AS Decoder, S/MIME decoder and AS2 disassembler components, but this does not seem to have any effect - the error stays the same.
I've also tried receiving unencrypted messages from the trading partner (by mutual agreement) but seem to be doing something wrong here as well as the message passed to the Message Box then ends up not being disassembled properly (the MIME part boundaries and AS2 signature is still visible in the actual message payload). Since the trading partner won't allow sending of unencrypted messages in a production environment anyway, I need to get this working with encryption. They also cannot change their platform's behavior as this will reportedly affect all of their other trading partners.
Here are the unfolded HTTP headers (ellipses denotes redacted values) of the encrypted and signed AS2 message received at the point of being suspended:
Date: Mon, 20 Jan 2020 17:30:53 GMT
Content-Length: 8014
Content-Type: application/pkcs7-mime; name="smime.p7m"; smime-type=enveloped-data
From: ...
Host: ...
User-Agent: Jakarta Commons-HttpClient/3.1
AS2-To: ...
Subject: AS2 Message from ... to ...
Message-Id: <1C20200120-173053-740219#xxx.xxx.130.163>
Disposition-Notification-To: <mailto:...> ...
Disposition-Notification-Options: signed-receipt-protocol=optional, pkcs7-signature; signed-receipt-micalg=optional, sha1
AS2-From: ...
AS2-Version: 1.1
content-disposition: attachment; filename="smime.p7m"
X-Original-URL: /as2
Here is the unencrypted (ellipses denotes redacted content) payload when exact same message is sent from source party without encryption:
------=_Part_16155_1587439544.1579506174880
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
...
------=_Part_16155_1587439544.1579506174880
Content-Type: application/pkcs7-signature; name=smime.p7s; smime-type=signed-data
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
...
------=_Part_16155_1587439544.1579506174880--
Question: does BizTalk Server support the quoted-printable encoding method? If it does, what am I doing wrong? If it does not, what are my options in terms of a workaround?
For anyone else that may encounter this same issue, I thought I'd share the solution I ended up with.
Since the error was encountered during AS2 receive pipeline processing, naturally my solution was focussed around creating a custom receive pipeline component that does more or less the same than the out-of-the-box AS2 decoder component, but with support for the quoted-printable encoding method:
1. Decode and decrypt the CMS/PKCS#7 data envelope
This is actually the easiest step with only 5 lines of code:
EnvelopedCms envelopedCms = new EnvelopedCms();
envelopedCms.Decode(encryptedData);
envelopedCms.Decrypt();
byte[] decryptedData = envelopedCms.Encode();
string decryptedMessageString = Encoding.ASCII.GetString(decryptedData);
-encryptedData is a byte-array instantiated from the body-part data stream of the AS2 message received bythe HTTP adapter.
-The Decrypt method automatically searches the user and computer certificate stores for the appropriate certificate private key and uses this to decrypt the AS2 payload. For more information on the `EnvelopedCms' class follow this link.
2. Convert any quoted-printable content in the payload to normal UTF-8 text
First we have to get the MIME boundary name from the content type string at the beginning of the decrypted payload:
int firstBlankLineInMessage = decryptedMessageString.IndexOf(Environment.NewLine + Environment.NewLine);
string contentType = decryptedMessageString.Substring(0, firstBlankLineInMessage);
Regex boundaryRegex = new Regex("boundary=\"(?<boundary>.*)\"");
Match boundaryMatch = boundaryRegex.Match(contentType);
if (!boundaryMatch.Success)
throw new Exception("Failed to get boundary name from content type");
string boundary = "--" + boundaryMatch.Groups["boundary"].Value;
Then we split the envelope and re-merge without the content-type header part:
string[] messageParts = decryptedMessageString.Split(new string[] {boundary}, StringSplitOptions.RemoveEmptyEntries);
string signedMessageString = boundary + messageParts[1] + boundary + messageParts[2] + boundary + "--\r\n";
Next we get the `Content-Transfer-Encoding' value in the MIME body-part header:
int firstBlankLineInBodyPart = messageParts[1].IndexOf(Environment.NewLine + Environment.NewLine);
string partHeaders = messageParts[1].Substring(0, firstBlankLineInBodyPart);
Regex cteRegex = new Regex("Content-Transfer-Encoding: (?<cte>.*)");
Match cteMatch = cteRegex.Match(partHeaders);
if (!cteMatch.Success)
throw new Exception("Failed to get CTE from body part headers");
string cte = cteMatch.Groups["cte"].Value;
string payload = messageParts[1].Substring(firstBlankLineInBodyPart).Trim();
And finally we check the CTE and decode if neccessary:
string payload = messageParts[1].Substring(firstBlankLineInBodyPart).Trim();
if (cte == "quoted-printable")
{
// Get charset
Regex charsetRegex = new Regex("Content-Type: .*charset=(?<charset>.*)");
Match charsetMatch = charsetRegex.Match(partHeaders);
if (!charsetMatch.Success)
throw new Exception("Failed to get charset from body part headers");
string charset = charsetMatch.Groups["charset"].Value;
QuotedPrintableDecode(payload, charset);
}
Note: There are many different implementations out there for decoding QP, including a .NET implementation that has (reportedly) been found buggy by some users. I decided to use this implementation shared by Gonzalo.
3. Update the Content-Type HTTP header and BizTalk message body-part stream
string httpHeaders = objHttpHeaders.ToString().Replace("Content-Type: application/pkcs7-mime; name=\"smime.p7m\"; smime-type=enveloped-data", "Content-Type: application/xml");
inMessage.Context.Write("InboundHttpHeaders", "http://schemas.microsoft.com/BizTalk/2003/http-properties", httpHeaders);
MemoryStream payloadStream = new MemoryStream(Encoding.UTF8.GetBytes(payload));
payloadStream.Seek(0, SeekOrigin.Begin);
pipelineContext.ResourceTracker.AddResource(payloadStream);
inMessage.BodyPart.Data = payloadStream;
-pipelineContext is the IPipelineContext variable passed to the Execute method of the custom pipeline component
-inMessage is the IBaseMessage variable passed to the Execute method
Last Thoughts
The code above can still be improved in a number of ways:
Checking HTTP headers for encryption before attempting to decrypt
Re-encrypting payload before passing message to AS2 disassembler component (if required by BizTalk party configuration)
Adding support for compression
If you'd like a copy of the source code drop me a message and I'll see about upping it to an online repo.
I had ticket opened with Microsoft BizTalk tech support on the issue. Their response is that
The quoted-printable encoding is not supported by MS BizTalk Server 2013R2" and most likely is not supported by MS BizTalk Server 2020
Related
The official documentation does not provide an example for any SDK's (including the Java SDK): https://learn.microsoft.com/en-us/graph/api/user-sendmail?view=graph-rest-1.0&tabs=java#example-4-send-a-new-message-using-mime-format. As there is no example, I have tried in vain to send the MIME content using the SDK (microsoft-graph 5.0.0):
Message sending = new Message();
ItemBody body = new ItemBody();
final String mimeMessageRFC822 = input.getMimeMessageRFC822();
body.content = Base64.getMimeEncoder().encodeToString(mimeMessageRFC822.getBytes());
sending.body = body;
GraphServiceClient service = getService(acHost, configuration);
service
.me()
.sendMail(UserSendMailParameterSet.newBuilder().withMessage(sending).withSaveToSentItems(true).build())
.buildRequest(new HeaderOption("Content-Type", "text/plain"))
.post();
The above code sets the request's content-type to text/plain, however the request body that is being sent is JSON (xxxxxx below is a placeholder for a valid Base64 encoded MIME content string).
{
"message":
{
"body":
{
"content": xxxxxx
}
},
"saveToSentItems": true
}
The response is a 404, stating:
GraphServiceException: Error code: ErrorMimeContentInvalidBase64String
Error message: Invalid base64 string for MIME content.
I can understand why it is responding with this error as the graph endpoint is parsing the text/plain content as base64 encoded MIME but finds the JSON structure instead. I have been on a video call with a Microsoft Graph support agent, and they have seen that my MIME content is valid. Sadly, they are not able to help with the Microsoft Graph Java SDK even though it is developed by Microsoft!
This suggests that we are not supposed to use the Java SDK at all for sending MIME formatted emails. Is this correct? Surely it can't be otherwise what is the point of a library that can receive MIME formatted emails but can't send them? Does anyone have a solution?
For now at least the solution is to send a CustomRequest with MIME content instead of using the fluent API provided by the Graph client.
final String encodedContent = Base64.getMimeEncoder().encodeToString(mimeMessageRFC822.getBytes());
CustomRequest<String> request = new CustomRequest<>(requestUrl, service, List.of(new HeaderOption("Content-Type", "text/plain")), String.class);
request.post(encodedContent);
A zapier web hook has been set up to catch JSON sent to it.
The issue is that if the JSON contains any non-standard characters, e.g. accented characters, the hook never catches the data (no error is displayed, it just doesn't log anything).
Id the catch hook is switched to a 'catch raw hook' then the data is received, but I then don't know how to transform the raw data into JSON for future steps. With the catch raw hook the data caught is e.g. as follows (with a special char ø in the name value):
raw_body
[{"id":2426,"name":"James Hømmett"}]
headers__http_host
hooks.zapier.com
headers__http_x_request_id
b8578a4455fea95c3287e939e304752c
headers__http_x_real_ip
[redacted IP address]
headers__http_x_forwarded_for
[redacted IP address]
headers__http_x_forwarded_host
hooks.zapier.com
headers__http_x_forwarded_port
443
headers__http_x_forwarded_proto
https
headers__http_x_scheme
https
headers__http_x_original_forwarded_for
[redacted IP address]
headers__content_length
559
headers__http_accept_encoding
gzip,deflate
headers__content_type
application/json; charset=utf-8
headers__http_user_agent
Apache-HttpClient/4.5.13 (Java/11.0.9.1)
As you can see charset=utf8 is specified in the content-type header.
The JSON validates with jsonlint.com
Any ideas?
If you're on a paid account, you can add a Code by Zapier step that returns JSON.parse(inputData.raw_body), so that the data is available in future steps.
But, not handling non-ascii characters is likely a bug, so it's worth reaching out to support if you haven't already: https://zapier.com/contact
Delphi 10 Seattle
IntraWeb 15.0.23
Chilkat Trial
Used Chilkat (Delphi DLL) PDF File Encoding to Base64 to Encode my file.
https://www.example-code.com/delphidll/base64_pdf.asp
While parsing it to the Chilkat Delphi DLL code to request remote signing I get the below mentioned response.
https://www.example-code.com/delphidll/docusign_request_signature_via_email.asp
Please see Response code while trying to send a file through the Docusign API. I have the PDF to base64 code working and vice versa.
Response Status Code = 400
Response Header:
Cache-Control: no-cache
Content-Length: 226
Content-Type: application/json; charset=utf-8
Date: Wed, 27 May 2020 12:12:16 GMT
Response Body:
{
"errorCode": "UNSPECIFIED_ERROR",
"message": "The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. "
}
Response Body End
Request Body:
{"emailSubject":"DocuSign REST API Quickstart Sample","emailBlurb":"Shows how to create and send an envelope from a document.","recipients":{"signers":[{"email":"123#gmail.com","name":"XYZ","recipientId":"1","routingOrder":"1"}]},"documents":[{"documentId":"1","name":"C:\Doe_John.pdf","documentBase64":"JVBERi0xLjQNJabpz8QNCjEgMCBvYmoNPDwvQ3JlYXRvcij+/wBNAGkAYwByAG8AcwBvAGYAdAAgAFcAbwByAGQAIAAtACAARABvAGMAdQBtAGUAbgB0ADEpL1Byb2R1Y2VyKP7/AFMAYwBhAG4AUwBvAGYAdAAgAFAARABGACAAQwByAGUAYQB0AGUAIQAgADUpL0NyZWF0aW9uRGF0ZShEOjIwMDkxMjIyMTUwOTMyLTA1JzAwJykvTW9kRGF0ZShEOjIwMDkxMjIyMTUwOTMzLTA1JzAwJykvQXV0aG9yKP7/AGUAcwBmAG8AeCkvVGl0bGUo/v8ATQBpAGMAcgBvAHMAbwBmAHQAIABXAG8AcgBkACAALQAgAEQAbwBjAHUAbQBlAG4AdAAxKT4+DWVuZG9iag0yIDAgb2JqDTw8L1R5cGUvQ2F0YWxvZy9QYWdlcyAzIDAgUi9QYWdlTW9kZS9Vc2VOb25lL091dGxpbmVzIDcgMCBSL01l驠"}],"status":"sent"}
Your error message is very helpful in this case. Check the validity of your Base64 string - Looks to me like a pipe and mandarin character sitting at the end of your encoded string.
{ "errorCode": "UNSPECIFIED_ERROR", "message": "The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. " }
I am trying to consume a REST service using TRestClient but I believe there is an issue with the boundary string for multipart content.
I am capturing the body of the request I am sending, and this is the content type header:
Content-Type: multipart/form-data; boundary=-------Embt-Boundary--07CC944C29DA577E
Then, this is the first section of the multipart form:
-----------Embt-Boundary--07CC944C29DA577E
Content-Disposition: form-data; name="file"; filename="ce.csv"
Content-Type: text/csv
And this is how it ends:
---------Embt-Boundary--07CC944C29DA577E--
I don't think this is an issue on the server, as even my proxy is not able to parse the body:
When I compare this same request vs postman, I notice that the starting and ending boundaries do not match!
Starting: -----------Embt-Boundary--07CC944C29DA577E
Ending: ---------Embt-Boundary--07CC944C29DA577E--
I found that the boundary generation is done in TMultipartFormData.GenerateBoundary() from System.Net.Mime:
When checking the starting and ending boundaries from postman, they match, so I am almost sure this is the issue. I don't think it is related to my code, but let me know if you need it.
I have a COBOL batch program where I am able to send mail notification to an ID once my job is complete however, I also want to add an attachment in the mail of the processed file.
The following code attaches another mail as an attachment.
HELO SANTAANA
MAIL FROM:<abc#somting.com>
RCPT TO:<abc#something.com>
DATA
From: LandT P2P - LO <abc#something.com>
To: abc#something.com
Subject: File processed - Price_Change_10-27-15 07-08-44
MIME-VERSION: 1.0
CONTENT-TYPE: MULTIPART/MIXED;name="Price_Change_10-27-15.csv"
CONTENT-DISPOSITION: ATTACHMENT;
FILENAME="Price_Change_10-27-15 07-08-44.csv"
Note: I have also tried using SMTP and still does not work
Here is the sample of the mail i receive on running this code.
If you are generating the text of the email from within your Cobol program, which it sounds like, you would need to append another section, specify the Content-Type and Content-Disposition, filename and encoding, and then follow it with the properly encoded data, similar to this:
Content-Type: application/xml; name="Price_Change_10-27-15 07-08-44"
Content-Disposition: attachment; filename="Price_Change_10-27-15 07-08-44"
Content-Transfer-Encoding: base64
UEsDBBQABgAIAAAAIQDfrfoCnAEAAEcGAAATAAgCW0NvbnRlbnRfVHlwZXNdLnhtbCCiBAIooAAC
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
... and so on ...
I did notice that you had the content of those tags in upper case, that might be a problem. RFC1341 specifies those as "multipart/mixed" and "attachment" and so on. It is possible that your lack of mixed case is messing you up.
CONTENT-TYPE: MULTIPART/MIXED;name="Price_Change_10-27-15.csv"
CONTENT-DISPOSITION: ATTACHMENT;
FILENAME
Even easier than generating your own, have you looked into the excellent XMITIP package by Lyonel B. Dyck, it manages all that for you and you write a few config arms to control it, and you can easily call it from a Cobol program just like any other Rexx. Or you could add it to the end of your job stream as a separate step and make the task really easy.