http response got truncated in docker - docker

I've built a http server using netty. Everything is fine when it's running in my mac, but when I run it in a docker image, the http response always get truncated when great than 460k.
What's the problem will be? Please help.

Do you use aggregator to aggregate the http response or not? Take a look at the source code of HttpObjectDecoder. It will chunk bigger http response no matter if the http message itself is transfer coding or not.
The default maxChunk size is 8k.And even readable bytes is enough, it will chunk it. see the code below:
` case READ_FIXED_LENGTH_CONTENT: {
int readLimit = actualReadableBytes();
// Check if the buffer is readable first as we use the readable byte count
// to create the HttpChunk. This is needed as otherwise we may end up with
// create a HttpChunk instance that contains an empty buffer and so is
// handled like it is the last HttpChunk.
//
// See https://github.com/netty/netty/issues/433
if (readLimit == 0) {
return;
}
int toRead = Math.min(readLimit, maxChunkSize);
if (toRead > chunkSize) {
toRead = (int) chunkSize;
}
ByteBuf content = readBytes(ctx.alloc(), buffer, toRead);
chunkSize -= toRead;
`

Related

Dart TCP socket concatenates all 'write' sync calls as a single packet

I'm trying to send multiple packets at once to a server, but the socket keeps "merging" all sync calls to write as a single call, I did a minimal reproducible example:
import 'dart:io';
void main() async {
// <Server-side> Create server in the local network at port <any available port>.
final ServerSocket server =
await ServerSocket.bind(InternetAddress.anyIPv4, 0);
server.listen((Socket client) {
int i = 1;
client.map(String.fromCharCodes).listen((String message) {
print('Got a new message (${i++}): $message');
});
});
// <Client-side> Connects to the server.
final Socket socket = await Socket.connect('localhost', server.port);
socket.write('Hi World');
socket.write('Hello World');
}
The result is:
> dart example.dart
> Got a new message (1): Hi WorldHello World
What I expect is:
> dart example.dart
> Got a new message (1): Hi World
> Got a new message (2): Hello World
Unfortunately dart.dev doesn't support dart:io library, so you need to run in your machine to see it working.
But in summary:
It creates a new tcp server at a random port.
Then creates a socket that connects to the previous created server.
The socket makes 2 synchronous calls to the write method.
The server only receives 1 call, which is the 2 messages concatenated.
Do we have some way to receive each synchronous write call in the server as separated packets instead buffering all sync calls into a single packet?
What I've already tried:
Using socket.setOption(SocketOption.tcpNoDelay, true); right after Socket.connect instantiation, this does modify the result:
final Socket socket = await Socket.connect('localhost', server.port);
socket.setOption(SocketOption.tcpNoDelay, true);
// ...
Using socket.add('Hi World'.codeUnits); instead of socket.write(...), also does not modify the result as expected, because write(...) seems to be just a short version add(...):
socket.add('Hi World'.codeUnits);
socket.add('Hello World'.codeUnits);
Side note:
Adding an async delay to avoid calling write synchronously:
socket.add('Hi World'.codeUnits);
await Future<void>.delayed(const Duration(milliseconds: 100));
socket.add('Hello World'.codeUnits);
make it works, but I am pretty sure this is not the right solution, and this isn't what I wanted.
Environment:
Dart SDK version: 2.18.4 (stable) (Tue Nov 1 15:15:07 2022 +0000) on "windows_x64"
This is a Dart-only environment, there is no Flutter attached to the workspace.
As Jeremy said:
Programmers coding directly to the TCP API have to implement this logic themselves (e.g. by prepending a fixed-length message-byte-count field to each of their application-level messages, and adding logic to the receiving program to parse these byte-count fields, read in that many additional bytes, and then present those bytes together to the next level of logic).
So I chose to:
Prefix each message with a - and suffix with ..
Use base64 to encode the real message to avoid conflict between the message and the previously defined separators.
And using this approach, I got this implementation:
// Send packets:
socket.write('-${base64Encode("Hi World".codeUnits)}.');
socket.write('-${base64Encode("Hello World".codeUnits)}.');
And to parse the packets:
// Cache the previous parsed packet data.
String parsed = '';
void _handleCompletePacket(String rawPacket) {
// Decode the original message from base64 using [base64Decode].
// And convert the [List<int>] to [String].
final String message = String.fromCharCodes(base64Decode(rawPacket));
print(message);
}
void _handleServerPacket(List<int> rawPacket) {
final String packet = String.fromCharCodes(rawPacket);
final String next = parsed + packet;
final List<String> items = <String>[];
final List<String> tokens = next.split('');
for (int i = 0; i < tokens.length; i++) {
final String char = tokens[i];
if (char == '-') {
if (items.isNotEmpty) {
// malformatted packet.
items.clear();
continue;
}
items.add('');
continue;
} else if (char == '.') {
if (items.isEmpty) {
// malformatted packet.
items.clear();
continue;
}
_handleCompletePacket(items.removeLast());
continue;
} else {
if (items.isEmpty) {
// malformatted packet.
items.clear();
continue;
}
items.last = items.last + char;
continue;
}
}
if (items.isNotEmpty) {
// the last data of this packet was left incomplete.
// cache it to complete with the next packet.
parsed = items.last;
}
}
client.listen(_handleServerPacket);
There are certainly more optimized solutions/approaches, but I got this just for chatting messages within [100-500] characters, so that's fine for now.

Flush data added to websocket

I'm writing a speed test, but i'm having trouble on the client side for uploading.
I have a the following setup, which basically continues to write data into the socket while a condition is true, and then closes the socket:
var ws = await createWebSocket(sb.serverAddress, sb.authToken);
while (condition) {
var bytes = generateRandomBytes(_BUFFER_SIZE_BYTES);
ws.add(bytes);
print('added');
var megabits = (bytes.length * 8) / 1000000;
channel.sink.add(megabits);
}
await ws.close();
My problem is that I can't work out how to wait for the bytes to be accepted by the underlying buffer. Even if I set _BUFFER_SIZE_BYTES to an huge size it still loops at break neck speed printing out added, where I really want to wait until all the bytes are accepted by the send buffer (having been accepted by the server) before adding a new list of bytes.
With an http post request you can do: await postReq.flush();, but I don't see any such method for web sockets.
Ok so I think I have a reasonable solution to this problem.
Client side has to wait for a response from the server before sending more bytes:
var bytes = generateRandomBytes(_CHUNK_SIZE_BYTES);
ws.listen((data) async {
ws.add(bytes);
var megabits = (bytes.length * 8) / 1000000;
channel.sink.add(megabits);
}
});
Server (Go) sends a message to the client signalling that it can send a chunk, and then reads the entire response from the client, before signalling to the client that it is ready to accept another one:
for start := time.Now(); time.Since(start) < time.Second*maxDuration; {
err := conn.WriteMessage(websocket.TextMessage, []byte("next"))
if err != nil {
break
}
// will get an error if try writing to closed socket
_, bytes, err := conn.ReadMessage()
if err != nil {
fmt.Println(err)
break
}
fmt.Println(len(bytes))
}
I think this solution is ok. I've set the chunk size to 10Mb which seems to work ok. Let me know if anyone has a better idea.

Solve issue POSTING to webhook for IFTTT from Arduino MKR1010

I am aiming to make a post request to trigger a IFTTT webhook action. I am using the MKR1010 board. I am able to connect to the network and turn the connected LED on and off using the cloud integration.
The code is as follows, but doesn't trigger the web hook. I can manually paste the web address in a browser and this does trigger the web hook. When the code is posted it returns a 400 bad request error.
The key has been replaced in the below code with a dummy value.
Does anybody know why this is not triggering the web hook? / Can you explain why the post request is being rejected by the server? I don't even really need to read the response from the server as long as it is sent.
Thank you
// ArduinoHttpClient - Version: Latest
#include <ArduinoHttpClient.h>
#include "thingProperties.h"
#define LED_PIN 13
#define BTN1 6
char serverAddress[] = "maker.ifttt.com"; // server address
int port = 443;
WiFiClient wifi;
HttpClient client = HttpClient(wifi, serverAddress, port);
// variables will change:
int btnState = 0; // variable for reading the pushbutton status
int btnPrevState = 0;
void setup() {
// Initialize serial and wait for port to open:
Serial.begin(9600);
// This delay gives the chance to wait for a Serial Monitor without blocking if none is found
delay(1500);
// Defined in thingProperties.h
initProperties();
// Connect to Arduino IoT Cloud
ArduinoCloud.begin(ArduinoIoTPreferredConnection);
/*
The following function allows you to obtain more information
related to the state of network and IoT Cloud connection and errors
the higher number the more granular information you’ll get.
The default is 0 (only errors).
Maximum is 4
*/
setDebugMessageLevel(2);
ArduinoCloud.printDebugInfo();
// setup the board devices
pinMode(LED_PIN, OUTPUT);
pinMode(BTN1, INPUT);
}
void loop() {
ArduinoCloud.update();
// Your code here
// read the state of the pushbutton value:
btnState = digitalRead(BTN1);
if (btnPrevState == 0 && btnState == 1) {
led2 = !led2;
postrequest();
}
digitalWrite(LED_PIN, led2);
btnPrevState = btnState;
}
void onLed1Change() {
// Do something
digitalWrite(LED_PIN, led1);
//Serial.print("The light is ");
if (led1) {
Serial.println("The light is ON");
} else {
// Serial.println("OFF");
}
}
void onLed2Change() {
// Do something
digitalWrite(LED_PIN, led2);
}
void postrequest() {
// String("POST /trigger/btn1press/with/key/mykeyhere")
Serial.println("making POST request");
String contentType = "/trigger/btn1press/with/key";
String postData = "mykeyhere";
client.post("/", contentType, postData);
// read the status code and body of the response
int statusCode = client.responseStatusCode();
String response = client.responseBody();
Serial.print("Status code: ");
Serial.println(statusCode);
Serial.print("Response: ");
Serial.println(response);
Serial.println("Wait five seconds");
delay(5000);
}
Why do you want to make a POST request and send the key in the POST body? The browser sends a GET request. It would be
client.get("/trigger/btn1press/with/key/mykeyhere");
In HttpClient post() the first parameter is 'path', the second parameter is contentType (for example "text/plain") and the third parameter is the body of the HTTP POST request.
So your post should look like
client.post("/trigger/btn1press/with/key/mykeyhere", contentType, postData);

tweetinvi Chunked Uploads for large uploads

I am trying to file upload with tweetinvi.File upload work for image but same code does not work for video(large video more than 20 mb)
I asked here but now answear
TweetInvi Large Video Upload Failing With Null Reference
so I look for another solution.There is tweetinvi chunked uploads
I coded this but it does not work,it does not give error but it does not work
if (file.ContentType.Contains("video"))//video
{
var video1 = System.IO.File.ReadAllBytes(path);
var chunk = Upload.CreateChunkedUploader(); //Create an instance of the ChunkedUploader class (I believe this is the only way to get this object)
using (FileStream fs = System.IO.File.OpenRead(path))
{
chunk.Init("video/mp4", (int)fs.Length); //Important! When initialized correctly, your "chunk" object will now have a type long "MediaId"
byte[] buffer = new byte[video1.Length]; //Your chunk MUST be 5MB or less or else the Append function will fail silently.
int bytesRead = 0;
while ((bytesRead = fs.Read(buffer, 0, buffer.Length)) > 0)
{
byte[] copy = new byte[bytesRead];
Buffer.BlockCopy(buffer, 0, copy, 0, bytesRead);
TimeSpan s = new TimeSpan();
chunk.Append(copy, chunk.NextSegmentIndex.ToString()); //The library says the NextSegment Parameter is optional, however I wasn't able to get it to work if I left it out.
}
}
var video = chunk.Complete(); //This tells the API that we are done uploading.
listMedia.Add(video);
}
I wanted to say that after working on your bug I was able to identify the problem you were encountering.
The issue is that you were not specifying the media_category of the upload.
In addition to this you need to wait for the media to be processed by Twitter.
In Tweetinvi 2.1, the process should be made easier. Please use the following code :
var binary = File.ReadAllBytes(#"video_path");
var media = Upload.UploadVideo(binary);
// The upload is completed but it does not mean it succeeded!
if (!media.HasBeenUploaded)
{
// Something went wrong during the upload.
// Please retry or check the video type/settings.
return;
}
// Just wait for Twitter to have processed the upload (RECOMMENDED)
Upload.WaitForMediaProcessingToGetAllMetadata(media);
// Now the media is ready to be used in a Tweet
var tweet = Tweet.PublishTweet("hello", new PublishTweetOptionalParameters
{
Medias = { media }
});
You can read more about the upload in the updated documentation : https://github.com/linvi/tweetinvi/wiki/Upload#upload-status-video
Finally please note that further improvements of upload are planned for version 2.2 and 2.3.
Have a great day and thank you for reporting this problem,
Cheers
Linvi

Getting error when uploading video using Tweetinvi

I am trying to upload files larger than 5MB but less than 15MB. In this case its 10MB... sample video from sample-videos.com
I am using Tweetinvi, and it works great with files less than 5MB, but fails on chunked uploads. I've tried the easy and the hard way.
Easy way:
var video = File.ReadAllBytes(#"D:\Projects\SampleVideo_1280x720_10mb.mp4");
var media = Upload.UploadVideo(video); // Error here... Invalid Content
var tweet = user.PublishTweet(message, new PublishTweetOptionalParameters
{
Medias = { media }
});
I've pulled in the Tweetinvi solution from Git (currently 0.9.13.0 repository here) and see that the above is getting an error "Invalid Content" when calling Upload.UploadVideo(...). It appears to fail on the command FINALIZE.
Tried the hard way:
using (var fileStream = File.OpenRead(#"D:\Projects\SampleVideo_1280x720_10mb.mp4"))
{
var initSucceeded = uploader.Init("video/mp4", (int)fileStream.Length);
byte[] buffer = new byte[4900000]; //Your chunk MUST be 5MB or less or else the Append function will fail silently.
int bytesRead = 0;
while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) > 0)
{
byte[] copy = new byte[bytesRead];
Buffer.BlockCopy(buffer, 0, copy, 0, bytesRead);
var appendResult = uploader.Append(new ChunkUploadAppendParameters(copy, "video/mp4", null) { SegmentIndex = uploader.NextSegmentIndex });
}
var video = uploader.Complete(); // Fails here... Returned error: Segments do not add up to provided total file size
var tweet = user.PublishTweet(message, new PublishTweetOptionalParameters()
{
//Medias = { video }
MediaIds = { video.MediaId.Value }
});
}
The above fails on upload.Complete() with the Twitter API returning "Segments do not add up to provided total file size"
What am I missing?
TIA
I think the problem you have is with the video file. The video seems to be using a 6 channels audio and Twitter Public Upload API only allow developers to upload video with mono or stereo audio.
Source : https://dev.twitter.com/rest/media/uploading-media
I am no expert in video properties so feel free to prove me wrong if I am.
The above fails on upload.Complete() with the Twitter API returning "Segments do not add up to provided total file size"
This error means that you are not actually sending all the byte that you promised Twitter. During the INIT, you tell Twitter the size of your media, if what it receives in the combined APPEND is not equals to the value you specified in the INIT, the error you described is thrown.
PS : I have tried using a 2 channels 14.8 MB mp4 and it works properly. var media = Upload.UploadVideo(binary);

Resources