I have problem with FrameDecoder of netty as following.
packet is fragmented in to {n} frames then FrameDecoder processing correctly.
If many packets are composed to one frame and send to server(when client sending small packet data continuously), then FrameDecoder only reads the first packet. The remaing is ignored.
Can i get the remaining data to be continued executed by FrameDecoder?
public class BinaryFrameDecoder extends FrameDecoder {
#Override
protected Object decode(ChannelHandlerContext ctx, Channel channel, ChannelBuffer buffer){
if(buffer.readableBytes() < 2){
return null;
}
int length = buffer.getShort(buffer.readerIndex());
if(buffer.readableBytes() < length + 2){
return null;
}
buffer.skipBytes(2);
return buffer;
}
}
It should work if you replace:
return buffer;
with:
return buffer.readBytes(length);
because it is very likely that buffer contains more than one message.
Related
I am creating a batch data streamer in apache ignite, and need to control what happening after data receive.
My batch has a structure:
public class Batch implements Binarylizable, Serializable {
private String eventKey;
private byte[] bytes;
etc..
Then i trying to stream my data:
try (IgniteDataStreamer<Integer, Batch> streamer = serviceGrid.getIgnite().dataStreamer(cacheName);
StreamBatcher batcher = StreamBatcherFactory.create(event) ){
streamer.receiver(StreamTransformer.from(new BatchDataProcessor(event)));
streamer.autoFlushFrequency(1000);
streamer.allowOverwrite(true);
statusService.updateStatus(event.getKey(), StatusType.EXECUTING);
int counter = 0;
Batch batch = null;
IgniteFuture<?> future = null;
while ((batch = batcher.batch()) != null) {
future = streamer.addData(counter++, batch);
}
Object getted = future.get();
Just for test use lets get only the last future, and try to analyze this object. In the code above I'm using BatchDataProcessor, that look like this:
public class BatchDataProcessor implements CacheEntryProcessor<Integer, Batch, Object> {
private final Event event;
private final String eventKey;
public BatchDataProcessor(Event event) {
this.event = event;
this.eventKey = event.getKey();
}
#Override
public Object process(MutableEntry<Integer, Batch> mutableEntry, Object... objects) throws EntryProcessorException {
Node node = NodeIgniter.node(Ignition.localIgnite().cluster().localNode().id());
ServiceGridContainer container = (ServiceGridContainer) node.getEnvironmentContainer().getContainerObject(ServiceGridContainer.class);
ProcessMarshaller marshaller = (ProcessMarshaller) container.getService(ProcessMarshaller.class);
LocalProcess localProcess = marshaller.intoProccessing(event.getLambdaExecutionKey());
try {
localProcess.addBatch(mutableEntry);
} catch (IOException e) {
e.printStackTrace();
} finally {
return new String("111");
}
}
}
So after localProcess.addBatch(mutableEntry) I want to send back an information about the status of this particular batch, so I think that I should do this in IgniteFuture object, but I don't find any information how to control the future object that's received in addData function.
Can anybody help with understanding, where can I control future that receives in addData function or some other way to realize a callback to streamed batch?
When you do StreamTransformer.from(), you forfeit the result of your BatchDataProcessor, because
for (Map.Entry<K, V> entry : entries)
cache.invoke(entry.getKey(), this, entry.getValue());
// ^ result of cache.invoke() is discarded here
DataStreamer is for one-directional streaming of data. It is not supposed to return values as far as I know.
If you depend on the result of cache.invoke(), I recommend calling it directly instead of relying on DataStreamer.
BTW, be careful with fut.get(). You should do dataStreamer.flush() first, or DataStreamer's futures will wait indefinitely.
I read the "when to use parallel stream?" by DougLea et.al http://gee.cs.oswego.edu/dl/html/StreamParallelGuidance.html.
I wonder did any one had a guide lines(do's/ don't dos)/ observations which felt them that old way of coding is better in some cases than sequential stream?
I found one here https://jaxenter.com/java-performance-tutorial-how-fast-are-the-java-8-streams-118830.html
I know it's a abstract question but it will be helpful if somebody can share their experience in performance of seq stream vs java 7 way
I've done this just a few days ago; we had to sum a very large array and was wondering what would be the fastest way to do it - so I measured (don't guess; I've used jmh):
#State(Scope.Thread)
public static class Holder {
#Param({ "1000", "10000", "50000", "100000", "1000000" })
public int howManyEntries;
int array[] = null;
#Setup
public void setUp() {
array = new int[howManyEntries];
for (int i = 0; i < howManyEntries; ++i) {
array[i] = i;
}
}
#TearDown
public void tearDown() {
array = null;
}
}
#Fork(1)
#Benchmark
public int iterative(Holder holder) {
int total = 0;
for (int i = 0; i < holder.howManyEntries; ++i) {
total += holder.array[i];
}
return total;
}
#Fork(1)
#Benchmark
public int stream(Holder holder) {
return Arrays.stream(holder.array).sum();
}
#Fork(1)
#Benchmark
public int streamParallel(Holder holder) {
return Arrays.stream(holder.array).parallel().sum();
}
The winner is always the old style java-7 way.
// 1000=[iterative, stream, streamParallel]
// 10000=[iterative, stream, streamParallel]
// 50000=[iterative, stream, streamParallel]
// 100000=[iterative, stream, streamParallel]
// 1000000=[iterative, stream, streamParallel]
Even for 1 million elements. But the result differs in up to 60 ms - if that bites you or not is entirely your choice.
Streams are not meant for speed, they will not replace the old style, neither do they want to - it could add extra visibility to your code for example.
I have seen lots of DataSnap examples in Delphi, but fewer in C++ Builder, and have not figured out how to specify that a TStream should be returned to the calling client.
I am using a simple configuration, similar to the tutorials I have seen. An example server method is:
System::UnicodeString GetData(int PatientID, int& count, TStream* stream);
I have no trouble calling that method from my client. Because count is passed as a reference, the DataSnap server knows to send it back to the client. Generate Client Classes, on the TSQLConnection in the client, connects to the server, and generates the following:
System::UnicodeString __fastcall TServerMethods1Client::GetData(int PatientID, int &count, TStream* stream)
{
if (FGetDataCommand == NULL)
{
FGetDataCommand = FDBXConnection->CreateCommand();
FGetDataCommand->CommandType = TDBXCommandTypes_DSServerMethod;
FGetDataCommand->Text = "TServerMethods1.GetData";
FGetDataCommand->Prepare();
}
FGetDataCommand->Parameters->Parameter[0]->Value->SetInt32(PatientID);
FGetDataCommand->Parameters->Parameter[1]->Value->SetInt32(count);
FGetDataCommand->Parameters->Parameter[2]->Value->SetStream(stream, FInstanceOwner);
FGetDataCommand->ExecuteUpdate();
count = FGetDataCommand->Parameters->Parameter[1]->Value->GetInt32();
System::UnicodeString result = FGetDataCommand->Parameters->Parameter[3]->Value->GetWideString();
return result;
}
One can see that the generated code is setting the count from the returned parameter, indicating that the server is sending it back. However, the stream is only sent to the server, and not set back on the client.
In Delphi, I would use var to indicate that the reference should be passed back to the caller. However, using a reference on TStream does not work, either.
For this definition:
System::UnicodeString GetData(int PatientID, int& count, TStream& stream);
I get this generated code:
System::UnicodeString __fastcall TServerMethods1Client::GetData(int PatientID, int &count, TStream* &stream)
{
if (FGetDataCommand == NULL)
{
FGetDataCommand = FDBXConnection->CreateCommand();
FGetDataCommand->CommandType = TDBXCommandTypes_DSServerMethod;
FGetDataCommand->Text = "TServerMethods1.GetData";
FGetDataCommand->Prepare();
}
FGetDataCommand->Parameters->Parameter[0]->Value->SetInt32(PatientID);
FGetDataCommand->Parameters->Parameter[1]->Value->SetInt32(count);
FGetDataCommand->Parameters->Parameter[2]->Value->SetStream(stream, FInstanceOwner);
FGetDataCommand->ExecuteUpdate();
count = FGetDataCommand->Parameters->Parameter[1]->Value->GetInt32();
stream = FGetDataCommand->Parameters->Parameter[2]->Value->GetStream(FInstanceOwner);
System::UnicodeString result = FGetDataCommand->Parameters->Parameter[3]->Value->GetWideString();
return result;
}
Which throws an access violation in the ExecuteUpdate() call.
Is there a way I can pass a pointer to the server method and mark it in some way that the stream should be passed back to the calling client?
Remy's response was correct.
The server method is
System::UnicodeString TServerMethods1::GetData(int PatientID, int& count,
TStream*& stream) {
count = 10;
String newSimpleString = "New Simple String";
TByteDynArray theBytes;
theBytes.Length = newSimpleString.Length();
for (int i = 0; i < newSimpleString.Length(); i++) {
theBytes[i] = newSimpleString[i + 1];
}
TDBXBytesStream* newStream =
new TDBXBytesStream(theBytes, newSimpleString.Length());
stream = newStream;
return "StringResult";
}
and the client method is
void __fastcall TForm1::Button2Click(TObject *Sender) {
int count = 1;
TStringStream* stringStream = new TStringStream(String("Passed In"));
TStream* str = stringStream;
String result = ClientModule1->ServerMethods1Client->GetData(1, count, str);
int len = str->Size;
ShowMessage("Result is: '" + result + "' and " + String(count) +
", and stream is " + String(len) + " bytes long");
}
With FInstanceOwner at true (the default as generated), the stream passed in as a client method gets freed the next time the method is called.
On the server, the newly created TDBXBytesStream gets freed the next time the server method is called.
TDBXBytesStream is not a fully functional stream - for example, I could not get CopyFrom to copy from a TStringStream (for example), which is why I use the TByteDynArray.
Also, I could not modify the stream passed in to the server method by writing to it, which is why I needed to assign a new TDBXBytesStream to the passed in reference.
Thanks, Remy, for helping me with this.
in my project i am using popupscreen with GaugeField for http request.Currently we are just incrementing the value of gaugefield with fixed rate and after http response we just remove the popupscreen. so some times http request is completed when gauge field is in 40% or 60%.
But i want to synchronize gaugefield value with http request/responses. it means that popupscreen will always remove at 100%.
I don't have the code in front of me, but I something similar in a project several years ago.
I wrote a subclass of InputStream that wrapped around the InputStream object I got back from openInputStream(), reimplementing all the read() methods so they would increment a counter with the number of bytes read. Whenever the counter reached a certain threshold, it would update a GaugeField object that was passed into the subclass's constructor.
So your subclass would look something like this:
public GaugedInputStream extends InputStream
{
private InputStream _inputStream = null;
private GaugeField _gaugeField = null;
private int _counter = 0;
private int _threshold = 0;
public void GaugedInputStream(InputStream inputStream, GaugeField gaugeField)
{
_inputStream = inputStream;
_gaugeField = gaugeField;
... other constructor stuff ...
}
public int read()
{
int byte = _inputStream.read();
increment(1);
return byte;
}
public int read(byte[] b)
{
int bytes = _inputStream.read(b);
increment(bytes);
return bytes;
}
public int read(byte[] b, int off, int len)
{
int bytes = _inputStream.read(b, off, len);
increment(bytes);
return bytes;
}
... override other InputStream methods here ...
private void increment(int bytes)
{
_counter = _counter + bytes;
_threshold = _threshold + bytes;
updateGaugeIfNeeded();
}
private void updateGaugeIfNeeded()
{
if (_threshold > 100)
{
updateGauge();
_threshold = 0;
}
}
private void updateGauge()
{
... code to update the gauge ...
}
}
I'm leaving out a lot of the guts here, but I hope this sets you in the right direction.
I was inspecting the oflaDemo from Red5.
I was wondering how I could change the oflaDemo so that I could stream a certain flv file to all the subscribers...
/** {#inheritDoc} */
#Override
public boolean appConnect(IConnection conn, Object[] params) {
// Trigger calling of "onBWDone", required for some FLV players
measureBandwidth(conn);
if (conn instanceof IStreamCapableConnection) {
IStreamCapableConnection streamConn = (IStreamCapableConnection) conn;
SimpleConnectionBWConfig bwConfig = new SimpleConnectionBWConfig();
bwConfig.getChannelBandwidth()[IBandwidthConfigure.OVERALL_CHANNEL] =
1024 * 1024;
bwConfig.getChannelInitialBurst()[IBandwidthConfigure.OVERALL_CHANNEL] =
128 * 1024;
streamConn.setBandwidthConfigure(bwConfig);
}
serverStream = StreamUtils.createServerStream(appScope, "live0");
SimplePlayItem item = new SimplePlayItem();
item.setStart(0);
item.setLength(10000);
item.setName("IronMan");
serverStream.addItem(item);
item = new SimplePlayItem();
item.setStart(20000);
item.setLength(10000);
item.setName("DarkKnight");
serverStream.addItem(item);
serverStream.start();
serverStream.setRepeat(true);
return super.appConnect(conn, params);
}
The result of the code above is just a blank screen for all subscribers...
Does anyone of you have any idea?
thanks in advance!
This would be easier to accomplish on the client side and if you didn't want to hard-code the stream name, you could pass its name as a flash var.
I got the same problem. I found that blank screen is because of what line
serverStream.setRepeat(true);
if you set repeat to false all works fine
UPD:
I think that this is red5 bug. I made a hack to repeat continiously one flv. I added custom play list controller:
public class MyPlayListController implements IPlaylistController {
#Override
public int nextItem(IPlaylist playlist, int itemIndex) {
return 0;
}
#Override
public int previousItem(IPlaylist playlist, int itemIndex) {
return 0;
}
}