I have a simple TCP client in dart:
import 'dart:io';
void main() {
const sendData = "\$I,Z,0.5,5,0*\r\n";
final socket = Socket.connect("192.168.1.100", int.parse("8008"))
.timeout(Duration(seconds: 5))
.whenComplete(() {
print("Connected!");
}).catchError((_) {
print("Error!");
});
socket.then((soc) {
soc.write(sendData);
soc.listen((data) {
print(String.fromCharCodes(data).trim);
});
});
}
This program sends a special message to server and after that, the server sends back a bunch of data every 10 ms. The output is as follows:
$I,1,250,0,206*$I,1,248,0,192*$I,1,246,0,178*$I,1,245,0,165*
$I,1,244,0,153*$I,1,244,0,141*$I,1,244,0,131*$I,1,245,0,121*
$I,1,246,0,113*$I,1,248,0,105*$I,1,250,0,98*
$I,1,253,0,92*$I,2,0,0,86*$I,2,4,0,82*$I,2,8,0,79*$I,2,12,0,76*
$I,2,18,0,74*$I,2,23,0,74*$I,2,29,0,74*$I,2,36,0,75*$I,2,42,0,77*$I,2,50,0,80*$I,2,58,0,84*$I,2,66,0,89*$I,2,74,0,94*
$I,2,83,0,101*$I,2,93,0,109*$I,2,103,0,117*$I,2,113,0,126*$I,2,124,0,136*$I,2,134,0,147*
$I,2,146,0,159*$I,2,157,0,171*$I,2,169,0,185*$I,2,182,0,199*$I,2,194,0,214*$I,2,207,0,230*$I,2,220,0,246*$I,2,233,1,8*$I,2,247,1,26*$I,3,5,1,44*$I,3,19,1,64*$I,3,33,1,84*
$I,3,48,1,105*$I,3,63,1,126*$I,3,77,1,148*$I,3,93,1,171*$I,3,108,1,194*$I,3,123,1,217*
$I,3,138,1,242*$I,3,154,2,10*$I,3,169,2,35*$I,3,185,2,61*$I,3,201,2,87*$I,3,216,2,113*$I,3,232,2,140*$I,3,248,2,167*$I,4,7,2,195*
$I,4,23,2,223*$I,4,39,2,251*$I,4,54,3,23*$I,4,70,3,51*
The server sends data in $I,1,250,0,206* format, i.e. starts with $ and ends with *. As one may note, several consecutive data packages are concatenated incorrectly.
Whenever I increase the interval, for example to 200 ms, everything is ok.
What should I do?
UPDATE
Besides the BrettSutton answer which is true, the contributers in dart github gave a more complete answer here.
I decided to parse the packet in Socket listen handler, split that and append it to a list. As I want to show the data on a chart, I reset the list after 100 data. However the data could be logged as well.
soc.listen((data) {
var av = data.length;
if (av != 0) {
var stList = String.fromCharCodes(data).trim().split("\$");
stList.forEach((str) {
if (str.isNotEmpty) {
var strS = str.split(",");
if (strS != null) y = parseData(strS);
sampleList.add(y);
}
});
print(sampleList);
if (sampleList.length > 100) {
sampleList.clear();
}
print("==========");
}
});
I'm trying to understand the correct way to use the Flume RpcClient in a multithreaded application. Information I have found so far indicates that the components are thread safe, but the example in the Flume documentation clouds the issue when it comes to error handling. This code:
public void sendDataToFlume(String data) {
// Create a Flume Event object that encapsulates the sample data
Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));
// Send the event
try {
client.append(event);
} catch (EventDeliveryException e) {
// clean up and recreate the client
client.close();
client = null;
client = RpcClientFactory.getDefaultInstance(hostname, port);
// Use the following method to create a thrift client (instead of the above line):
// this.client = RpcClientFactory.getThriftInstance(hostname, port);
}
}
If more then one thread calls this method, and the exception is thrown, then there will be a problem as multiple threads try and recreate the client in the exception handler.
Is the intent of the SDK that it should only be used by a single thread? Should this method be synchronized, as it appears to be in the log4jappender that is part of the Flume source? Should I put this code in its own worker and pass it events via a queue?
Does anyone have an example of RpcClient being used by more then one thread (included the error condition)?
Would I be better off using the "embedded agent"? Is that multithread friendly?
With the embedded agent, you get the same case except you don't know what to do:
try {
agent.put(event);
} catch (EventDeliveryException e) {
// ???
}
You could stop the agent, and restart it - but you would need a synchronized block (or a ReentrantReadWriteLock, to not block thread while "reading" the client field). But since I'm not a Flume expert, I can't tell you which one is better.
Example:
class MyClass {
private final ReentrantReadWriteLocklock;
private final Lock readLock;
private final Lock writeLock;
private RpcClient client;
private final String hostname;
private final Integer port;
// Constructor
MyClass(String hostname, Integer port) {
this.hostname = Objects.requireNonNull(hostname, "hostname");
this.port = Objects.requireNonNull(port, "port");
this.lock = new ReentrantReadWriteLock();
this.readLock = this.lock.readLock();
this.writeLock = this.lock.writeLock();
this.client = buildClient();
}
private RpcClient buildClient() {
return RpcClientFactory.getDefaultInstance(hostname, port);
}
public void sendDataToFlume(String data) {
// Create a Flume Event object that encapsulates the sample data
Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));
// Send the event
readLock.lock(); // lock for reading 'client'
try {
try {
client.append(event);
} catch (EventDeliveryException e) {
writeLock.lock(); // lock for reading/writing client
try {
// clean up and recreate the client
client.close();
client = null;
client = buildClient();
} finally {
writeLock.unlock();
}
}
} finally {
readLock.unlock();
}
}
}
Beside, the example will lose the event because it is not sent back. Some kind of loop + a max retry would probably do the trick:
int i = 0;
for (; i < maxRetry; ++i) {
try {
client.append(event);
break;
} catch (EventDeliveryException e) {
// clean up and recreate the client
client.close();
client = null;
client = RpcClientFactory.getDefaultInstance(hostname, port);
// Use the following method to create a thrift client (instead of the above line):
// this.client = RpcClientFactory.getThriftInstance(hostname, port);
}
}
if (i == maxRetry) {
logger.error("flume client is offline, loosing events {}", event);
}
That's the idea, but I don't think that should be the task of the user (eg: us), but an option in the client or the agent to store event that could not be processed due to such errors.
Background:
I'm attempting to add some level fault tolerance to an application that uses Apache Commons.net FTPSClient to transfer files. If the connection between the client and server fails, I'd like to capture the produced exception/return code, log the details, and attempt to reconnect/retry the transfer.
What works:
The retrieveFile() method. If the connection fails, (i.e. I disable the server's public interface), I receive a CopyStreamException caused by a SocketTimeoutException after the amount of time I specified as the timeout.
What doesn't work:
The storeFile() method. If I initiate a transfer via storeFile() and disable the server's public interface, the storeFile() method blocks/hangs indefinitely with out throwing any exceptions.
Here is a simple app that hangs if the connection is terminated:
public class SmallTest {
private static Logger log = Logger.getLogger(SmallTest.class);
/**
* #param args
* #throws IOException
*/
public static void main(String[] args) throws IOException {
FTPSClient client = new FTPSClient(true);
FTPSCredentials creds = new FTPSCredentials("host", "usr", "pass",
"/keystore/ftpclient.jks", "pass",
"/keystore/rootca.jks");
String file = "/file/jdk-7u21-linux-x64.rpm";
String destinationFile = "/jdk-7u21-linux-x64.rpm";
client.setTrustManager(TrustManagerUtils.getValidateServerCertificateTrustManager());
client.setKeyManager(creds.getKeystoreManager());
client.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out), true));
client.setCopyStreamListener(createListener());
client.setConnectTimeout(5000);
client.setDefaultTimeout(5000);
client.connect(creds.getHost(), 990);
client.setSoTimeout(5000);
client.setDataTimeout(5000);
if (!FTPReply.isPositiveCompletion(client.getReplyCode())) {
client.disconnect();
log.error("ERROR: " + creds.getHost() + " refused the connection");
} else {
if (client.login(creds.getUser(), creds.getPass())) {
log.debug("Logged in as " + creds.getUser());
client.enterLocalPassiveMode();
client.setFileTransferMode(FTP.BLOCK_TRANSFER_MODE);
client.setFileType(FTP.BINARY_FILE_TYPE);
InputStream inputStream = new FileInputStream(file);
log.debug("Invoking storeFile()");
if (!client.storeFile(destinationFile, inputStream)) {
log.error("ERROR: Failed to store " + file
+ " on remote host. Last reply code: "
+ client.getReplyCode());
} else {
log.debug("Stored the file...");
}
inputStream.close();
client.logout();
client.disconnect();
} else {
log.error("Could not log into " + creds.getHost());
}
}
}
private static CopyStreamListener createListener(){
return new CopyStreamListener(){
private long megsTotal = 0;
#Override
public void bytesTransferred(CopyStreamEvent event) {
bytesTransferred(event.getTotalBytesTransferred(), event.getBytesTransferred(), event.getStreamSize());
}
#Override
public void bytesTransferred(long totalBytesTransferred,
int bytesTransferred, long streamSize) {
long megs = totalBytesTransferred / 1000000;
for (long l = megsTotal; l < megs; l++) {
System.out.print("#");
}
megsTotal = megs;
}
};
}
Is there any way to make the connection ACTUALLY timeout?
SW Versions:
Commons.net v3.3
Java 7
CentOS 6.3
Thanks in advance,
Joe
I ran into this same problem, and I think that I was able to get something that seems to work with the desired timeout behavior when I unplug the ethernet cable on my laptop.
I use 'storeFileStream' instead of 'storeFile', and then use 'completePendingCommand' to finish the transfer. You can check the Apache commons docs for 'completePendingCommand' to see an example of this kind of transfer. It took about 15 mins for it to timeout for me. One other thing: the aforementioned docs include calling 'isPositiveIntermediate' to check for an error, but this wasn't working. I replaced it with 'isPositivePreliminary' and now it seems to work. I'm not sure if that's actually correct, but it's the best I've found so far.