I am using paho library Classes for Mqtt Connections org.eclipse.paho.client.mqttv3.MqttClient. (not MqttAsyncClient)
In my case when I publish using
mqttClient.publish(uid + "/p", new MqttMessage(payload.toString().getBytes()));
This method does the task for me but doesn't return anything so I can't check the latency between publish and pubAck.
To get the latency I use the following instead of directly calling publish function of mqttClient.
public long publish(JsonObject payload , String uid, int qos) {
try {
MqttTopic topic = mqttClient.getTopic(uid + "/p");
MqttMessage message = new MqttMessage(payload.toString().getBytes());
message.setQos(qos);
message.setRetained(true);
long publishTime = System.currentTimeMillis();
MqttDeliveryToken token = topic.publish(message);
token.waitForCompletion(10000);
long pubCompleted = System.currentTimeMillis();
if (token.getResponse() != null && token.getResponse() instanceof MqttPubAck) {
return pubCompleted-publishTime;
}
return -1;
} catch (Exception e) {
e.printStackTrace();
return -1;
}
}
This gets the work done, but I am not sure whether this is the right approach or not. Please let me know in case there is some other way to to do this.
Related
I used the netmq (VisualStudio 2022, by Nuget install netmq) as https://www.nuget.org/packages/NetMQ/ described.
One SubscriberSocket one thread to connect and receive message from one publisher. source code like below:
public void ZMQReceiveThread(string serverIP, int port)
{
//Create SubscriberSocket
SubscriberSocket subSocket = new SubscriberSocket();
//Connect to Publisher
subSocket.Connect("tcp://" + serverIP + ":" + port.ToString());
//Subscribe all topics
subSocket.Subscribe("");
//set timeout value
int timeout = 10000 * 300; //300ms
TimeSpan ts = new TimeSpan(timeout);
while (!_isStopEnabled)
{
NetMQMessage recvMessage = null;
bool bSuccess = subSocket.TryReceiveMultipartMessage(ts, ref recvMessage, 1);
if(bSuccess == true) //Recieve data successfully
{
//Handle the recvMessage
}
else //Timeout
{
//output log message
Loger.Error($"TryReceiveMultipartMessage({ts.TotalMilliseconds} ms) timeout...");
continue;
}
}
}
sometimes the subSocket.TryReceiveMultipartMessage() timeout although the publisher sent message continuously (we used another test app written in C language linked libzmq(https://github.com/zeromq/libzmq) which can receive the continuous message).
Any comments about this topic?
thanks a lot in advance.
I looked through the netmq source code(https://github.com/zeromq/netmq) but cannot find any clues about TryReceiveMultipartMessage()
My question is, behind the scene, for element-wise Beam DoFn (ParDo), how does the Cloud Dataflow parallel workload? For example, in my ParDO, I send out one http request to an external server for one element. And I use 30 workers, each has 4vCPU.
Does that mean on each worker, there will be 4 threads at maximum?
Does that mean from each worker, only 4 http connections are necessary or can be established if I keep them alive to get the best performance?
How can I adjust the level of parallelism other than using more cores or more workers?
with my current setting (30*4vCPU worker), I can establish around 120 http connections on the http server. But both server and worker has very low resource usage. basically I want to make them work much harder by sending out more requests out per second. What should I do...
Code Snippet to illustrate my work:
public class NewCallServerDoFn extends DoFn<PreparedRequest,KV<PreparedRequest,String>> {
private static final Logger Logger = LoggerFactory.getLogger(ProcessReponseDoFn.class);
private static PoolingHttpClientConnectionManager _ConnManager = null;
private static CloseableHttpClient _HttpClient = null;
private static HttpRequestRetryHandler _RetryHandler = null;
private static String[] _MapServers = MapServerBatchBeamApplication.CONFIG.getString("mapserver.client.config.server_host").split(",");
#Setup
public void setupHttpClient(){
Logger.info("Setting up HttpClient");
//Question: the value of maxConnection below is actually 10, but with 30 worker machines, I can only see 115 TCP connections established on the server side. So this setting doesn't really take effect as I expected.....
int maxConnection = MapServerBatchBeamApplication.CONFIG.getInt("mapserver.client.config.max_connection");
int timeout = MapServerBatchBeamApplication.CONFIG.getInt("mapserver.client.config.timeout");
_ConnManager = new PoolingHttpClientConnectionManager();
for (String mapServer : _MapServers) {
HttpHost serverHost = new HttpHost(mapServer,80);
_ConnManager.setMaxPerRoute(new HttpRoute(serverHost),maxConnection);
}
// config timeout
RequestConfig requestConfig = RequestConfig.custom()
.setConnectTimeout(timeout)
.setConnectionRequestTimeout(timeout)
.setSocketTimeout(timeout).build();
// config retry
_RetryHandler = new HttpRequestRetryHandler() {
public boolean retryRequest(
IOException exception,
int executionCount,
HttpContext context) {
Logger.info(exception.toString());
Logger.info("try request: " + executionCount);
if (executionCount >= 5) {
// Do not retry if over max retry count
return false;
}
if (exception instanceof InterruptedIOException) {
// Timeout
return false;
}
if (exception instanceof UnknownHostException) {
// Unknown host
return false;
}
if (exception instanceof ConnectTimeoutException) {
// Connection refused
return false;
}
if (exception instanceof SSLException) {
// SSL handshake exception
return false;
}
return true;
}
};
_HttpClient = HttpClients.custom()
.setConnectionManager(_ConnManager)
.setDefaultRequestConfig(requestConfig)
.setRetryHandler(_RetryHandler)
.build();
Logger.info("Setting up HttpClient is done.");
}
#Teardown
public void tearDown(){
Logger.info("Tearing down HttpClient and Connection Manager.");
try {
_HttpClient.close();
_ConnManager.close();
}catch (Exception e){
Logger.warn(e.toString());
}
Logger.info("HttpClient and Connection Manager have been teared down.");
}
#ProcessElement
public void processElement(ProcessContext c) {
PreparedRequest request = c.element();
if(request == null)
return;
String response="{\"my_error\":\"failed to get response from map server with retries\"}";
String chosenServer = _MapServers[request.getHardwareId() % _MapServers.length];
String parameter;
try {
parameter = URLEncoder.encode(request.getRequest(),"UTF-8");
} catch (UnsupportedEncodingException e) {
Logger.error(e.toString());
return;
}
StringBuilder sb = new StringBuilder().append(MapServerBatchBeamApplication.CONFIG.getString("mapserver.client.config.api_path"))
.append("?coordinates=")
.append(parameter);
HttpGet getRequest = new HttpGet(sb.toString());
HttpHost host = new HttpHost(chosenServer,80,"http");
CloseableHttpResponse httpRes;
try {
httpRes = _HttpClient.execute(host,getRequest);
HttpEntity entity = httpRes.getEntity();
if(entity != null){
try
{
response = EntityUtils.toString(entity);
}finally{
EntityUtils.consume(entity);
httpRes.close();
}
}
}catch(Exception e){
Logger.warn("failed by get response from map server with retries for " + request.getRequest());
}
c.output(KV.of(request, response));
}
}
Yes, based on this answer.
No, you can establish more connections. Based on my answer, you can use a async http client to have more concurrent requests. As this answer also describes, you need to collect the results from these asynchronous calls and output it synchronously in any #ProcessElement or #FinishBundle.
See 2.
Since your resource usage is low, it indicates that the worker spends most of its time waiting for a response. I think with the described approach above, you can utilize your resources far better and you can achieve the same performance with far less workers.
I'm trying to understand the correct way to use the Flume RpcClient in a multithreaded application. Information I have found so far indicates that the components are thread safe, but the example in the Flume documentation clouds the issue when it comes to error handling. This code:
public void sendDataToFlume(String data) {
// Create a Flume Event object that encapsulates the sample data
Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));
// Send the event
try {
client.append(event);
} catch (EventDeliveryException e) {
// clean up and recreate the client
client.close();
client = null;
client = RpcClientFactory.getDefaultInstance(hostname, port);
// Use the following method to create a thrift client (instead of the above line):
// this.client = RpcClientFactory.getThriftInstance(hostname, port);
}
}
If more then one thread calls this method, and the exception is thrown, then there will be a problem as multiple threads try and recreate the client in the exception handler.
Is the intent of the SDK that it should only be used by a single thread? Should this method be synchronized, as it appears to be in the log4jappender that is part of the Flume source? Should I put this code in its own worker and pass it events via a queue?
Does anyone have an example of RpcClient being used by more then one thread (included the error condition)?
Would I be better off using the "embedded agent"? Is that multithread friendly?
With the embedded agent, you get the same case except you don't know what to do:
try {
agent.put(event);
} catch (EventDeliveryException e) {
// ???
}
You could stop the agent, and restart it - but you would need a synchronized block (or a ReentrantReadWriteLock, to not block thread while "reading" the client field). But since I'm not a Flume expert, I can't tell you which one is better.
Example:
class MyClass {
private final ReentrantReadWriteLocklock;
private final Lock readLock;
private final Lock writeLock;
private RpcClient client;
private final String hostname;
private final Integer port;
// Constructor
MyClass(String hostname, Integer port) {
this.hostname = Objects.requireNonNull(hostname, "hostname");
this.port = Objects.requireNonNull(port, "port");
this.lock = new ReentrantReadWriteLock();
this.readLock = this.lock.readLock();
this.writeLock = this.lock.writeLock();
this.client = buildClient();
}
private RpcClient buildClient() {
return RpcClientFactory.getDefaultInstance(hostname, port);
}
public void sendDataToFlume(String data) {
// Create a Flume Event object that encapsulates the sample data
Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));
// Send the event
readLock.lock(); // lock for reading 'client'
try {
try {
client.append(event);
} catch (EventDeliveryException e) {
writeLock.lock(); // lock for reading/writing client
try {
// clean up and recreate the client
client.close();
client = null;
client = buildClient();
} finally {
writeLock.unlock();
}
}
} finally {
readLock.unlock();
}
}
}
Beside, the example will lose the event because it is not sent back. Some kind of loop + a max retry would probably do the trick:
int i = 0;
for (; i < maxRetry; ++i) {
try {
client.append(event);
break;
} catch (EventDeliveryException e) {
// clean up and recreate the client
client.close();
client = null;
client = RpcClientFactory.getDefaultInstance(hostname, port);
// Use the following method to create a thrift client (instead of the above line):
// this.client = RpcClientFactory.getThriftInstance(hostname, port);
}
}
if (i == maxRetry) {
logger.error("flume client is offline, loosing events {}", event);
}
That's the idea, but I don't think that should be the task of the user (eg: us), but an option in the client or the agent to store event that could not be processed due to such errors.
I am trying to write a simple application to send messages to a topic from use input and show messages published on topic.
There are two command line executables - one for publisher and another for subscriber.
When I publish messages on a topic, I can see the messages getting submitted to the topic.
The following command shows that there are messages on the topic (see F1.gif):-
show stat EMS.Test.Topic
The following command shows that the messages are getting consumed by the subscribers (see F2.gif)
show stat consumers topic=EMS.Test.Topic
However, I am not able to retrieve messages the EMS .NET API. It gets stuck on Message msg = subscriber.Receive();. I made sure the connection details and authentication details are correct because they are used when publishing the messages.
public string ReceiveMessagesFromTopic(string topicName)
{
TopicConnection connection = null;
string messageFromPublisher = string.Empty;
try
{
var factory = new TIBCO.EMS.TopicConnectionFactory(serverUrl);
connection = factory.CreateTopicConnection(userName, password);
TopicSession session = connection.CreateTopicSession(false, Session.AUTO_ACKNOWLEDGE);
Topic topic = session.CreateTopic(topicName);
TopicSubscriber subscriber = session.CreateSubscriber(topic);
connection.Start();
while (true)
{
Message msg = subscriber.Receive();
if (msg == null)
{
break;
}
if (msg is TextMessage)
{
TextMessage tm = (TextMessage) msg;
messageFromPublisher = tm.Text;
}
}
connection.Close();
}
catch (EMSException e)
{
if (connection!=null)
{
connection.Close();
}
throw;
}
return messageFromPublisher;
}
There was a silly mistake in my .NET code. the following while loop never returns so there is no return. I need to break the while loop when I get a message. Duh!!!!
while (true)
{
Message msg = subscriber.Receive();
if (msg == null)
{
break;
}
if (msg is TextMessage)
{
TextMessage tm = (TextMessage) msg;
messageFromPublisher = tm.Text;
break;
}
}
Can we call a webservice from the scheduled periodic task class firstly, if yes,
Am trying to call a webservice method with parameters in scheduled periodic task agent class in windows phone 7.1. am getting a null reference exception while calling the method though am passing the expected values to the parameters for the webmethod.
am retrieving the id from the isolated storage.
the following is my code.
protected override void OnInvoke(ScheduledTask task)
{
if (task is PeriodicTask)
{
string Name = IName;
string Desc = IDesc;
updateinfo(Name, Desc);
}
}
public void updateinfo(string name, string desc)
{
AppSettings tmpSettings = Tr.AppSettings.Load();
id = tmpSettings.myString;
if (name == "" && desc == "")
{
name = "No Data";
desc = "No Data";
}
tservice.UpdateLogAsync(id, name,desc);
tservice.UpdateLogCompleted += new EventHandler<STservice.UpdateLogCompletedEventArgs>(t_UpdateLogCompleted);
}
Someone please help me resolve the above issue.
I've done this before without a problem. The one thing you need to make sure of is that you wait until your async read processes have completed before you call NotifyComplete();.
Here's an example from one of my apps. I had to remove much of the logic, but it should show you how the flow goes. This uses a slightly modified version of WebClient where I added a Timeout, but the principles are the same with the service that you're calling... Don't call NotifyComplete() until the end of t_UpdateLogCompleted
Here's the example code:
private void UpdateTiles(ShellTile appTile)
{
try
{
var wc = new WebClientWithTimeout(new Uri("URI Removed")) { Timeout = TimeSpan.FromSeconds(30) };
wc.DownloadAsyncCompleted += (src, e) =>
{
try
{
//process response
}
catch (Exception ex)
{
// Handle exception
}
finally
{
FinishUp();
}
};
wc.StartReadRequestAsync();
}
private void FinishUp()
{
#if DEBUG
try
{
ScheduledActionService.LaunchForTest(_taskName, TimeSpan.FromSeconds(30));
System.Diagnostics.Debug.WriteLine("relaunching in 30 seconds");
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.ToString());
}
#endif
NotifyComplete();
}