I'm connected to the stream Aggregate Trade Streams from the Binance API, and I can't find from the event returned if it's a bid or a ask. And even from the documentation I don't see how I can distinguish https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md#aggregate-trade-streams
I don't see the point of this stream if I can't distinguish the bid/ask.
An example of event I'm getting:
EventTime:1637704801311
Symbol:DOGEBTC
TradeID:0
Price:3.92e-06
Quantity:326
BuyerOrderID:0
SellerOrderID:26377001
TradeTime:1637704801311
IsMarketMaker:false
Ignore:true
I finally found it, the flag IsMarketMaker determines if it's a BUY or SELL
Related
The official documentation states that one can use the x-stream-offset consumer argument to specify where to start reading the Stream from. But it doesn't say where to get these values from.
For example, what does timestamp mean here? The point in time when the chunk was inserted into the stream? If one is reading messages from the Stream, is there a way to get this timestamp to store it so that when the process is restarted it can continue from where it left off?
Same with the numerical offset value... how does a consumer know the current offset when reading messages from the stream?
The offset for each message can be retrieved from its headers.
Usually:
message.properties.headers["x-stream-offset"]
I am trying to backprocess data in Kafka topics using a Kafka Streams application that involves a join. One of the streams to be joined has much larger volume of data per unit of time in the corresponding topic. I would like to control the consumption from the individual topics so that I get roughly the same event timestamps from each topic in a single consumer.poll(). However, there doesn't appear to be any way to control the behavior of the KafkaConsumer backing the source stream. Is there any way around this? Any insight would be appreciated.
Currently Kafka cannot control the rate limit on both Producers and Consumers.
Refer:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
But if you are using Apache Spark as the stream processing platform, you can limit the input rate for the Kafka receivers.
in the consumer side you can use consume([num_messages=1][, timeout=-1])
function instead of poll.
consume([num_messages=1][, timeout=-1]):
Consumes a list of messages (possibly empty on timeout). Callbacks may be executed as a side effect of calling this method.
The application must check the returned Message object’s Message.error() method to distinguish between proper messages (error() returns None) and errors for each Message in the list (see error().code() for specifics). If the enable.partition.eof configuration property is set to True, partition EOF events will also be exposed as Messages with error().code() set to _PARTITION_EOF.
num_messages (int) – The maximum number of messages to return (default: 1).
timeout (float) – The maximum time to block waiting for message, event or callback (default: infinite (-1)). (Seconds)
After a mailing at t0, I will have several "delivered" (and open and click) events (schema and example)
mailing_name, timestamp, email_id, event_type
niceattack, 2016-07-14 12:11:00, 42, open
niceattack, 2016-07-14 12:11:08, 842, open
niceattack, 2016-07-14 12:11:34, 847, open
I would like to see for a mailing how long it takes to be delivered to half of the recipients. So say that I'm sending an email to 1000 addresses now, the first open event is in 2 min, the last one is going to be in a week (and min/max first last seems to be easy to find) but what I'd like to see is that half of the recipients opened it in the first 2 hours after it was sent.
The goal is to send being able to compare is sending now vs on sat morning makes a difference on how fast it's open on average, or if one specific mailing get quicker exposure, and correlate that with other events (how many click on a link, take a specific action on our site...)
I tried to use a cumulate function (how many open event for mailing for each point), but it seems that the cumulative function isn't yet implemented https://github.com/influxdata/influxdb/issues/813
How do you solve that problem with influxdb?
Solving this problem with InfluxDB alone is not currently possible, however if you're willing to add Kapacitor into the mix, then it should be possible. In particular you'll need to write a User Defined Function (UDF) for that cumulative function in Kapacitor.
The general process will look like the following:
Install and Configure Kapacitor
Create a UDF for the cumulative function you're looking for
Enable that UDF inside of Kapacitor
Write a TICKscript that uses the UDF and writes the results back to InfluxDB
Enable a task defined by the TICKscript you've written
Query the InfluxDB instance to get the results of the cumulative function.
My appoligies for being so high level on this. This is a fairly involved process, but should give you the result you're looking for.
My aim is to measure MQTT device-to-device message latency (not throughput) and I'm looking for feedback on my code-hacks. The setup is simple; just one device serving as two end-points (old Linux PC with two terminal sessions; one running the subscriber and the other running the publisher sample app) and the default broker at tcp://m2m.eclipse.org:1883). I inserted time-capturing code-fragments into the C-language publish/subscribe sample apps on the src/samples folder.
Below are the changes. Please provide feedback.
Changes to the subscribe sample app (MQTTAsync_subscribe.c)
Inserted the lines below at the top of the msgarrvd (message arrived) function
//print arrival time
struct timeval tv;
gettimeofday (&tv, NULL);
printf("Message arrived: %ld.%06ld\n", tv.tv_sec, tv.tv_usec);
Changes to the publish sample app (MQTTAsync_publish.c)
Inserted the lines below at the top of the onSend (callback) function
struct timeval tv;
gettimeofday (&tv, NULL);
printf("Message with token value %d delivery confirmed at %ld.%06ld\n",
response->token, tv.tv_sec, tv.tv_usec);
With these changes (after subtracting the time message arrived at the subscriber from the time that the delivery was confirmed at the publisher), I get a time anywhere between 1 millisecond and 0.5 millisecond.
Questions
Does this make sense as a rough benchmark on latency?
Is this the round-trip time?
Is the round-trip time in the right ball-park? Should be less? more?
Is it the one-way time?
Should I design the latency benchmark in a different way? I need a rough measurements (I'm comparing with XMPP).
I'm using the default QoS value (1). Should I change it?
The publisher takes a finite amount of time to connect (and disconnect). Should these be added?
The 200ms latency is high ! Can you please upload your code here ?
Does this make sense as a rough benchmark on latency?
-- Yes it makes sense. But a better approach is to make an automated time subtract with subscribed message and both synchronized to NTP.
Is this the round-trip time? Is it the one-way time?
-- Messages got published - you received ACK for publisher and same message got transferred to subscribed client.
Is the round-trip time in the right ball-park? Should be less? more?
-- It should be less.
Should I design the latency benchmark in a different way? I need a rough measurements (I'm comparing with XMPP).
I'm using the default QoS value (1). Should I change it?
-- Try with QoS 0 ( fire and forget )
The publisher takes a finite amount of time to connect (and disconnect). Should these be added?
-- Yes. It needs to be added but this time should be very small.
What is the maximum number of custom events you can report per session with Flurry analytics?
The number of events you can report per session in Flurry is 1000. I asked this question to Flurry support, as I couldn't find it elsewhere (and none of the answers here really answered the question). They answered and also sent me a short document titled "Flurry Methodology and Best Practices" that contained, among other things, this summary:
300 unique events per app
1000 events max per session
100 unique event names per session
10 params max per event
Key/Value for params must be Strings
Key max length 255 chars
Value max length 255 chars
As the definition of "session" is important, I quote, from the same document:
Flurry analytics is based on a session model that only “phones home”
at the launch and backgrounding of a session. This prevents
“talkiness” from the SDK, conserves battery by not always running the
radios and allows data to travel in a coherent package."
(...)
One addition to the Flurry session model is the concept that a user
may bounce out of an app for a very brief time and reenter the app and
still be within the original session. A time may be set, in millis,
that is colloquially referred to as the “session timeout”. It is
configurable at app launch (see setContinueSessionMillis for more
details) within a range of 5 seconds to 1 minute with a default of 10
seconds. If, when the user comes back into the app, the “session
timeout” has not been exceeded, then the SDK will treat the “new”
session as a continuation of the former session.
Upon this new launch, if there are any sessions not sent, they will be
sent. At that time, as well, the SDK will make a decision on whether
or not to continue a session or start a new one.
The document is here. Flurry support sent it to me in late February, 2015.
The limit appears to be 300 different event ids, and therefore 300 custom events. Quoting: http://support.flurry.com/index.php?title=Analytics/GettingStarted/TechnicalQuickStart
Your application is currently limited to counting occurrences for 300
different Event ids (maximum length 255 characters).
Addional details from here
Yes, there is a limit of 300 Events for each application. Each event
can have up to 10 parameters, and each parameter can have any number
of values.
I believe it is infinite:
Each Event can have up to 10 parameters, and each parameter can have
an infinite number of values associated with it. For example, for the
‘Author’ parameter, there may be 1,000 possible authors who wrote an
article. We can keep track of each author via this single parameter.
So if you can have an infinite number of values you could have 10 million authors. Since they are all just values each one can be tracked (via the parameter). If they "can keep track of each author via this single parameter" then I don't think your event count would be mitigated. This would assume you setup your event types properly like in their example:
NSDictionary *articleParams =
[NSDictionary dictionaryWithObjectsAndKeys:
#"John Q", #"Author", // Capture author info
#"Registered", #"User_Status", // Capture user status
nil];
[Flurry logEvent:#"Article_Read" withParameters:articleParams];
One event with a maximum of 10 dictionary items, with an infinite number of possible values... I think it would be safe to say you aren't limited here.
There is a limit of 300 Events for each app. Each event can have up to 10 parameters, and each parameter can have any number of values. Please check all details here