I capture data with tshark and save certain data from the packet header to process them in order to detect some incedants in the network. I saved the data in a table in my lua program (which is running in the cmd with tshark using the command (-Xlua_script:))
and now i want to process the data of each minute alone while capturing is running. It's an online processing. Firstly:Any body knows if this could be implemented?Secondly I need a timer, I don't know how to do this, and i want a way that i can take the data in the tables to process them, reset the tables to get the new data of the next minute without losing any data.
Any suggestions or ideas??
there isn't the concept of a 'timer' in lua like some other languages, where you can create one and set up an event handler and have your main program notified when the timer goes off... however you can periodically check os.clock() to determine how long its been since you've done some processing and if a minute has elapsed, go ahead and process the data.
something like this might be what you need:
lastTimeProcessed = os.clock()
function IsTimeToProcess(currentTime)
span = currentTime - lastTimeProcessed
if span >= 60 then
lastTimeProcessed = currentTime
return true
end
return false
end
while true do
if IsTimeToProcess(os.clock()) then
-- process some data here
end
-- otherwise do another round of whatever you're doing
end
Related
Been trying to get a random song "I'm using fruit placeholders for now" and it always returns Orange
and I can't seem to figure out why.
math.randomseed(os.time())
local songLists = {"Apple", "Orange", "Pineapple"}
local songValue = math.random(#songLists)
local songSelected = songLists[songValue]
print(tostring(songSelected))
Lua uses PRNG (pseudorandom number generator) to calculate random number.
math.randomseed(os.time()) initalizes formula with some sort of data current time (not the best decision). os.time() gives current time with prezision of 1s, it means if you execute this code in loop within part of second you will get the same seed number (if clock not changes). The best practice is to initialize random math.randomseed(...) once application is started and with better random data as process id, os.clock(), ...
Most likely your problem is math.randomseed(...) is executed many times with the same seed data in this code or may be in anothe part of application.
i read about flink`s window assigners over here: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#window-assigners , but i cant find any solution for my problem.
as part of my project i need a windowing that the timer will start given the first element of the key and will be closed and set ready for processing after X minutes. for example:
first keyA comes at (hh:mm:ss) 00:00:02, i want all keyA will be windowing until 00:01:02, and then the timer of 1 minutes will start again only when keyA will be given as input.
Is it possible to do something like that in flink? is there a workaround?
hope i made it clear enough.
Implementing keyed windows that are aligned with the first event, rather than with the epoch, is quite difficult, in general, which I believe is why this isn't supported by Flink's window API. The problem is that with an out-of-order stream using event time processing, as earlier events arrive you may need to revise your notion of when the window began, and when it should end. For example, if the first keyA arrives at 00:00:02, but then some time later an event with keyA arrives with a timestamp of 00:00:01, now suddenly the window should end at 00:01:01, rather than 00:01:02. And if the out-of-orderness is large compared to the window length, handling this becomes quite complex -- imagine, for example, that the event from 00:00:01 arrives 2 minutes after the event from 00:00:02.
Rather than trying to implement this with the window API, I would use a KeyedProcessFunction. If you only need to support processing time windows, then these concerns about out-of-orderness do not apply, and the solution can be fairly simple. It suffices to keep one object in keyed state, which might be a list holding all of the events in the window, or a counter or other aggregator, depending on what you're trying to accomplish.
When an event arrives, if the state (for this key) is null, then there is no open window for this key. Initialize the state (i.e., create a new, empty list, or set the counter to zero), and create a Timer to fire at the appropriate time. Then regardless of whether the state had been null, add the incoming event to the state (i.e., append it to the list, or increment the counter).
When the timer fires, emit the window's result and reset the state to null.
If, on the other hand, you want to do this with event time windows, first sort the stream and then use the same approach. Note that you won't be able to handle late events, so plan your watermarking accordingly (reducing the likelihood of late events to a manageable level), or go for a more complex implementation.
Been rolling around the web and forums, cannot find a resource on this.
What I am to achieve is create an alert for when there is no change in data for a period of time.
We are monitoring openfiles for our webserver/s so this number fluctuates rather often. Noticed that when the number is stagnant it points to an issue on the server. So what we want is if openfile remains X for 2minutes alert us.
I made such an alert through a small succession of things:
I have an exclusive 'alerting dummy board', for all the alerts, since I can only have one alert per graph (grafana version 6.6.0)
I use the following query: avg_over_time(delta(Sensor_Data[1m])[20s:]) - this calculates the 20s average of 'first_value-last_value of 1min interval'
My data gathering program feeds into prometheus and this in turn into grafana -- if this program freezes, it might continue sending the last value to prometheus, and the above query will drop to strictly zero.
so I have an alert which goes off if the above query is within a range (-0.01, 0.01) for a minute (a typical value of the above query with system running is abs(query) > 0.18)
Thus, Grafana sends an alert if the Sensor_Data value does not change within about 2-3 minutes.
If you do use Prometheus and Alert manager, There is a nice function that worked for me.
changes
So using something like this in Alert manager will trigger if no changes for the time interval
changes(metric_name[5m]) = 0
This has worked for me. Make sure you're using a rate or increase function (no change means it will drop to zero) and filter the query like the following:
increase(metric_name) > 0
Then, in Alert Config, set "If no data or all values are null" to "Alerting". That way, when there's no data, the alert will be triggered.
Is there any way to measure the performance of a Redis Lua script?
I have a lua script and I ended up to a slightly different implementation and I am wondering if there is any way to measure which of the two implementations is faster.
You can call Redis' TIME command to perform in-script "benchmarking". Something like the following should work:
local start = redis.call('TIME')
-- your logic here
local finish = redis.call('TIME')
return finish[1]-start[1]
I read in the comments that someone mensioned finish[2]-start[2] which is not a good idea because [2] has "the amount of microseconds already elapsed in the current second" and not the entire timestamp (so if we finish in a different second, this calculation will fail.)
based on: https://redis.io/commands/TIME
to get time as microseconds, I would do:
local start = redis.call('TIME')
-- your logic here
local finish = redis.call('TIME')
return (finish[1]-start[1])*1000000+(finish[2]-start[2])
How to write a lua-program for tarantool that will perform some tasks regularly (e.g. once per 10 minutes) in background?
The first way
Use fibers. Fibers - is a set of instructions which are executed with cooperative multitasking. Fibers managed by the fiber package are associated with a user-supplied function called the fiber function. A fiber has three possible states: running, suspended or dead.
Example
fiber.create(function()
while true do
-- Let say you have space with tree index.
-- Where each row index is timestamp + interval.
-- So, here you can get lower/upper bound by current timestamp e.g.
-- space:select{fiber.now()} -- get expired tasks
fiber.sleep(1) -- interval
end
end)
The second way
Use expirationd - https://github.com/tarantool/expirationd