I am using Axibase Time Series Database Community Edition, version 10379. I try to store my data that comes from a force sensor and save it every 2 milliseconds, how can I configure the portal to accept this time resolution?
I made an attempt to send the data in that rate by using an Arduino board with WiFi shield but the TCP connection disconnected after sending a little data.
Time resolution in Axibase Time-Series Database is 1 millisecond by default, so the problem is probably occurring for other reasons such as:
Invalid timestamp
Missing end-of-line character at the end of the series command
Same timestamp for multiple commands with the same entity/metric/tags. For example, these commands are duplicates and one of the them will be discarded:
series ms:1445762625574 e:e-1 m:m-1=100
series ms:1445762625574 e:e-1 m:m-1=125
Overflow of receiving queue in ATSD. This can occur if ingestion rate is higher than disk write speed for long period of time. Open ATSD portal in the GUI and check the top right chart if rejected_count metric is greater than zero. This can be addressed by changing default configuration settings.
Other reasons specified in https://axibase.com/docs/atsd/api/data/#errors
I would recommend starting netcat in server mode and recording data from the Arduino board to file to see exactly what commands are sent into ATSD.
Stop ATSD with ./atsd-tsd.sh stop
Launch netcat in server mode and record received data to command.log file:
netcat -lk 8081 > command.log
Restart Arduino and send some data into ATSD (now netcat). Review command.log file
Start ATSD with ./atsd-tsd.sh start
Disclosure: I work for Axibase.
Related
Let's say you are writing a program that many clients use. One server will only be able to handle the connections to a certain amount. The more connections you need to handle, the more power you need until you get a server farm containing different devices.
If you for example run an application where different clients can store data on your servers how is it possible to synchronize data on each device? Which hardware/software solutions exist? Or how is all the data stored?
I can suggest an idea for a manual program creation , using file system only , you can exchange files between clients and server , but the server program will , in a period of time ( for example every 5 minutes ) broadcast the list of all his files to all connected clients and the exchanges will have to wait then ( if we are talking about a big volume of files) if its for small files then a 30 sec or 1 minute can be enough
There are many forms to do that...
I think you will need to have a single source of truth, let's say (the server on this case).
Then you can have a incremental number version (id) that contains the latest changes.
So each device can poll that number version only in order to know if is up to date.
If not, the device can ask the server the changes from the version that device has.
Each time a device makes a change, that change is stored on the server, and the version is incremented.
That can be one basic implementation. If you need real time updates, you can add to that implementation some publish-suscribe channel using Sockets... or some service like channels from pusher.com
So, every time a device made a change on the server, you can send a notificaiton on the channel with some information (the change) or the new ID of versión and all the devices can update the information via the change (if is only one) or ask the server all the changes if there are many (like the device was disconnected from internet for a while).
I have successfully managed to install ThingsBoard on a Ubuntu 18.04. In my application I want to send large data packages from a few (<20) devices via MQTT. I also want to display the incoming data packets in real time.
For testing purposes I now play with the chart dashboard of ThingsBoard. Unfortunately I am not able to set the plotting interval on the dashboard to less than 1 second.
This is the current situation and I try to increase the speed of the plot:
Dashboard GIF
Are there any other settings that would meet my needs?
Thank you very much.
change the "data aggregation function" to "none" and it will start displaying real-time, raw data.
Let's say we've got a server with ticking clock (just an integer variable that holds a number of milliseconds from epoch and being updated every time when the system time is changed).
Each client should be able to synchronize its own clock (not the system clock! just an abstract timer, an integer value that's increasing with time) with server's.
For example the client can send a "sync" request and the server will respond with its current clock value.
The problem is that we can't know exactly how much time will be taken by the sending and receiving the data. We know the time that was passed from sending the request to receiving the data, we can divide this difference by 2 and add this to the value received from server, but it is not very accurate!
Is there a common concept of synchronizing clock between server and a client with minimum difference of values on both sides after performing it?
I would recommend using NTP (Network Time Protocol) for this, it is just the way to synchronize clocks in an entire network.
http://de.wikipedia.org/wiki/Network_Time_Protocol
Almost every operating system out there (I manage Linux, Solaris, BSD and Windows) have clients and servers for NTP. The protocol has build in capabilities to take care of network latency.
I would like to use InfluXDB wit my Raspberry/Openhab home automatisation.
I am just worried about db size/performance.
So my plan would be: log only 1 month on Pi, let it be cleaned automatically.
Clean I understand is easy with retention. (Automatically clear old data )
For long time anaylsis I want to collect all the data on a server.
Now question: How can I export the data on PI before retention into flatfile and afterwards import that data in a seperate InfluXDB on different server?
(Or even better: is there a way to do this in a sort of cluster mode?)
thanks a lot,
Chris
I use InfluxDB on a pi for sensor logs. I log now 4 records every 5 seconds for more than 3 months and performances on my pi are really good. I don't have now the file size but was not more than 10Mb
You can use InfluxDB in cluster mode but not sure it will answer yor question for data cleaning.
To exprt data, you can use InfluxDB API to get all series in the data base, then all data and flush that in a json file. You can use the API to load that file in another DB
Environment:
Delphi 2009 client applications (and one Java), running on Windows 2003 server
connecting to InterBase 7.5.1 (another Windows 2003 Server) over dbExpress
The Delphi applications log the time to open the TSQLConnection using the AfterConnect event handler of the TSQLConnection object.
In random intervals, the connect need a three-minutes "extra time". I first suspected it could be a problem with the SQL query, but more detailed logging today showed that it is the SQLConnection.Connect which hangs.
I am not sure if this could be a problem with network, the InterBase server, or the Delphi / dbExpress layer.
Has anybody experienced a similar three-minutes "hang"?
p.s. the Java application does not log connect time so I can not say wheter it is affected by this problem.
This phenomenon appeared in the log files since we started with logging in 2012, but the rate has sharply increased last month. The only environment change has been the addition of new Windows servers (for RDP services, Mail, and Fax) so it could be a network-related problem.
Aside of a possible network problem, the cause of the delay can be that, from time to time, your query triggers a garbage collection in one of the table(s) that it is querying.
I don't know in detail Interbase 7.5 internals, but in my experience (with Firebird), this usually happens when a select is made on a table from which many records have been deleted/updated recently.
This doc at IBExpert.net explains it:
A garbage collection is only performed during a database sweep, database backup or when a SELECT query is made on a table (and not by INSERT, ALTER or DELETE). Whenever Firebird/InterBase® touches a row, such as during a SELECT operation, the versioning engine sweeps out any versions of the row where the transaction number is older than the Oldest Interesting Transaction (OIT). This helps to keep the version history small and manageable and also keeps performance reasonable.
A periodic sweep or backup made at low usage hours, can increase performance and minimize the risk of being hitted by an inconvenient garbage collection. See Sweep interval and automated housekeeping (page 6-20) and Facilitating garbage collection (page 11-19) at the Interbase 7.5 Operations Guide for more info on this.
Please check if hard disk power saving is activated on any disk on mentioned servers. That would explain if you have a delay in first connect and then no delay in following connections. Then, after a while power saving gets activated and the same problem raises again.
Since the rate has increased with the additions of new servers on the network you could have a packet loss and a long timeout to retry. For test that hypothesis you can change the connection timeout to a small value. You also can monitor the network traffic between the servers using wireshark or tcpdump.
Monitoring
To monitor the TCP handshake only you can use:
tcpdump -i eth0 'tcp[13] & 2 = 2