I process data that arrives on an amazon server, put the data in an influxDB (0.9.5) and visualise on Grafana. Now suddenly data from two measurements is not selected any-more before a certain timestamp (11 July), e.g.
SELECT * FROM cable WHERE time < '2016-07-11 00:00:00'
would return no results.
However, the data is still there: I re-precessed one data-file from 15 May 2016, and now the data from 9 May till 15 May shows again.
This is quite a strange behaviour, would anyone have a clue what is going on here?
Thanks,
Bart
Related
Good Afternoon,
I want to make my life easier by querying SQLite databases I find on mobile devices, as opposed to manually putting the values in MFT Stampede.
I have a database with 70 tables that I extracted from an iOS device. There is a table I have a particular interest in which keeps a record of all images that have been stored on the iOS device in a particular directory I'm interested in. I ran a timestamp number "680204956.051849" in MFT and got the "MAC (CF) Absolute Time of "Fri, 22 Jul 2022 17:49:16". I ran a query to extract all the dates:
SELECT
datetime(ZADDEDDATE, 'unixepoch')
FROM ZASSET
LIMIT 15;
For the same field I ran in MFT, I get "1991-07-22 17:49:16". The year is wrong, any idea how I can get the correct year?
We have a fact table which collects information detailing when an employee selected a benefit. The problem we are trying to solve is how to count the total benefits selected by all employee's.
We do have a BenefitSelectedOnDay flag and ordinarily, we can do a SUM on this to get a result, but this only works for benefit selections since we started loading the data.
For Example:
Suppose Client#1 has been using our analytics tool since October 2016. We have 4 months of data in the platform.
When the data is loaded in October, the Benefits source data will show:
Employee#1 selected a benefit on 4th April 2016.
Employee#2 selected a benefit on 3rd October 2016
Setting the BenefitSelectedOnDay flag for Employee#2 is very straight forward.
The issue is what to do with Employee#1 because we can’t set a flag on a day which doesn’t exist for that client in the fact table. Client#1's data will start on 1st October 2016.
Counting the benefit selection is problematic in some scenarios. If we’re filtering the report by date and only looking at benefit selections in Q4 2016, we have no problem. But, if we want a total benefit selection count, we have a problem because we haven’t set a flag for Employee#1 because the selection date precedes Client#1’s dataset range (Oct 1st 2016 - Jan 31st 2017 currently).
Two approaches seem logical in your scenario:
Load some historical data going back as far as the first benefit selection date that is still relevant to current reporting. While it may take some work and extra space, this may be your only solution if employees qualify for different benefits based on how long the benefit has been active.
Add records for a single day prior to the join date (Sept 30 in this case) and flag all benefits that were selected before and are active on the Client join date (Oct 1) as being selected on that date. They will fall outside of the October reporting window but count for unbounded queries. If benefits are a binary on/off thing this should work just fine.
Personally, I would go with option 1 unless the storage requirements are ridiculous. Even then, you could load only the flagged records into the fact table. Your client might get confused if he is able to select a period prior to the joining date and get broken data, but you can explain/justify that.
I'm developing a rails website and trying to implement a cache layer.In current website we are displaying the stock information for each stock using D3 chart rendering and every other second sending a request to server for new data and appending it with the current rendered D3 chart.
My Design approach*
I will implement a caching layer that will internally send request to database ,let say every 10 second and update the cache for last one hour so at given point my cache will always have last 1 hour of data so any request that match with in this time stamp will be served from cache.
Issues :
How to store data in cache. Currently,I'm thinking of memcached for distributed caching and key as timestamp but how I invalidate the earliest timestamp key when new key with latest 10 second data comes in ?
Some of the data don't come sequentially, let say data for 14:02:33 will come later than 14:02:38. How to avoid such scenarios
Let me know if you guys have better approach to design this problem.
Thanks
Memcached supports expiration natively. That means you can assign 1 hour expiration for every new key and they will be removed automatically by the Memcached.
Such problem does not exist as memcached doesn't care about the key sequences. It's a single key - single value system.
If you care about different timeframe, you can quantize time by let's say 5s, so 14:02:33 -> 14:02:30; 14:02:37 -> 14:02:35, and so on.
Iam very much new to Zabbix. i have tried my hands on triggers. what i was able to make out was it can set triggers on some constant threshold. what i need is that it should compare with the data which i exactly one week old for that exact time and if the change is above some particular % threshold then trigger an alert.
i had tried some steps like keeping the current data and one week old data in and external database and then querying that data with zabbix ODBC drivers but then i was stuck when i was not able to compare two items.
if i may be confusing stating my issue. let me know and i will be more clear with my problem
you can use the last() function for this.
For example if we sample our data every 5 minutes and we want to compare the last value with the value 10 minutes ago we can use
(item1.last(#1)/item2.last(#3)) > 1.2 - this will trigger an alert if the latest value is greater by 20% than the value 10 minutes ago.
From the documentation it is not very clear to me if you can use seconds or if they will be ignored (for example item.last(60) - to get the value 1 minute ago), but you can read more about the last function here:
https://www.zabbix.com/documentation/2.4/manual/appendix/triggers/functions
I'm working on an app that connects to a mysql backend. It's a little simliar to snapchat in that once the current user gets the pics from the users they follow and see them they can never again see these pics. However, I can't just delete the pics from the database, the user who uploaded the pic still needs to see them. So I've come up with an interesting design and I want to know if its good or not.
When uploading the pic I would also create a mysql event that would run the same time exactly one day after the pic was uploaded deleting itself. If I have people uploading pics all the time events would be created all the time. How does this effect the mysql database. Is this even scalable?
No, not scalable: Deleting of single records is quick, however if your volume increases, you run into trouble. You do however have a classic case for using partitioning:
Create table your_images (insert_date DATE,some_image BLOB, some_owner INT)
ENGINE=InnoDB /* row_format=compressed key_block_size=4 */
PARTITION BY RANGE COLUMNS (insert_date)
PARTITION p01 VALUES LESS THAN ('2015-07-12'),
PARTITION p02 VALUES LESS THAN ('2015-07-03'),
PARTITION p0x VALUES LESS THAN (ETC),
PARTITION p0n VALUES LESS THAN (MAXVALUE));
You can then insert just as you are used to, drop the partitions once per day (using 1 event for all your data), and create new partitions also once per day (using the same event which is dropping your old partitions).
To make certain a photo lives for 24 hours (minimum), the partition cleanup has to occur with a 1 day delay (So cleanup the day before yesterday, not yessterday itself).
A date filter in your query getting the image from the database is still needed to prevent the images from older then a day being displayed.