Using Google Spreadsheets, I need to enter data structured like the example below.
There will be multiple "quadrants"
Each "quadrant" can contain one or many "days",
Each "day" can contain one or many "times".
This data will ultimately be imported in some backend db (e.g. Access DB, SQL, MySQL).
Question: For each day, how do I represent multiple times? Do I create a new row?
Quadrant One Team Schedules
Sunday
10:00 AM - Red Team
3:00 PM - Green Team
Monday
6:00 AM - Red Team
10:00 AM - Yellow Team
3:30 PM - Green Team
Tuesday
Wednesday
6:00 PM - Yellow Team
Thursday
1:00 PM - Red Team
Friday
Saturday
10:00 AM - Blue Team
3:00 PM - Red Team
I’m not quite sure what answer you are expecting but wanting to post an image (and probably length!) is why this is not a comment.
Poor data layout that requires changes to help legibility or changes to facilitate further processing is, IMO, a very big issue – much more so than, it seems, is appreciated by novices (see perhaps Kruger-Dunning). Again merely my opinion, but I think about half of all questions on SO have data layout as an issue, in whole or part.
Some suggestions:
With databases, always have an index (ID) to identify unique records (rows). Often added automatically.
Try to ensure each record is complete for every field (nulls may cause issues). ID6 seems not required.
Use dates rather than days of the week (it is easier to get the day from the date than the date from the day!)
(Personal preference – not always viable) Use ‘scientific’ notation for dates (YYYYMMDD) to avoid ambiguity between ‘US’ and ‘UK’ systems – and the difficulties in switching between them.
Use the 24-hour clock (saves the space for AM and PM, reduces ambiguity and generally is easier to process).
Not so important nowadays but should consider codes (with a lookup table if desired) such as YL for Yellow rather than indeterminate length strings – saves on data storage so less cost, more speed win/win.
Related
I'm having a trouble when develop multiple timezones website.
Currently I'm storing time in UTC after some researches and it is working fine in most cases.
But there is one case that I couldn't find solution for it:
There are two kinds of user in two countries which are United States
and Thailand.
User in Thailand is worker (A).
User in US is manager (B).
When A starts working, their activities logged into our system and B
can watch those via a monitoring screen on web app and they can choose
the date on that.
Example user A starts working at 8 AM on 23 June with mobile
app, when B chooses 23 June date on the monitoring screen, they
can see the activities of user on Thailand on 23 June (because the results is queried by UTC time), but the
problem is he should see the activities on 22 June instead of 23
June because the time in Thailand is faster than United States 12
hours.
How can I show to user B activities of user A when he chooses the date 22 June?
You've not asked about any particular technology stack or implementation, so I can only answer from a general perspective.
Concepts worth understanding:
Thailand has a single time zone, which has an offset of UTC+7 all year.
The US has multiple time zones, whose offsets range from UTC-10 to UTC-4, depending on what part of the country you are referring to, whether or not daylight saving time is in effect, and whether or not a particular location observes daylight saving time. (Most of the country does, but all of Hawaii and much of Arizona does not.)
A "date" is just a year, month, and day on a calendar, but the time that which a date is observed is different depending on the time zone of the observer. There is a good visualization of this at everytimezone.com.
In your situation, you will have to decide the behavior you want depending on the specific needs of your application:
Do you want the period shown to represent all activities on the date as observed by the person choosing the date? If so, then determine the start of the current date and the start of the next date in the local time zone of the person selecting the date. Convert those to UTC, and query for all events in that UTC time range.
Example:
Example Activity Time: 2018-06-23T18:00:00+07:00 (Asia/Bangkok)
Stored as UTC: 2018-06-23T11:00:00Z
Date Selected: 2018-06-23 (America/New_York)
Local Range: [2018-06-23T00:00:00-04:00 , 2018-06-24T00:00:00-04:00 )
UTC Range: [2018-06-23T04:00:00Z , 2018-06-24T04:00:00Z )
Query: ... where ActivityUTC >= '2018-06-23 04:00:00' and ActivityUTC < '2018-06-24 04:00:00'
Or, do you want the date selected to always represent the date of the activity in the time zone of the person who recorded that activity, regardless of the time zone of the viewer? If so, then store that local date in a separate date-only column and just query on it without regard to time zone.
Example:
Example Activity Time: 2018-06-23T18:00:00+07:00 (Asia/Bangkok)
Local Date Stored: 2018-06-23
Date Selected: 2018-06-23
Query: ... where ActivityLocalDate = '2018-06-23'
Note, you might still store the UTC date and time in some other field, but it isn't relevant for this particular query.
From prior experience in the time and attendance industry, I can say that if it were me I would want the second option - as workers are typically paid based on their own time zones, not on those of their manager. However their are indeed edge cases and you'll have to decide for yourself which approach best matches your business requirements.
This Answer is specific to MySQL.
If you want B to see what A's clock says, use DATETIME; it will say 8AM.
If you want B to see A logging in in the middle of the night, use TIMESTAMP.
(This extends to A vs B, and to date as well as clock.)
Twice a year, DATETIME has a hiccup between 2AM and 3AM if there is a switch between standard and daylight-savings time.
I'm speccing an application that displays time periods to the user. The goal is to present periods in a simple view (no time, no timezones) and detailed view (date and time, with timezone data). The simple view should be unambiguous, in other words the user can glance at it and their assumptions about what they see are correct (they are valid in the local timezone).
For the end of the global period, displaying the date in the AoE timezone [1] will solve this problem. For example, a submission deadline might display as 2018-04-03 (actually 2018-04-03 23:59:59 AoE). This means submissions are accepted as long as it is April 3 somewhere on the planet.
But I also want to indicate that start of a global period. For example, if submissions open on April 2 2018 00:01, they are accepted as soon as it is April 2 somewhere on the planet. (This would currently be at UTC+14, matching the Line Islands.)
I can't see a way to use AoE to derive a global start time. Is there an equivalent to AoE (a standardized semantic timezone) that tracks the global start time?
Notes:
Hardcoding UTC-12 and UTC+14 is the simple answer for the modern day. But I'm looking for semantic timezones that would be updated if the values changed (and not reference non-existent historical datetimes).
I thought I'd seen Etc/AoE in the tz database but this is not the case.
References:
AoE
UTC-12:00
UTC+14:00
[1] The Anywhere on Earth (AoE) timezone represents the moment a datetime expires "anywhere on Earth". It currently matches time at Howland Island (UTC-12). If a UTC-13 timezone were invented, it would be updated to track that.
As far as I could understand, AoE is not a timezone as defined by IANA (AFAIK, a list of all offsets from some geographic region during history).
It's more like a "concept", an idea of a specific date being valid in any place on earth. As you said, this notion of "being valid" will change if more timezones are created or removed.
I don't even know if date/time API's can properly handle AoE automatically - maybe I should study more. But my conclusion is that the only way to achieve your goal is to check manually:
you could check all available timezones and see if the date is valid there, comparing to the current date/time at that zone
you could configure the UTC+14 as the offset to be compared, and make some scheduled job (daily/weekly/every-time-IANA-publishes-a-new-version?) to check all zones and set the correct one (with the biggest offset?). You must also take care if this zone has Daylight Saving changes, because the offset will change as well (and what to do with overlaps, when clocks shift 1 hour back and a local time may exist twice?)
We have a fact table which collects information detailing when an employee selected a benefit. The problem we are trying to solve is how to count the total benefits selected by all employee's.
We do have a BenefitSelectedOnDay flag and ordinarily, we can do a SUM on this to get a result, but this only works for benefit selections since we started loading the data.
For Example:
Suppose Client#1 has been using our analytics tool since October 2016. We have 4 months of data in the platform.
When the data is loaded in October, the Benefits source data will show:
Employee#1 selected a benefit on 4th April 2016.
Employee#2 selected a benefit on 3rd October 2016
Setting the BenefitSelectedOnDay flag for Employee#2 is very straight forward.
The issue is what to do with Employee#1 because we can’t set a flag on a day which doesn’t exist for that client in the fact table. Client#1's data will start on 1st October 2016.
Counting the benefit selection is problematic in some scenarios. If we’re filtering the report by date and only looking at benefit selections in Q4 2016, we have no problem. But, if we want a total benefit selection count, we have a problem because we haven’t set a flag for Employee#1 because the selection date precedes Client#1’s dataset range (Oct 1st 2016 - Jan 31st 2017 currently).
Two approaches seem logical in your scenario:
Load some historical data going back as far as the first benefit selection date that is still relevant to current reporting. While it may take some work and extra space, this may be your only solution if employees qualify for different benefits based on how long the benefit has been active.
Add records for a single day prior to the join date (Sept 30 in this case) and flag all benefits that were selected before and are active on the Client join date (Oct 1) as being selected on that date. They will fall outside of the October reporting window but count for unbounded queries. If benefits are a binary on/off thing this should work just fine.
Personally, I would go with option 1 unless the storage requirements are ridiculous. Even then, you could load only the flagged records into the fact table. Your client might get confused if he is able to select a period prior to the joining date and get broken data, but you can explain/justify that.
The question
How do you handle a change in grain (from weekly measurement to daily measurement) for a snapshot fact table.
Background info
For a star-schema design I want to incorporate the results of a survey as a fact (e.g. in week 2 of 2015 80% of the respondents have responded 'yes', in week 3 76% etc.)
This survey is conducted each week, and I only have access to the result of the survey (% of people saying yes this week) and not to the individual responses.
Based on (my interpretation of) Christopher Adamson's "Star Schema: The complete reference" I believe I should use a snapshot fact table for these kind of measurements.
The date dimension for this fact should be on the week-level, and be a conformed rollup of a more fine-grained date dimension for other facts in other stars that take place on a daily basis.
Here comes trouble
Now someone decides they want to conduct these surveys daily instead of weekly. What is the best way to handle this? Some of the options I'm currently considering:
change the week dimension to a daily one, and fake the old facts as if they happened on the last day of the week.
change the week dimension to a daily one, and add 7 facts for each weekly one.
create a new star, with the daily fact and dimension and treat the old one as an aggregate.
I'd appreciate any input. Please tell me if my logic is off, or my question is not clear :)
I'm not convinced that this is a snapshot. Each survey response represents a "transaction".
With an appropriate date dimension you can calculate the Yes/No percentages, rolled up by week.
Further, this would enable you to show results like "Surveys issued on a Sunday night get more responses", or "People who respond on Friday are more likely to answer 'Yes'". (contrived examples)
Following clarification, this does look like a periodic snapshot. The example of a bank account balance is often used to describe a similar scenario.
A key feature of a periodic snapshot is that every combination of every dimension should be present. If your grain is monthly, then every month you record the fact, even if it has not changed from the previous month.
I think that is the key to your problem. Knowing that your grain may change from weekly to daily, make your grain daily. It does mean you'll be repeating the weekly value on every day of the week, but that is a true representation of your knowledge of the fact; on Wednesday you only knew that its value was the same as Monday.
If you design your ETL right, you won't need to make any changes when the daily updates begin.
Your second option is the one I'd choose in your place.
I have a requirement to store dates and durations arising from multiple different calendars. In particular I need to store dates that:
Span the change to Gregorian calendars in different countries at different times
Cover a historic period of at least 500 years
Deal with multiple types of calendar - lunar, solar, Chinese, Financial, Christian, UTC, Muslim.
Deal with the change, in the UK, of the year end from 31st March to 31st December, and comparable changes in other countries.
I also need to store durations which I have defined as the difference between two timestamps (date and time). This implies the need to be able to store a "zero" date - so I can store durations of, say, three and a half hours; or 10 minutes.
I have details of the computations needed. Firebird's timestamp is based on a date function that starts at January 1st, 100 CE, so is not capable of being used for durations in the way I need to record them. In addition this data type is geared up (like most timestamp functions) to record the number of days since a base date; it is not geared up to record calendar dates.
Could anyone suggest:
A data structure to store dates and durations that meet the above requirements OR
A reference to such a data structure OR
Offer guidelines to approach the structuring of such storage OR
Any points that may help me to a solution.
EDIT:
#Warren P has provided some excellent work in his responses. I obviously have not explained what I am seeking clearly enough, as his work concentrates on the computations and how to go about calculating these. All valuable and useful stuff, but not what I intended my question to convey.
I do have details of all the computations needed to convert between various representations of dates, and I have a fairly good idea of how to implement them (using elements such as Warren suggests). However, my requirement is to STORE dates which meet the various criteria listed above. Example: date to be stored - 'Third June 13 Charles II'. I am trying to determine an appropriate structure within which to store such dates.
EDIT:
I have amended my proposed schema. I have listed the attributes on each table, and defined the tables and attributes by examples, given in the third section of the entity box. I have used the example given in this question and answer in my definition by example, and have amended the example in my question to correspond. Although I have proved my schema by describing somebody else's example, this schema may still be over complicated; over analysed; miss some obvious simplification and may prove very difficult to implement (Indeed, it may be plain wrong). Any comments or suggestions would be most welcome.
If you are writing your own, as I assume you intend to, I would make a class that contains a TDateTime, and other fields, and I would base it on the functionality in the very nicely written mxDateTime extension for Python, which is very easily readable, open source, C code, that you could use to extract the gregorian calendar logic you are going to need.
Within certain limits, TDateTime is always right. It's epoch value (0) is December 30, 1899 at midnight. From there, you can calculate other julian day numbers. It supports negative values, and thus it will support more than 400 years. I believe you will start having to do corrections, at the time of the last Gregorian calendar reforms. If you go from Friday, 15 October 1582, and figure out its julian day number, and the reforms before and after that, you should be able to do all that you require. Be aware that the time of day runs "backwards" before 1899, but that this is purely a problem in human heads, the computer will be accurate, and will calculate the number of minutes and seconds, up to the limit of double precision floating point math for you. Stick with TDateTime as your base.
I found some really old BorlandPascal/TurboPascal code that handles a really wide range of dates here.
If you need to handle arabic, jewish, and other calendars, again, I refer you to Python as a great source of working examples. Not just the mxdatetime extension, but stuff like this.
For database persistence, you might want to base your date storage around julian day numbers, and your time as C-like seconds since midnight, if the maximum resolution you need is 1 second.
Here's a snippet I would start with, and do code completion on:
TCalendarDisplaySubtype = ( cdsGregorian,cdsHebrew,cdsArabic,cdsAztec,
cdsValveSoftwareCompany, cdsWhoTheHeckKnows );
TDateInformation = class
private
FBaseDateTime:TDateTime;
FYear,FMonth,FDay:Integer; // if -1 then not calculated yet.
FCalendarDisplaySubtype:TCalendarDisplaySubtype;
public
function SetByDateInCE(Y,M,D,h,m,s:Integer):Boolean;
function GetAsDateInCE(var Y,M,D,h,m,s:Integer):Boolean;
function DisplayStr:String;
function SetByDateInJewishCalendar( ... );
property BaseDateTime:TDateTime read FDateTime write FDateTime;
property JulianDayNumber:Integer read GetJulianDayNumber write SetJulianDayNumber;
property CalendarDisplaySubType:TCalendarDisplaySubtype;
end;
I see no reason to STORE both the julian day number, and the TDateTime, just use a constant, subtract/add from the Trunc(FBaseDateTime) value, and return that, in the GetJulianDayNumber,SetJulianDayNumber functions. It might be worth having fields where you calculate the year, month, day, for the given calendar, once, and store them, making the display as string function much simpler and faster.
Update: It looks like you're better at ER Modelling than me, so if you posted that diagram, I'd upvote it, and that would be it. As for me, I'd be storing three fields; A Datetime field that is normalized to modern calendar standards, a text field (free form) containing the original scholarly date in whatever form, and a few other fields, that are subtype lookup table Foreign keys, to help me organize, and search on dates by the date and subtype. That would be IT for me.
Only a partial answer but an important piece.
Since you are going to store dates in a very broad range where a lot of things happened to calendars, you need to accommodate for those changes.
The timezone database TZ-database and the Delphi TZDB wrapper around the TZ-database will be of big help.
It has a database with rules how timezones historically behave.
I know they are based on the current calendar schemes, and you need to convert to UTC first.
You need to devise something similar for the other calendar schemes you want to support.
Edit:
The scheme I'd use would be like this:
find ways for all your calendars to convert to/from UTC
store the calendar type
store the dates in their original format, and the source of the date (just in case your source screwed up, and you need to recalculate).
use the UTC conversions to go from your original through UTC to the calendar types in your UI
--jeroen