I need to extract packets within certain time ranges from a large pcap. And I found editcap's -A and -B option a perfect fit for this task except my target time ranges are in epoch time and -A/B requires time in format YYYY-MM-DD HH:MM:SS.
My question is when I convert epoch time to YYYY-MM-DD HH:MM:SS, what time zone should I use? (I am not sure if this is relevant but the large pcap I use is a merge of smaller pcaps captured from differnt time zones).
I tried tshark which allow filtering based on epoch time (frame.time_epoch>=X) but tshark seems to be resouce expensive and get constantly killed by the ubuntu server I used.
Will appreciate any help!
Use your system's time.
100% correct. The time is parsed and then fed to a routine (mtkime()) that converts a year/month/day/hour/minute/second value, in local time in the machine's timezone, to POSIX time ("Epoch time", where the "Epoch" is the UN*X/POSIX Epoch of 1970-01-01 00:00:00 UTC).
am I right that the capture timstamps are stored as epoch time in pcap internally
Yes.
and thus once the system time I feed into editcap get converted into epoch time, editcap can extract the right packets no matter which time zone the packets are captured from?
Yes.
Related
I trying to parse a date in ISO 8601 format and some moments aren't clear for me.
For example, I have the next date: 2020-04-16T07:16:34.858215+03:00 in Europe/Moscow timezone.
Does it mean 07:16 in Moscow time or 10:16? I mean do I need to add 3 hours to date or date is in Moscow time already and timezone just shows how it diffs from UTC?
P.S. I tried to find information about it but everywhere is just common format description without details.
The time specified is the local time, so 2020-04-16T07:16:34.858215+03:00 means 7:16am in Moscow, or 4:16am UTC.
Wikipedia has a good example which clarifies things to at least some extent:
The following times all refer to the same moment: "18:30Z", "22:30+04", "1130−0700", and "15:00−03:30". Nautical time zone letters are not used with the exception of Z. To calculate UTC time one has to subtract the offset from the local time, e.g. for "15:00−03:30" do 15:00 − (−03:30) to get 18:30 UTC.
It's really unfortunate that ISO-8601 talks about this as a time zone, when it's only a UTC offset - it definitely doesn't specify the actual time zone. (So you can't tell what the local time will be one minute later, for example.)
Why does Delphi use double (8 bytes) to store date and time instead of Int64 (8 byte as well)? As a double precision floating point is not an exact value, I'm curious wether the precision of a unix date and time stored in an Int64 value will be better than the precision of a Delphi date and time?
The simple explanation is that the Delphi TDateTime type maps directly to the OLE/COM date/time format.
Embarcadero chose to use an existing date/time representation rather than create yet another one, and selected, what was at the time, the most obvious platform native option.
A couple of useful articles on Windows date/time representations:
How to recognize different types of timestamps from quite a long way away, Raymond Chen
Eric's Complete Guide To VT_DATE, Eric Lippert
As far as precision goes, you would like to compare Unix time to TDateTime. Unix time has second precision for both 32 or 64 bit values. For values close to the epoch, a double has far greater precision. There are 86,400 seconds in a day, and there are many orders of magnitude more double values between 0 and 1, for instance. You need to go to around year 188,143,673 before the precision of Unix time surpasses that of TDateTime.
Although you have focused on the size of the types, the representation is of course crucially important. For instance, if instead of representing date as seconds after epoch, it was represented as milliseconds after epoch, then the precision of Unix time would be 1000 times greater. Of course the range would be reduced by 1000 times also.
You need to be wary of considering precision of these types in isolation. These types don't exist in isolation, and the source of the values is important. If the time is coming from a file system say, then that will in fact determine the precision of the value.
Since the previous Yahoo Finance download URL is not working anymore, I are now left with something like this:
https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1492524105&period2=1495116105&interval=1d&events=history&crumb=tO1hNZoUQeQ
Does anybody know how the period1 (and period2) translate into a date (and viceversa)
Any help appreciated!
Thanks!
It looks like they are just unix time stamps, or the seconds from the Epoch. Here is a website that can covert the information for you: http://www.unixtimestamp.com
This is oriented at recreating (kinda) the old functionality. I am running Fedora with Chrome.
(1) Figure out your user agent string for your web browser (I googled something like "what's my user agent string" and quickly found a page that would print it out for the browser you are using). It will be something like "Mozilla/5.0 (X11; Linux x86_64) ..."
(2) Install a package that will export a "cookies.txt" file for Chrome (also googled for something like "Chrome export cookies.txt" and quickly found a chrome extension that would export a cookies.txt for a page you are viewing).
(3) Go to the historical downloads page for a symbol of interest at Yahoo finance. Save the "crumb" in the download link (like you have above) and the cookies.txt for that page.
(4) Now you can use wget to get data. The command will be something like:
wget --load-cookies [THE COOKIES FILE YOU SAVED] -U "[THE USER AGENT STRING YOU FOUND]" -O [DESIRED OUTPUT CSV] https://query1.finance.yahoo.com/v7/finance/download/[THE SYMBOL YOU WANT HISTORICAL DATA FOR]?period1=[UNIX START TIME]\&period2=[UNIX END TIME]\&interval=1d\&events=history\&crumb=[THE CRUMB YOU SAVED]
The period1=... is the UNIX timestamp (seconds since 1970-Jan-1 00:00:00 GMT) of the start date and period2=... is the UNIX timestamp of the end date.
I was able to programmatically download a number of symbols this way. The column ordering of the resulting CSV file had changed from the old ichart API and the number of errors I found in the historical data was noticeably higher than the already quite high error rate in the data.
No guess on how long this will work or if it is stable over a long period of time. YMMV.
They are unix timestamps, or seconds since Epoch.
Quote taken from http://www.unixtimestamp.com/
What is the unix time stamp?
The unix time stamp is a way to track
time as a running total of seconds. This count starts at the Unix
Epoch on January 1st, 1970 at UTC. Therefore, the unix time stamp is
merely the number of seconds between a particular date and the Unix
Epoch. It should also be pointed out (thanks to the comments from
visitors to this site) that this point in time technically does not
change no matter where you are located on the globe. This is very
useful to computer systems for tracking and sorting dated information
in dynamic and distributed applications both online and client side
The following C# code helps to convert between DateTime to Unix Timestamp
//credits to ScottCher
//reference http://stackoverflow.com/questions/249760/how-to-convert-a-unix-timestamp-to-datetime-and-vice-versa
private static DateTime UnixTimestampToDateTime(double unixTimeStamp)
{
//Unix timestamp Is seconds past epoch
return new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc).AddSeconds(unixTimeStamp).ToLocalTime();
}
//credits to Dmitry Fedorkov
//reference http://stackoverflow.com/questions/249760/how-to-convert-a-unix-timestamp-to-datetime-and-vice-versa
private static double DateTimeToUnixTimestamp(DateTime dateTime)
{
//Unix timestamp Is seconds past epoch
return (dateTime.ToUniversalTime() - new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc)).TotalSeconds;
}
NOTE: make sure you round it up to zero decimal before substituting into period1 and period2
The period1 and period2 are just timestamps (POSIX timestamps). I think you can use any time of the date that you want to use. For example, for 2014-05-20, you can use any time from 00:00:00 to 23:59:59. (Update: Yahoo has just made a change here. You have to use a time that is after market close time, otherwise the data will miss the last day. I think 18:00:00 is working fine.)
I have put together some quick Python3 code to use that new API. Please check out GitHub for source code yahoo_quote_download.
> os:system_time(seconds).
1470361698
This is correct UTC timestamp, but I'm not sure if it is not system-related value. Is it?
How then I can ensure that same program ran on different machine will still operate in UTC timezone?
os:system_time returns the "OS System Time" which is defined to be approximately the "POSIX Time", the number of seconds since Epoch, which is defined to be 00:00:00 UTC, 1970-01-01, so it should always return the "UTC Timestamp". Erlang has excellent documentation about Time Correction, have a look at the OS System Time and POSIX Time sections.
OS System Time
The operating systems view of POSIX time. To retrieve it, call os:system_time(). This may or may not be an accurate view of POSIX time. This time may typically be adjusted both backwards and forwards without limitation. That is, time warps may be observed.
POSIX Time
Time since Epoch. Epoch is defined to be 00:00:00 UTC, 1970-01-01. A day in POSIX time is defined to be exactly 86400 seconds long. Strangely enough Epoch is defined to be a time in UTC, and UTC has another definition of how long a day is. Quoting the Open Group "POSIX time is therefore not necessarily UTC, despite its appearance". The effect of this is that when an UTC leap second is inserted, POSIX time either stops for a second, or repeats the last second. If an UTC leap second would be deleted (which has not happened yet), POSIX time would make a one second leap forward.
As the subject asks; do UNIX timestamps change in each timezone?
For example, if I sent a request to another email the other side of the world saying, "Send out an email when the time is 1397484936", would the other server's timestamp be 12 hours behind my own?
The definition of UNIX timestamp is time zone independent. The UNIX timestamp is the number of seconds (or milliseconds) elapsed since an absolute point in time, midnight of Jan 1 1970 in UTC time. (UTC is Greenwich Mean Time without Daylight Savings time adjustments.)
Regardless of your time zone, the UNIX timestamp represents a moment that is the same everywhere. Of course you can convert back and forth to a local time zone representation (time 1397484936 is such-and-such local time in New York, or some other local time in Djakarta) if you want.
The article at http://en.wikipedia.org/wiki/Unix_time is pretty impressive if you'd like a longer read.
Unix time is defined as the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970. So the answer is no
Unix timestamps do not change accross timezones, they are created for the purpose of having a standard time across globe.
NOTE:-
Timestamps are calculated on the basis of current time in the computer thus do not rely on them until and unless you are very sure about the time settings in the participating machines.
Someone stated that "UTC is Greenwich Mean Time without Daylight Savings time adjustments." This is simply untrue. GMT does not have Dayllight Savings Time. GMT is measured in Greenwich, England (at the Naval Obeservatory) [0 longitude, but not 0 lattitude]. UTC is measured at the equator [0 longitude and 0 lattitude - which happens to lie in the ocean off the cost of Africa].
What difference does it make? It doesn't make a difference in terms of "what time of day is it?" It does, however, make a difference in terms of calculating a year. Now you'd think a year would be measured based upon the location of the center (the core) of the earth, right? When the earth's core is back in the same location it was ~365 days ago, it has been a year. It isn't measured that way. It is measured by a specific location on the earth getting back to the same location (relative to the sun) that it was ~365 days ago. But the period of a day and a year don't divide evenly. Once the earth is back to about where it was a year ago, the earth isn't facing the same direction it was last year, so that spot on the earth isn't facing the same direction it was a year ago. Being further north, Greenwich isn't going to get back to the same spot (relative to the sun) that it was last year at the same time that 0 Lat / 0 Long is. So if you base the definition on Greenwith vs. 0/0, you get a, albeit slightly, different answer to the question "how many days are in a year". To put it another way, when a given spot on the earth gets back to where it was a year ago (relative to the Sun), the core of the earth isn't in the same spot it was a year ago, so what spot you pick matters because the core of the earth is going to be in a different spot (relative to the sun) than it was one year ago, if you pick a different spot on the earth.
Neither UTC nor GMT have daylight savings time. Europe/London time, the timezone that Greenwich resides in, does. But GMT does not. GMT is, what Americans would call a "Standard Time" - i.e. without DST.
Getting back to the question, Epoch time doesn't technically have a timezone. It is based on a particular point in time, which just so happens to line up to an "even" UTC time (at the exact beginning of a year and a decade, etc.). If that concept doesn't fit well in your brain, and if it helps to think of Epoch time as being in UTC, go right ahead. You're in good company and in the grand scheme of things, it really doesn't matter. You ever see those law suits where somoene is awarded $1. It's kind of a "you're right, but it doesn't really matter" type of verdict. If someone sued you for saying Epoch time is in the UTC timezone, they would win $1. That wouldn't buy them a cup of coffee at any Starbucks in any timezone on the planet.
IF both computers are set up correctly with their clocks set for the correct timezone and UTC values, they should return the same value.
Of course that's a big IF. There's almost certain to be a difference of at least a second, more often minutes between the time reported by two computers. And many computers are set up to have incorrect timezone settings, and will report their local time when asked a timestamp rather than UTC.
And in that lies the difference between theory and practice. In theory it's all the same, in practice you should not rely on it.
No, epoch timestamp should not change, because it has a fixed timezone which is UTC.
If you want to use a time object in other time zone, just look it up in libraries of the language you use, but do NOT try to add/substract a couple of hours from epoch timestamp and assume it's in another time zone, which will make things very confusing to other people, especially when you expose it in your API.
If you use C++, I recommend this library. I heard it will soon be added into standard library.
For all, I understand sometimes time object is hard to deal with and it looks easier to add/substruct on epoch timestamp. Please don't do it and do not persuade others to do it. A time object is much easier once you get used to it and can take care of time zone conversion easily without messing up with historical time zone changes due to politics/law etc...