I have an HDF5 files, global coverage of temperature. The file was converted from netcdf. The conversion process set longitude from 0 to 360 and additionally flipped the map upside down, so north is now south. I have used HDFView and I can display the file but there is no way to interact with the map so locate a specific lat/long combination. The file doesn't display properly in arcmap even after setting the correct projection.
Is there anyway I can display the data and click on a location and extract lat/long or draw a point in a specific lat/long?
Short answer: No, that's not possible.
Long answer: Unlike NetCDF, HDF5 is a general purpose file format. It allows you to store n-dimensional numerical arrays (called datasets), grouped into folders (hence the name "hierarchical"). Nothing more. There is no semantics. To HDF5, your data is not a "map", it's just an array. Therefore, HDFView does not "know" about latitudes and longitudes. That information was lost in the NetCDF => HDF5 conversion process. Actually, the lat/lon arrays are probably still in the file but they no longer have any inherent meaning. NetCDF, on the other hand, imposes a common data model including coordinate systems. That's why the various visualization tools let you interact with your data in a more sophisticated way.
What tool did you use to convert your NetCDF-file to HDF5?
You can use HDF5 to store meteorological data (I do that, it works well). But then you have to write your own tools for georeferencing and visualization. Check out the h5py project if you're into Python.
As #heron13 has said, HDF5 is a file format.
What version of NetCDF was your file? As version 4 uses an enhanced version of HDF5 as the storage layer.
Does your NetCDF file follow (have) the CF conventions or COARDS conventions? If so I would look at the program you used to convert it to HDF5, as HDF5 can support the same conventions. For example.
Once you confirm that the conventions are in the HDF5 file, arcmap is meant to support them too (sorry I do not have access to arcmap to confirm).
Here's a look at a NetCDF file with the CF conventions:
$ ncdump tos_O1_2001-2002.nc | less
netcdf tos_O1_2001-2002 {
dimensions:
lon = 180 ;
lat = 170 ;
time = UNLIMITED ; // (24 currently)
bnds = 2 ;
variables:
double lon(lon) ;
lon:standard_name = "longitude" ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;
lon:axis = "X" ;
lon:bounds = "lon_bnds" ;
lon:original_units = "degrees_east" ;
...
While here is a view of the same file only using h5dump:
$ h5dump tos_O1_2001-2002.nc | less
HDF5 "tos_O1_2001-2002.nc" {
GROUP "/" {
ATTRIBUTE "Conventions" {
DATATYPE H5T_STRING {
STRSIZE 6;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "CF-1.0"
}
}
...
One other question, is there any reason why you are not using the NetCDF file in arcmap?
Related
my task is to create a database with all street names and town names in Germany. As this is a large query I chose to download the pbf file with the python pyrosm package. Once I unpack the data with OSM() and use get_network() I run into memory issues as the loaded DataFrame is to large. See here for roads (this works for smaller areas such as regions in germany):
from pyrosm import get_data
from pyrosm import OSM
import pandas as pd
#Downloading the germany Data
de = get_data("germany")
#Turning it into an OSM object
de_osm = OSM(de)
#Extracting all driving objects, e.g. roads
roads = osm_object.get_network(network_type="driving")
#Extract all road names and turning it into a list
road_names = pd.Series(roads.name).values
road_names = list(road_names)
I wanted to solve this problem with generator functions but I cant seem to iterate over the data like i would with a csv file. Here are my attempts that failed:
osm.object= (OSM(obj) for obj in de)
#Extracting all driving objects, e.g. roads
roads = osm_object.get_network(network_type="driving")
#Extract all road names and turning it into a list
road_names = pd.Series(roads.name).values
road_names = list(road_names)
Alternative:
def generator_osm():
for i in OSM(de).get_network(network_type="driving"):
yield i
res = generator_osm()
#Extract all road names and turning it into a list
road_names = list()
for i in res:
road_names = road_names.append(pd.Series(i.name).values)
Thank you in advance for any tipps you can provide :)
I would suggest to use pyosmium. It allows you to analyse osm files easily without having to deal with the geometry. I tried pyrosm a bit and I think it tries to create a road network when using .get_network(…), which is unnecessary if you only want to know what names the roads objects in your osm files have.
I took an example of the pyosmium documentation and applied it to collecting road names in a short example:
import osmium
from collections import Counter
# handler that processes your file
class RoadNameHandler(osmium.SimpleHandler):
def __init__(self):
super(RoadNameHandler, self).__init__()
self.road_names = []
def way(self, o):
if 'highway' in o.tags and 'name' in o.tags:
self.road_names.append(o.tags['name'])
# process file
h = RoadNameHandler()
h.apply_file("germany-latest.osm.pbf")
# some examples to print & count the names
print(h.road_names)
print(Counter(h.road_names))
print(len(h.road_names))
This script did not take more than 500-600 MB of memory, the pbf file had to be downloaded manually from Geofabrik.
P.S.: If you want to have a distinct list of the road names you can either use a Counter() or a set instead of the list for self.road_names.
I have a set of HDF5 files that have the following header:
netcdf control-A-2017-05-12-090000-g1 {
dimensions:
phony_dim_0 = 16 ;
phony_dim_1 = 16 ;
phony_dim_2 = 200 ;
phony_dim_3 = 2 ;
phony_dim_4 = 1 ;
phony_dim_5 = 4 ;
variables:
...
Since it's HDF5, the dimensions are created as phony_dim_x. In this case, phony_dim_0 and phony_dim_1 are the y and x coordinates, respectively. I would like to rename the dimensions appropriately. Since renaming dimensions in HDF5 is not possible (since they don't technically exist), I need to convert to netcdf first. to do so I use ncks in.h5 out.nc.
However, the header info of the converted file is:
netcdf control-A-2017-05-12-090500-g1 {
dimensions:
phony_dim_0 = 16 ;
phony_dim_1 = 200 ;
phony_dim_2 = 2 ;
phony_dim_3 = 1 ;
phony_dim_4 = 4 ;
variables:
...
Here's the important part: the two phony_dim_[0,1] dimensions have been combined to a single dimension, phony_dim_0. I assume this is because they have the same value, and so the netcdf conversion assumes they're the same.
A variable that was listed in the hdf5 file as
ACCPA(phony_dim_0, phony_dim_1) ; is now ACCPA(phony_dim_0, phony_dim_0) ;, with two identical dimensions.
Thus, I am not able to rename the dimensions individually. If I do ncrename -d phony_dim_0,y out.nc, I get ACCPA(y, y) ;
Can anyone point me in the right direction to get around this?
The problem ended up being with ncks. Converting the file with ncks resulting in repeated dimensions (e.g. ACCPA(phony_dim_0, phony_dim_0) ;)
Using nccopy instead, the converted netCDF file did not produce repeated dimensions (ACCPA(phony_dim_0, phony_dim_1) ;)
i have shape file (.shp) with data in EPSG:2180.
I have extracted positions like this one:
303553.0249270061 580466.2644065879
Same values i see in equivalent .gml file.
That i am sure values are ok.
But how can i convert it to Latitude and Longitude?
(from comments GPS is WGS 84 (a.k.a. EPSG:4326)).
i see that params about EPSG:2180:
latitude_of_origin=0
central_meridian=19
scale_factor=0.9993
false_easting=500000
false_northing=-5300000
SPHEROID = 6378137,298.257222101
degree=0.0174532925199433
I have tried such simple calculation like
one_degree = 111196.672; //meters
X:= 303553.0249270061;
Y:= 580466.2644065879;
X:= X - false_northing;
Y:= Y - false_easting;
X:= latitude_of_origin + X/one_degree;
Y:= central_meridian + Y/one_degree;
but this show me:
50.3931720629823
19.7236391427847
which is not true. It is near but should be >20.
How this calculation should looks like?
I need it in Delphi application.
I don't know solution for You in Delphi but i found one in js and php.
I found this topic while i was searching "how to transform location units from one system to another" and google EPSG:2180 convert.
This problem name is:
Transform a point coordinate from one map projection to another
So solution is to make mathematical transformation and maby other calculations. You can Check:
Proj4 Library
with API Interfaces: C, C++, Python, Java, Ruby
My solution was to use library Proj4js.
Maby try to open source's on github of this library and analize function transform and with luck You will find answer ; )
I used php version of this package => Proj4jsphp
My problem was how to transform Coordinates in EPSG:2180 => GPS is WGS 84 (a.k.a. EPSG:4326).
Later i found out there are different names for this systems:
EPSG:2180 responds to Poland CS92
EPSG:4326 responds to WGS84 -> where WGS84 is the most popular system (lat, lon).
I am trying to represent in an ontology a few geometric objects (polygon, lines, points, etc.) and calculate their spatial/topological relations, through the adoption of GeoSPARQL relevant functions (sfTouches, sfEquals, sfContains, etc.). I am using GraphDB, with the GeoSPARQL plugin enabled.
I have seen that in the WKT representation of the geometric object, GeoSPARQL uses the concept of a default spatial reference system (i.e. the <http://www.opengis.net/def/crs/OGC/1.3/CRS84> URI which corresponds to the WGS84 coordinate reference system (CRS)). However, in my use case, the coordinates of the geometrical objects actually correspond to values in a 2D Cartesian coordinate system.
I found in the EPSG Geodetic Parameter Registry the proper CRS for representing Cartesian coordinates and I attached the proper URI in the WKT representation, but the GeoSPARQL functions do not return any result or error.
My question is the following: "Do the GeoSPARQL functions operate properly when representing spatial objects in any other type of CRS, apart from the default one?".
Thank you in advance.
Currently GDB does not support alternative CRS in WKT literals but supports them in GML literals (issue GDB-3142). GML literals are slightly more complex but still easy enough to generate, let us know if you need help with that.
However, I question your assertion that you have Cartesian coordinates. On one hand, any pair (lat,long) or (nothing,easting) is a Cartesian coordinate. On the other hand, since the Earth is not flat, any CRS or projection method is only an approximation, and many of them are tuned for specific localities.
So please tell us which EPSG CRS you picked, and a bit about the locality of your data.
Your example, slightly reformatted, and using normal turtle shortenings:
ex:polygon_ABCD rdf:type ex:ExampleEntity ;
geo:hasGeometry ex:geometry_polygon_ABCD .
ex:geometry_polygon_ABCD a geo:Geometry, sf:Polygon ;
geo:asWKT "<opengis.net/def/cs/EPSG/0/4499> Polygon((389.0 1052.0, 563.0 1052.0, 563.0 1280.0, 389.0 1280.0, 389.0 1052.0))"^^geo:wktLiteral .
ex:point_E rdf:type ex:ExampleEntity ;
geo:hasGeometry ex:geometry_point_E .
ex:geometry_point_E a geo:Geometry, sf:Point ;
geo:asWKT "<opengis.net/def/cs/EPSG/0/4499> Point(400.0 1100.0)"^^geo:wktLiteral ; .
You must use a specific URL for the CRS and cannot omit http:, so the correct URL is http://www.opengis.net/def/crs/EPSG/0/4499.
But you can see from the returned description that this CRS is applocable to "China - onshore and offshore between 120°E and 126°E". I'm not an expert in geo projections so I can't guarantee whether this CRS will satisfy your need "leave my coordinates alone, they are just meters". I'd look for a UK (OrdnanceSurvey) CRS with easting and northing coordinates.
To learn how to format GML:
see the GeoSPARQL spec (OGC 11-052r4) p18, whchc gives an example about gml:Point.
then google for gml:Polygon. There are many links but one that gives examples is http://www.georss.org/gml.html
Armed with this knowledge, we can reformat your example to GML:
ex:polygon_ABCD rdf:type ex:ExampleEntity ;
geo:hasGeometry ex:geometry_polygon_ABCD .
ex:geometry_polygon_ABCD a geo:Geometry, sf:Polygon ;
geo:asGML """
<gml:Polygon xmlns:gml="http://www.opengis.net/gml" srsName="http://www.opengis.net/def/crs/EPSG/0/TODO">
<gml:exterior>
<gml:LinearRing>
<gml:posList>
389.0 1052.0, 563.0 1052.0, 563.0 1280.0, 389.0 1280.0, 389.0 1052.0
</gml:posList>
</gml:LinearRing>
</gml:exterior>
</gml:Polygon>
"""^^geo:gmlLiteral.
ex:point_E rdf:type ex:ExampleEntity ;
geo:hasGeometry ex:geometry_point_E .
ex:geometry_point_E a geo:Geometry, sf:Point ;
geo:asGML """
<gml:Point xmlns:gml="http://www.opengis.net/gml" srsName="http://www.opengis.net/def/crs/EPSG/0/TODO">
<gml:pos>
400.0 1100.0
</gml:pos>
</gml:Point>
"""^^geo:gmlLiteral.
The """ (long quote) allows us to use " inside the literal without quoting
replace TODO with the better CRS you picked
the documentation http://graphdb.ontotext.com/documentation/master/enterprise/geosparql-support.html#geosparql-examples gives an example similar to yours but it cheats a bit because all coordinates are in the range (-90,+90) so it can just use WGS.
after you debug using geof: topology functions, turn on indexing and switch to geo: predicates, because the functions are slow (they check every geometry against every other) while the predicates use the special geo index
Let me know how it goes!
I'm getting ready to release a tool that is only effective with regular hard drives, not SSD (solid state drive). In fact, it shouldn't be used with SSD's because it will result in a lot of read/writes with no real effectiveness.
Anyone knows of a way of detecting if a given drive is solid-state?
Finally a reliable solution! Two of them, actually!
Check /sys/block/sdX/queue/rotational, where sdX is the drive name. If it's 0, you're dealing with an SSD, and 1 means plain old HDD.
I can't put my finger on the Linux version where it was introduced, but it's present in Ubuntu's Linux 3.2 and in vanilla Linux 3.6 and not present in vanilla 2.6.38. Oracle also backported it to their Unbreakable Enterprise kernel 5.5, which is based on 2.6.32.
There's also an ioctl to check if the drive is rotational since Linux 3.3, introduced by this commit. Using sysfs is usually more convenient, though.
You can actually fairly easily determine the rotational latency -- I did this once as part of a university project. It is described in this report. You'll want to skip to page 7 where you see some nice graphs of the latency. It goes from about 9.3 ms to 1.1 ms -- a drop of 8.2 ms. That corresponds directly to 60 s / 8.2 ms = 7317 RPM.
It was done with simple C code -- here's the part that measures the between positions aand b in a scratch file. We did this with larger and larger b values until we have been wandered all the way around a cylinder:
/* Measure the difference in access time between a and b. The result
* is measured in nanoseconds. */
int measure_latency(off_t a, off_t b) {
cycles_t ta, tb;
overflow_disk_buffer();
lseek(work_file, a, SEEK_SET);
read(work_file, buf, KiB/2);
ta = get_cycles();
lseek(work_file, b, SEEK_SET);
read(work_file, buf, KiB/2);
tb = get_cycles();
int diff = (tb - ta)/cycles_per_ns;
fprintf(stderr, "%i KiB to %i KiB: %i nsec\n", a / KiB, b / KiB, diff);
return diff;
}
This command lsblk -d -o name,rota lists your drives and has a 1 at ROTA if it's a rotational disk and a 0 if it's an SSD.
Example output :
NAME ROTA
sda 1
sdb 0
Detecting SSDs is not as impossible as dseifert makes out. There is already some progress in linux's libata (http://linux.derkeiler.com/Mailing-Lists/Kernel/2009-04/msg03625.html), though it doesn't seem user-ready yet.
And I definitely understand why this needs to be done. It's basically the difference between a linked list and an array. Defragmentation and such is usually counter-productive on a SSD.
You could get lucky by running
smartctl -i sda
from Smartmontools. Almost all SSDs has SSD in the Model field. No guarantee though.
My two cents to answering this old but very important question... If a disk is accessed via SCSI, then you will (potentially) be able to use SCSI INQUIRY command to request its rotational rate. VPD (Vital Product Data) page for that is called Block Device Characteristics and has a number 0xB1. Bytes 4 and 5 of this page contain a number with meaning:
0000h "Medium rotation rate is not reported"
0001h "Non-rotating medium (e.g., solid state)"
0002h - 0400h "Reserved"
0401h - FFFEh "Nominal medium rotation rate in rotations per minute (i.e.,
rpm) (e.g., 7 200 rpm = 1C20h, 10 000 rpm = 2710h, and 15 000 rpm = 3A98h)"
FFFFh "Reserved"
So, SSD must have 0001h in this field. The T10.org document about this page can be found here.
However, the implementation status of this standard is not clear to me.
I wrote the following javascript code. I needed to determine if machine was ussing SSD drive and if it was boot drive. The solution uses MSFT_PhysicalDisk WMI interface.
function main()
{
var retval= false;
// MediaType - 0 Unknown, 3 HDD, 4 SSD
// SpindleSpeed - -1 has rotational speed, 0 has no rotational speed (SSD)
// DeviceID - 0 boot device
var objWMIService = GetObject("winmgmts:\\\\.\\root\\Microsoft\\Windows\\Storage");
var colItems = objWMIService.ExecQuery("select * from MSFT_PhysicalDisk");
var enumItems = new Enumerator(colItems);
for (; !enumItems.atEnd(); enumItems.moveNext())
{
var objItem = enumItems.item();
if (objItem.MediaType == 4 && objItem.SpindleSpeed == 0)
{
if (objItem.DeviceID ==0)
{
retval=true;
}
}
}
if (retval)
{
WScript.Echo("You have SSD Drive and it is your boot drive.");
}
else
{
WScript.Echo("You do not have SSD Drive");
}
return retval;
}
main();
SSD devices emulate a hard disk device interface, so they can just be used like hard disks. This also means that there is no general way to detect what they are.
You probably could use some characteristics of the drive (latency, speed, size), though this won't be accurate for all drives. Another possibility may be to look at the S.M.A.R.T. data and see whether you can determine the type of disk through this (by model name, certain values), however unless you keep a database of all drives out there, this is not gonna be 100% accurate either.
write text file
read text file
repeat 10000 times...
10000/elapsed
for an ssd will be much higher, python3:
def ssd_test():
doc = 'ssd_test.txt'
start = time.time()
for i in range(10000):
with open(doc, 'w+') as f:
f.write('ssd test')
f.close()
with open(doc, 'r') as f:
ret = f.read()
f.close()
stop = time.time()
elapsed = stop - start
ios = int(10000/elapsed)
hd = 'HDD'
if ios > 6000: # ssd>8000; hdd <4000
hd = 'SSD'
print('detecting hard drive type by read/write speed')
print('ios', ios, 'hard drive type', hd)
return hd