I'm storing some data in a Go app in a struct's vector.Vector for convenience. I want to display all the data from the vector on Google App Engine webpage through template.Execute. Is it possible and how would I access the data in the parsed html file? Would it be easier if I used an array or slice instead?
Use slices.
Go Weekly Snapshot History 2011-10-18
The container/vector package has been deleted. Slices are better:
SliceTricks.
Related
I want to create a 3d dask array from data that I have that is already chunked. My data consists of 216 blocks containing 1024x1024x1024 uint8 voxels each, each stored as a compressed hdf5 file with one key called data. Compressed, my data is only a few megabytes per block, but decompressed, it takes 1GB per block. Furthermore, my data is currently stored in Google Cloud storage (gcs), although I could potentially mirror it locally inside a container.
I thought the easiest way would be to use zarr, following these instructions (https://pangeo.io/data.html). Would xarray have to decompress my data before saving to zarr format? Would it have to shuffle data and try to communicate across blocks? Is there a lower level way of assembling a zarr from hdf5 blocks?
There are a few questions there, so I will try to be brief and hope that some edits can flesh out details I may have omitted.
You do not need to do anything in order to view your data as a single dask array, since you can reference the individual chunks as arrays (see here) and then use the stack/concatenate functions to build up into a single array. That does mean opening every file in the client, in order to read the meatadata, though.
Similarly, xarray has some functions for reading sets of files, where you should be able to assume consistency of dtype and dimensionality - please see their docs.
As far as zarr is concerned, you could use dask to create the set of files for you on GCS or not, and choose to use the same chunking scheme as the input - then there will be no shuffling. Since zarr is very simple to set up and understand, you could even create the zarr dataset yourself and write the chunks one-by-one without having to create the dask array up front from the zarr files. That would normally be via the zarr API, and writing a chunk of data does not require any change to the metadata file, so can be done in parallel. In theory, you could simply copy a block in, if you understood the low-level data representation (e.g., int64 in C-array layout); however, I don't know how likely it is that the exact same compression mechanism will be available in both the original hdf and zarr (see here).
I want to download DBLP dataset, which consists of bibliographic data in computer science.
I want to select a list of conferences from two research areas i.e., computer security (ISI, NDSS, ARES, ACSAC FC, and SP) and information retrieval (AIRS, CIKM, SIGIR, JCDL, ICTIR, ECIR, TREC, and WSDM).
Although DBLP dataset is available on https://aminer.org/citation (V4), I want to avoid parsing by using query as we use on Web of Science.
Get the DBLP XML dump from https://dblp.org/faq/1474679.html
This is the recommended way to extract larger parts from DBLP. You can easily get per-author bibliographies, but not entire conference series, except by using this.
Then 3xyradt whatever parts you want to use.
I have a data ([x1,y1],....,[xn,yn])
where $n$ is large, around 700.
I want to plot these data in wxmaxima.
The question is:
can I store these data in an exterior .txt or .wxmx file and call it from within the code.
How achieve this?
Thanks a lot
If the data are coming from some external source, the function read_nested_list can read the data as a list of lists of 2 elements [[x1, y1], [x2, y2], ...]. There are other functions for reading from a file -- ?? read will find the documentation for them.
If the data are generated in Maxima, you can save them to a plain text file via write_data and then read them with read_nested_list or another read function. Another option is to save them via save("myfile.lisp", mydata) which saves mydata as Lisp expressions. Then you can say load("myfile.lisp") to restore mydata in another session.
I am collecting GIS data consisting of normalized four values for whole world. I am curious on what would be the best way to store this data and wanted to take your advise. Would it be more efficient (in terms of size) to store the four values of the quadtree, along with a Geohash index via Z-order (Morton) or Hilbert curve? Or would it be more efficient to store it in a PNG file using alpha = 0 for empty spaces and lossless compression? The enclosed image 1 only visualizes one of the four values over Google Maps and I need to store this global data each day. Please, note that I will only store leaf nodes as visualized in the image 1 rather than the whole quadtree. I will also store this over time so I would also like to know your ideas about how video compression would improve.
Thank you all in advance for your time and consideration!
New user of hadoop and mapreduce, i would like to create a mapreduce job to do some measure on images. this why i would like to know if i can passe an image as input to mapreduce?if yes? any kind of example
thanks
No.. you cannot pass an image directly to a MapReduce job as it uses specific types of datatypes optimized for network serialization. I am not an image processing expert but I would recommend to have a look at HIPI framework. It allows image processing on top of MapReduce framework in a convenient manner.
Or if you really want to do it the native Hadoop way, you could do this by first converting the image file into a Hadoop Sequence file and then using the SequenceFileInputFormat to process the file.
Yes, you can totally do this.
With the limited information provided, I can only give you a very general answer.
Either way, you'll need to:
1) You will need to write a custom InputFormat that instead of taking chunks of files in HDFS locations (like TextInputFormat and SequenceFileInputFormat do), it actually passes to each map task the Image's HDFS path name. Reading the image from that won't be too hard.
If you plan to have a Reduce phase in which Images are passed around through the framework, you'll need to:
2) You will need to make an "ImageWritable" class that implements Writable (or WritableComparable if you're keying on the image). In your write() method, you'll need to serialize your image to a byte array. When you do this, what I would do is first write to the output an int/long which is the size of the array you're going to write. Lastly, you'll want to write the array as bytes.
In your read() method, you'll read an int/long first (which will describe the payload of the image), create an byte array of this size, and then read the bytes fully into your byte array up to the length of your int/long that you captured.
I'm not entirely sure what you're doing, but that's how I'd go about it.