I'm going to be using the Azure Storage REST API to create and retrieve images uploaded by users using my iOS app. I'd like a directory structure something like
container_name/user_Id/group_Id/item_Id/image.jpg
Each user can have multiple group_Ids and each item can have multiple images.
Is this even possible and if so, should each user have their own container or have them all under one container?
Please note that Azure Blob Storage doesn't really has directory structure in the server side. Instead, the structure is simply two level: container, blob.
However, there is a workaround: you can name your blobs with "virtual directory" prefix, just like container_name/user_Id/group_Id/item_Id in your example, and then list blobs under your container with prefix specified.
container_name/user_Id/group_Id/item_Id/image.jpg
As mentioned by #Zhaoxing Lu - Microsoft, By including path information in blob names, you can create a virtual directory structure you can organize and traverse as you would a traditional file system. The directory structure is virtual only--the only resources available in Blob storage are containers and blobs. However, the storage client library offers a CloudBlobDirectory object to refer to a virtual directory and simplify the process of working with blobs that are organized in this way
For example, consider the following set of block blobs in a container named photos:
photo1.jpg
2010/architecture/description.txt
2010/architecture/photo3.jpg
2010/architecture/photo4.jpg
2011/architecture/photo5.jpg
2011/architecture/photo6.jpg
2011/architecture/description.txt
2011/photo7.jpg
Full documentation can be found here
Related
I have CSV files in a directory of an S3 bucket. I would like to use all of the files as a single table in Dremio, I think this is possible as long as each file has the same header/columns as the others.
Do I need to first add an Amazon S3 data source using the UI or can I somehow add one as a Source using the Catalog API? (I'd prefer the latter.) The REST API documentation doesn't provide a clear example of how to do this (or I just didn't get it), and I have been unable to find how to get the "New Amazon S3 Source" configuration screen as shown in the documentation, perhaps because I've not logged in as an administrator?
For example, let's say I have a dataset split over two CSV files in an S3 bucket named examplebucket within a directory named datadir:
s3://examplebucket/datadir/part_0.csv
s3://examplebucket/datadir/part_1.csv
Do I somehow set the S3 bucket/path s3://examplebucket/datadir as a data source and then promote each of the files contained therein (part_0.csv and part_1.csv) as a Dataset? Is that sufficient to allow all the files to be used as a single table?
It turns out that this is only possible for admin users, normal users can't add a source. To do what I have proposed above you put the files into an S3 bucket which has already been configured as a Dremio source by an admin user. Then you promote the files or folder as a data source using the Dremio Catalog API.
When I attempt load data into BigQuery from Google Cloud Storage it asks for the Google Cloud Storage URI (gs://). I have reviewed all of your online support as well as stackoverflow and cannot find a way to identify the URL for my uploaded data via the browser based Google Developers Console. The only way I see to find the URL is via gsutil and I have not been able to get gsutil to work on my machine.
Is there a way to determine the URL via the browser based Google Developers Console?
The path should be gs://<bucket_name>/<file_path_inside_bucket>.
To answer this question more information is needed. Did you already load your data into GCS?
If not, the easiest would be to go to the project console, click on project, and Storage -> Cloud Storage -> Storage browser.
You can create buckets there and upload files to the bucket.
Then the files will be found at gs://<bucket_name>/<file_path_inside_bucket> as #nmore says.
Couldn't find a direct way to get the url. But found an indirect way and below are the steps:
Go to GCS
Go into the folder in which the file has been uploaded
Click on the three dots at the right end of your file's row
Click rename
Click on gsutil equivalent link
Copy the url alone
Follow the following steps :
1. Go to GCS
2. Go into the folder in which the file has been uploaded
3. On the top you can see overview option
4. You can see there will be Link URL and link for GSUtil
Retrieving the Google Cloud Storage URI
To create an external table using a Google Cloud Storage data source, you must provide the Cloud Storage URI.
The Cloud Storage URI comprises your bucket name and your object (filename). For example, if the Cloud Storage bucket is named mybucket and the data file is named myfile.csv, the bucket URI would be gs://mybucket/myfile.csv. If your data is separated into multiple files you can use a wildcard in the URI. For more information, see Cloud Storage Request URIs.
BigQuery does not support source URIs that include multiple consecutive slashes after the initial double slash. Cloud Storage object names can contain multiple consecutive slash ("/") characters. However, BigQuery converts multiple consecutives slashes into a single slash. For example, the following source URI, though valid in Cloud Storage, does not work in BigQuery: gs://[BUCKET]/my//object//name.
To retrieve the Cloud Storage URI:
Open the Cloud Storage web UI.
CLOUD STORAGE WEB UI
Browse to the location of the object (file) that contains the source data.
At the top of the Cloud Storage web UI, note the path to the object. To compose the URI, replace gs://[BUCKET]/[FILE] with the appropriate path, for example, gs://mybucket/myfile.json. [BUCKET] is the Cloud Storage bucket name and [FILE] is the name of the object (file) containing the data.
If you need help on subdirectories, check this out on https://cloud.google.com/storage/docs/gsutil/addlhelp/HowSubdirectoriesWork
And https://cloud.google.com/storage/images/gsutil-subdirectories-thumb.png, if you need to see how gsutil provides a hierarchical view of objects in a bucket.
I need a storage system with the following requirements:
1. It should support data/service clustering
2. It should be open-source so that I can extend functionalities later if needed
3. It should support file system because I want to access some files as public url(direct access). So that I can store my scripts in these files and directly refer these files.
4. Supports some kind of authentication
5. I want it to be on premise (Not cloud).
Ceph seems to qualify all the criteria but does it support the public access of files just like a URL(Point 3) ? It has ability to generate temporary URLs though but I want permanent URLs for few files.
You could run Nextcloud and have your data volume (and database, if you feel so inclined) stored on the Ceph cluster. That's open-source, you can setup direct links to files including permanent links, and is authenticated.
In the executable I am reverse-engineering, there are several references to a path in my D:\ drive. However, I do not have a D:\ drive connected. Is it possible that it creates a temporary storage site in the executable?
For example, there is a string:
D:\BuildAgent\...\bin\...\fileIWantToSee.jpg
IDA even believes that the symbol information is in the D drive, and attempts to look for it, to no avail. There are many instances of file references within these strings, and many of them end with a:
Line: **LINENUMBER**
Where would I go about trying to find where this storage is located? Thank you!
EDIT: Could it be in a specific section?
Is it possible that it creates a temporary storage site in the executable?
This is possible. There exists at least one product (http://www.boxedapp.com/, kind of our competitor :) that lets the application create such container -- the calls to file APIs are intercepted by the code added to the application by this product, and this added code handles specific paths in a different way (emulating file operations), letting all other calls go to Windows API.
Imagine that a user uploads a file called "myAssignment.docx" to Moodle to complete the submission of an assignment. Where would the folder containing "myAssignment.docx" be? Is it in the moodledata folder outside the public web directory or somewhere else? I am trying to locate that folder because I want to monitor its growth to determine when I should upgrade my web hosting service to provide more storage space for files that students upload to the server.
https://docs.moodle.org/dev/File_API_internals#File_storage_on_disk
The simple answer is inside:
$CFG->dataroot.'/filedir'
It is stored in /repository directory from Ver 3.62.
You may have subdirectories. Check which one is used for the purpose.