When I attempt load data into BigQuery from Google Cloud Storage it asks for the Google Cloud Storage URI (gs://). I have reviewed all of your online support as well as stackoverflow and cannot find a way to identify the URL for my uploaded data via the browser based Google Developers Console. The only way I see to find the URL is via gsutil and I have not been able to get gsutil to work on my machine.
Is there a way to determine the URL via the browser based Google Developers Console?
The path should be gs://<bucket_name>/<file_path_inside_bucket>.
To answer this question more information is needed. Did you already load your data into GCS?
If not, the easiest would be to go to the project console, click on project, and Storage -> Cloud Storage -> Storage browser.
You can create buckets there and upload files to the bucket.
Then the files will be found at gs://<bucket_name>/<file_path_inside_bucket> as #nmore says.
Couldn't find a direct way to get the url. But found an indirect way and below are the steps:
Go to GCS
Go into the folder in which the file has been uploaded
Click on the three dots at the right end of your file's row
Click rename
Click on gsutil equivalent link
Copy the url alone
Follow the following steps :
1. Go to GCS
2. Go into the folder in which the file has been uploaded
3. On the top you can see overview option
4. You can see there will be Link URL and link for GSUtil
Retrieving the Google Cloud Storage URI
To create an external table using a Google Cloud Storage data source, you must provide the Cloud Storage URI.
The Cloud Storage URI comprises your bucket name and your object (filename). For example, if the Cloud Storage bucket is named mybucket and the data file is named myfile.csv, the bucket URI would be gs://mybucket/myfile.csv. If your data is separated into multiple files you can use a wildcard in the URI. For more information, see Cloud Storage Request URIs.
BigQuery does not support source URIs that include multiple consecutive slashes after the initial double slash. Cloud Storage object names can contain multiple consecutive slash ("/") characters. However, BigQuery converts multiple consecutives slashes into a single slash. For example, the following source URI, though valid in Cloud Storage, does not work in BigQuery: gs://[BUCKET]/my//object//name.
To retrieve the Cloud Storage URI:
Open the Cloud Storage web UI.
CLOUD STORAGE WEB UI
Browse to the location of the object (file) that contains the source data.
At the top of the Cloud Storage web UI, note the path to the object. To compose the URI, replace gs://[BUCKET]/[FILE] with the appropriate path, for example, gs://mybucket/myfile.json. [BUCKET] is the Cloud Storage bucket name and [FILE] is the name of the object (file) containing the data.
If you need help on subdirectories, check this out on https://cloud.google.com/storage/docs/gsutil/addlhelp/HowSubdirectoriesWork
And https://cloud.google.com/storage/images/gsutil-subdirectories-thumb.png, if you need to see how gsutil provides a hierarchical view of objects in a bucket.
Related
I have found documentation for uploading an image from iOS to a blob container in Azure - https://learn.microsoft.com/en-us/azure/storage/blobs/storage-ios-how-to-use-blob-storage
using the https://github.com/Azure/azure-storage-ios library
But I wish to upload directly to a file share. Is there a way to do this?
It needs to be implemented using SAS authentication.
Unfortunately I am not familiar with iOS programming thus I will not be able to provide you any code. However you can use the steps below to write code.
Assuming you have a SAS URL for the file share in which you wish to upload the file, you can simply use Azure Storage REST API to upload the file in a file share. You should be able to use built-in HTTP functionality in the programming language of your choice to do that.
Let's assume that you have a SAS URL for the file share in the following format: https://<account-name>.file.core.windows.net/<share-name>?<sas-token>.
First thing you would need to do is insert the file name that you wish to upload in this SAS URL so that you get a SAS URL for the file. Your SAS URL would look something like: https://<account-name>.file.core.windows.net/<share-name>/<file-name>?<sas-token>.
Next you would need to create an empty file. You will use Create File REST API operation. Do not worry about the Authorization request header there as it is already included in the SAS. Only request header you would need to include is x-ms-content-length value of which should be the size of the file you want to upload. This will create an empty file having size as that of the file you want to upload.
Once this operation completes, next you would need to upload the data in the empty file you just created. You will use Put Range operation. The request headers you need to include are x-ms-range (value of which should be bytes=0-file-length - 1) and Content-Length (value of which should be the length of your file). The request body will contain the file contents.
Using these steps you should be able to upload a file in a file share.
I have CSV files in a directory of an S3 bucket. I would like to use all of the files as a single table in Dremio, I think this is possible as long as each file has the same header/columns as the others.
Do I need to first add an Amazon S3 data source using the UI or can I somehow add one as a Source using the Catalog API? (I'd prefer the latter.) The REST API documentation doesn't provide a clear example of how to do this (or I just didn't get it), and I have been unable to find how to get the "New Amazon S3 Source" configuration screen as shown in the documentation, perhaps because I've not logged in as an administrator?
For example, let's say I have a dataset split over two CSV files in an S3 bucket named examplebucket within a directory named datadir:
s3://examplebucket/datadir/part_0.csv
s3://examplebucket/datadir/part_1.csv
Do I somehow set the S3 bucket/path s3://examplebucket/datadir as a data source and then promote each of the files contained therein (part_0.csv and part_1.csv) as a Dataset? Is that sufficient to allow all the files to be used as a single table?
It turns out that this is only possible for admin users, normal users can't add a source. To do what I have proposed above you put the files into an S3 bucket which has already been configured as a Dremio source by an admin user. Then you promote the files or folder as a data source using the Dremio Catalog API.
I'm going to be using the Azure Storage REST API to create and retrieve images uploaded by users using my iOS app. I'd like a directory structure something like
container_name/user_Id/group_Id/item_Id/image.jpg
Each user can have multiple group_Ids and each item can have multiple images.
Is this even possible and if so, should each user have their own container or have them all under one container?
Please note that Azure Blob Storage doesn't really has directory structure in the server side. Instead, the structure is simply two level: container, blob.
However, there is a workaround: you can name your blobs with "virtual directory" prefix, just like container_name/user_Id/group_Id/item_Id in your example, and then list blobs under your container with prefix specified.
container_name/user_Id/group_Id/item_Id/image.jpg
As mentioned by #Zhaoxing Lu - Microsoft, By including path information in blob names, you can create a virtual directory structure you can organize and traverse as you would a traditional file system. The directory structure is virtual only--the only resources available in Blob storage are containers and blobs. However, the storage client library offers a CloudBlobDirectory object to refer to a virtual directory and simplify the process of working with blobs that are organized in this way
For example, consider the following set of block blobs in a container named photos:
photo1.jpg
2010/architecture/description.txt
2010/architecture/photo3.jpg
2010/architecture/photo4.jpg
2011/architecture/photo5.jpg
2011/architecture/photo6.jpg
2011/architecture/description.txt
2011/photo7.jpg
Full documentation can be found here
Currently I am using a saved_model file stored on my local disk to read an inference graph and use it in servers. Unfortunately giving a GCS path doesn't work for SavedModelBundle.load api.
Tried providing GCS path for the file but did not work.
Is this even supported, if not how can i achieve this using the SavedModelBundle api because i have some production servers running on google cloud that i want to serve some tensor-flow graphs.
A recent commit inadvertently broke the ability to load files from GCS. This has been fixed and is available in github.
I am working on Amazon S3 sdk for storing files on cloud server,i am using codeplex's threesharp(http://threesharp.codeplex.com) for implementing this, I have successfully uploaded file on server now i have to download it, and for this i have to download it with the URL eg https://s3.amazonaws.com/MyBucket/Filename
I can download the uploaded file but it is appearing blank, if i upload a text file then after downloading it's showing nothing in it,same as images and other files. I have read on Amazon S3 documentation that i'll have to make the object publically readable(http://docs.amazonwebservices.com/AmazonS3/latest/gsg/OpeningAnObject.html) i dont have any idea how to achieve this.
How can i accomplish the download functionality?
Threesharp project is a desktop based and i am working on web based application
During file upload set proper ACL:
Eg.:
AmazonS3 client = GetS3Client();
SetACLRequest request = new SetACLRequest();
request.BucketName = "my-new-bucket";
request.Key = "hello.txt";
request.CannedACL = S3CannedACL.PublicRead;
client.SetACL(request)
Amazon S3 provides a rich set of mechanisms for you to manage access to your buckets and objects.
Check this for detail: Amazon S3 Bucket Public Access Considerations
Also, You can Download Explorer for Amazon S3 (Eg. CloudBerry Explorer for Amazon S3) & they you can assign appropriate rights to your buckets.
CloudBerry Explorer for Amazon S3: Data Access Feature:
Bucket Policy Editor
Create and edit conditional rules for managing access to the buckets and objects.
ACL Editor
Manage access permission to any of your objects by setting up 'Access Control List'. ACL will also apply to all 'child objects' inside S3 buckets.
Also, you can do the same using Amazon S3 admin console.
Eg.
Have you tried the following:
Right-click the object and click Make public
Select the object and in the Permissions section checked Open/Download ?
edit:
have you taken a look here:
How to set the permission on files at the time of Upload through Amazon s3 API
and here:
How to set a bucket's ACL on S3?
It might guide you in the right direction