I have a Azure Analytics Serverless Pool ( where I am the admin)
I also have got access to ADLS Gen-2 through ACL(Access Control List) for specific list of folders [ In the below image - all green boxes represents the access that I have ]
I am running below SQL statement on Synapse Studio ( Serverless Pool)
SELECT
TOP 100 *
FROM
OPENROWSET(
BULK 'https://SAFINCAL.dfs.core.windows.net/Container-2/Post-Sale/shipments/2021/*.parquet',
FORMAT='PARQUET'
) AS [result]
I am getting below error message
Started executing query at Line 1
File 'https://SAFINCAL.dfs.core.windows.net/Container-2/Post-Sale/shipments/2021/*.parquet' cannot be opened because it does not exist or it is used by another process.
Visit this article to learn more about this error
Total execution time: 00:00:12.269
The article points to https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/resources-self-help-sql-on-demand#query-fails-because-file-cannot-be-opened
The solution proposed in the article is get myself "Storage Blob Data Contributor". The question that I have is - if I get myself "Storage Blob Data Contributor" - then I may end up getting access to entire container and that defeats the purpose of ACL access granted at folder level ( green highlighted boxes)
Is that the right understanding ? If yes - how do I ensure I can still query the data in "shipments" folder from Azure Synapse Analytics Serverless pool without getting myself added as Contributor/Reader for entire container/storage account
Thanks
If you are using ACLs you need to have at least "X" permissions on containers as well.
I'm not familiar about Azure Analytics Serverless Pool, but I think it maybe the same as Azure SQL. This problem usually happens because you don't have rights to access the file.
So I used to Create an account SAS, then use the SAS key to access the files.
Related
I would like to programmatically retrieve and process all logs available from the Office 365 Unified Audit Logs for the purpose of forensic investigation. From the front end, these logs are available through the Office 365 Compliance Admin Center.
I have tried the following options to access these logs from a script, with no success:
Microsoft 365 Management API - This contains the correct data, but is of limited usefulness for forensic investigations due to the short 7 day retention period.
Microsoft Graph - This does not contain all the relevant data - you cannot access the Unified Audit Logs directly through Graph, and the usage reports do not cover all items contained in the Audit Logs (e.g. Exchange actions).
Search-UnifiedAuditLog on Exchange Online PowerShell - Microsoft themselves recommend not to use this programmatically, and I've experienced extremely unreliable results and unmanageable rate-limiting when trying to do so.
So is there something I'm missing here, or is there no way to programmatically retrieve all items from the Unified Audit Logs for the entire retention period? (generally 90 days).
As far as I know the only way to do this is to use the Management API on a regular basis and output the results to some solution for long term storage (Azure Log Analytics Workspace comes to mind, or SIEM like Splunk / Graylog). I.e. write a script that retrieves logs for the last week, and run it at least weekly.
I'll explain how to retrieve logs manually and also show a tool which already exists for this at the bottom of the post.
Manually:
1: Enable Audit logging on the tenant if not already enabled
2: Create an App registration in Azure AD and for getting single tenant audit logs choose "Accounts in this organizational directory only (xyz only - Single tenant)"
3: Create a 'secret key' from within the newly created App Registration. Store it somewhere safe as it's only shown once. From the overview page of the App Registration also store the "Tenant ID" and "Application (Client) ID". You will need all three.
4: From within the new App Registration go to "API permissions" and add 'Application type' permissions for: 'ActivityFeed.Read' and 'ActivityFeed.ReadDlp'.
5: For the following steps you will need to start calling the Office API's, for which you will need a bearer token in the header. To obtain this send the following POST request:
URL: https://login.microsoftonline.com/***tenant_ID***/oauth2/token
Headers: "{'Content-Type': 'application/x-www-form-urlencoded'}"
Data: "grant_type=client_credentials&client_id=Application_ID&client_secret=Secret_Key&resource=https://manage.office.com"
You will receive a JSON response which contains 'access_token'. For all the upcoming API calls, use the following header:
"{'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'bearer access_token'}"
6: Subscribe to the audit log feeds you would like to retrieve. The following exist: 'Audit.General', 'Audit.AzureActiveDirectory', 'Audit.Exchange', 'Audit.SharePoint', 'DLP.All'. The POST for Exchange for example would look like: "https://manage.office.com/api/v1.0/tenant_ID/activity/feed/subscriptions/start?contentType=Audit.Exchange"
7: You are now ready to start retrieving actual logs. Individual logs live inside content blobs, which live inside pages, which live inside feeds (e.g. the Audit.Exchange feed). Therefore, for each feed you would like to retrieve logs from, you must collect all the content blobs (iterating through the pages of them) and then retrieve the actual content from that blob.
To retrieve a page of content blobs use the following URL (change bolded content to your situation): "https://manage.office.com/api/v1.0/tenant_ID/activity/feed/subscriptions/content?contentType=Audit.Exhange&startTime=2022-04-13T09:42:52&endTime=2022-04-14T08:42:52"
This will give you a JSON response with content blobs inside. In the response header check "NextPageUri"; if it contains a URL, call that URL for the next page of content.
Now that you have content blobs, use them to retrieve the actual logs. Each content blob is a JSON dict, which contains a "contentUri" field. Call that URL to retrieve a JSON response with the actual logs inside.
You can do this in most programming/scripting languages, but for larger amounts of logs you will want to retrieve logs in parallel, or it will take a long time.
With a tool
In case you want to use an existing tool, this one is free, works on Linux and Windows, and supports multiple outputs.
Microsoft introduced a special folder in OneDrive (I only see it in a personal OneDrive account, not in OneDrive for Business) called "Personal Vault". I searched the documentation of MS Graph API but could not find this mentioned.
So my question is: is there any way to access this personal vault as a third-party app?
Same problem here.
Funny thing is that:
It does not appear in the root children
It appears if you call the delta API where you can see a folder with the
specialFolder attribute not null and whose specialFolder.name is
"vault". This specialFolder, though, has the deleted facet, with the
state set to "hardDeleted". If you extract the ID from the delta API
call and try to address the folder directly with
https://graph.microsoft.com/v1.0/me/drive/items/{id}, however, you
get an access denied error
I am in the process of switching the LDAP backend that we use to authenticate access to Gerrit.
When a user logs in via LDAP, a local account is created within Gerrit. We are running version 2.15 of Gerrit, and therefore our local user accounts have migrated from the SQL DB into NoteDB.
The changes in our infrastructure, mean that once the LDAP backend has been switched, user logins will appear to Gerrit as new users and therefore a new local account will be generated. As a result we will need perform a number of administrative tasks to the existing local accounts before and after migration.
The REST API exposes some of the functionality that we need, however two key elements appear to be missing:
There appears to be no way to retrieve a list of all local accounts through the API (such that I could then iterate through to perform the administrative tasks I need to complete). The /accounts/ endpoint insists on a query filter being specified, which does not appear to include a way to simply specify 'all' or '*'. Instead I am having to try and think of a search filter that will reliably return all accounts - I haven't succeeded yet.
There appears to be no way to delete an account. Once the migration is complete, I need to remove the old accounts, but nothing is documented for the API or any other method to remove old accounts.
Has anybody found a solution to either of these tasks that they could share?
I came to the conclusion that the answers to my questions were:
('/a/' in the below examples is accessing the administrative endpoint and so basic Auth is required and the user having appropriate permissions)
Retrieving all accounts
There is no way to do this in a single query, however combining the results of:
GET /a/accounts?q=is:active&n=<number larger than the number of users>
GET /a/accounts?q=is:inactive&n=<number larger than the number of users>
will give effectively the same thing.
Deleting an account
Seems that this simply is not supported. The only option appears to be to set an account inactive:
DELETE /a/accounts/<account_id>/active
I am currently working on a solution that is accessing OneDrive in Office 365 using Microsoft Graph. I am using the adal4j library to handle authentication and have configured the app in portal.azure.com.
My question relates the call to get the children for a specified drive. I am using a query similar to the one shown below, as I want to get folders and files at the root level of a specified users drive:
https://graph.microsoft.com/v1.0/users/*user id*/drives/*drive id*/root/children
When I login to the Graph Explorer and execute the query, I get a json result showing the root folder contents for the drive and user specified. All works as expected.
When I call it from my java application, the JSON node value is empty ([]).
Initially my thought was, because the Graph Explorer uses a different app id in the portal it was possibly something to do with access rights. However, I successfully read user profiles in our O365 tenant, the drive id's for each user, and if I execute the following:
https://graph.microsoft.com/v1.0/users/*user id*/drives/*drive id*/root/search(q='')
It provides me a complete list of all of the folders, sub folders etc within the appropriate user's drive.
Therefore, making me think this is a bug with the Graph query I am attempting to use rather than an authorization issue, but, that wouldn't explain why it works in the Graph Explorer.
The same java method is used for all calls, and the url is passed in as a parameter.
Just to follow up, the azure portal app permissions has the capability of adding permissions for the graph api. This was, indeed the problem. It would appear that the search was ignoring the permission and successfully reading the data whereas the /children call was honouring the security model. This caused a lot of confusion, but is now resolved.
Thanks Marc for your help.
I have a screen where a user selects database source from a drop down. Once that's selected I would like the information passed onto other screens so the user does not keep having to select.
How can I pass information such as this from one screen to another? Note that the information is just very small things like:
DatasourceID - 2 characters
SubjectID - 2 characters
As I am running on Azure can I assume the best place to store this would be on the client side? I saw one implementation that stored data like this:
Session["abc"] = "def";
if (Session["abc"] != null)
etc ...
Is this the best way or am I missing something. Also how would the above work when the page could be served by different servers each time around? Does the above store information locally?
The Session is stored on the Server Side. Now in Azure you have a few options where exactly it is stored. It depends on what you would like to do with this datasource. If this is something you just need in the following screen, you can store it in TempData which is stored in the session. It is kept there until you read it.
Now you have these options to store the session state:
in Azure AppFabric Cache
in a SQL Azure DB
in blob storage
Azure AppFabric Caching has got a Session provider which is very easy to set up. You can just create a new cache in the Azure portal and get the required web.config entries by clicking the according button on the toolbar. this is also explained in detail here.
Using that you can store things in the Session out of process. The downside is that it's a bit expensive (about 45$/month for a 128 MB cache). So the alternative would be to store session state in SQL Azure. There's a Session provider for SQL Azure.
Here's a link to a great introduction by Scott Hanselman to the ASP.NET universal providers. If you're not using membership, then you just need to setup System.Web.Providers.DefaultSessionStateProvider.
Just make sure you point the connection string to your SQL Azure DB. Note: You must set MultipleActiveResultSets=True in the connection string, so be sure to add it back if you’ve copied the SQL Azure connection string from the portal.
Then there is also a session provider for blog storage in the training Kit, available with a sample app at http://code.msdn.microsoft.com/windowsazure/Windows-Azure-ASPNET-03d5dc14.
I believe it is unsupported by MS.
Hope this helps.