This is regarding the access issue that I am facing in reading the data residing in ADLS Gen-2 from Synapse Analytics Serverless Pool.
I have full access on Synapse Studio
I have "Storage Blob Data Contributor" on the container/folder where my parquet file is residing
While I try to below SQL ( with right ADLS name along with folder/container details)
SELECT
TOP 100 *
FROM
OPENROWSET(
BULK 'https://.dfs.core.windows.net/<container_name>////2009/DIM_INFO_2009.parquet',
FORMAT='PARQUET'
) AS [result]
I get below message -
File 'https://.dfs.core.windows.net/<container_name>////2009/DIM_INFO_2009.parquet' cannot be opened because it does not exist or it is used by another process.
When I try the same scenario by creating the external table ( which points to the same parquet file) and uses MANAGED IDENTITY as credentials - the results are being displayed as expected.
So - I am wondering, what is causing this issue - when I try to run the SQL and make the results get displayed. I was hoping with having Storage Blob Data Contributor - it should allow me to access the data on ADLS from Synapse Studio Serverless Pool.
Any hints on what might be causing this issue ?
For AAD passthrough auth make sure you also follow these steps to allow Synapse Serverless to get past the ADLS firewall with an identity other than the MSI.
If the PowerShell approach above doesn’t appeal to you then it is also now available in the Azure Portal UI.
Related
I'm trying to configure Azure Data Lake Store Gen1 (ADLS) as a Data Source in a Data Virtualization software (Tibco's).
Reading through the Azure documentation, I got to the webhdfs/v1/ endpoint associated with ADLS and managed to simulate the (non-standard, since it requieres an additional "resource" parameter) OAuth2 authentication needed within Postman. All good so far.
I am guessing that the connection should be configured as an OData service, since the OData connector within TDV (Tibco DV) has an "Azure AD resource" parameter that should be used only if you are trying to authenticate through azure's active directory (the non-standard part from before). However I am failing to configure this connection.
This must be because of one of these reasons:
The /webhdfs/v1/ endpoint is NOT an OData service.
I am wrongly configuring the OData connector in TDV.
In any case, how can I connect to this repository and introspect everything behind it? list folders, files, and all that within TDV?
Some suggestions I have stumbled upon are using the Hive connectors or the REST connector, but after trying I still feel closer to success with OData.
I have implemented Umbraco using UmbracoIdentity for membership and everything was going fine until I deployed my solution to an Azure Web App. On azure I am getting permission errors because UmbracoIdentity is using a SQL Server CE database stored in the App_Data folder.
For reference the error I am getting is:
Exception type: SqlCeException
There is a file sharing violation. A different process might be using the file. [ ...\wwwroot\App_Data\UmbracoIdentity.sdf ]
My Umbraco data is being stored in an SQL database and I would like to store my UmbracoIdentity membership data here as well. I would appreciate any help in how to setup SQL Server as the user store for membership data.
You need to implement the IExternalLoginStore.cs interface and then configure the application to use it. It should be fairly simple to implement as you can use the SQL Server CE implementation as an example. I've done one for Azure Table Storage - you can check the Readme at https://github.com/alindgren/UmbracoIdentity.AzureLoginStore to get an idea of how to configure the app to use a custom external login store (which for me was the least obvious part).
I am creating an iOS project that needs to read a SQL database in Azure. I have the database Server location, port, username, and password. I don't need to write to the database, just read it. I am more familiar with Firebase or Parse and have never used Azure. How do I even go about starting this? I tried the sample project that Azure makes for you but I don't have any tables? Do I need this? Any help would be welcome.
Azure Mobile App Service can connect with your existing SQL database
this thread explains the process where you use the existing SQL database. With this option Azure manages most of the inner workings for you.
If you want to build the Rest API from "scratch" using your existing SQL database You have some more options:
Azure API Management allows you to publish API's securely and at scale, A server less Azure Function like the example in this article Rest API with Azure Functions and Azure SQL Database or build a rest API using an Azure Logic App which doesn't require you to write code. You could also use Nodejs or many other tools you just need to evaluate what would work best for your use case.
There are a lot of ways to have a database on server ( I can use my own server or I can use MS Azure (for Azure I found REST API Description https://msdn.microsoft.com/en-us/library/azure/gg715283.aspx , but I didn't find a way to add some rows to a table or to get information from the table)), but I want to edit the database and get proper rows from the server database from iOS code without having server API. There are a lot of ready solutions such as Backendless.com , but they are not stable.
So is there any way to connect to the remote SQL server database on the remove server or to MS Azure database and to edit rows and fetch data?
May there are some framework for iOS to do than? (for example in .Net there are Linq framework and Entity Framework).
In your question, you linked to a management API for SQL Database service - this has nothing to do with data manipulation; it's strictly a management API for dealing with servers and databases.
If you want to direct-connect from an IOS app to SQL Database Service (or SQL Server in a VM), you'd need to find a client library for IOS (and recommendations of such a library are not in scope for StackOverflow). There is no native SQL Server client library specifically provided through the Azure SDKs / APIs.
Outside of a native client library, there is Azure Mobile Services, which provides an API stack specifically designed around use by mobile apps (whether IOS, Android, Windows Phone, or even JavaScript). The API stack provided here, by default, gives CRUD operations for SQL Database tables. Additionally, it supports adding custom API calls, where you have complete control of your API calls.
Any other API stack would require you to choose the stack and run it yourself (whether in a Linux or Windows VM, Cloud Service, or Web App - there's no single right way to do this).
Documentation for Azure Mobile Apps (and related API feature, such as monitoring) is here.
Since you are developing in Azure consider automatic REST API from SlashDB, which is available from their Marketplace.
https://azure.microsoft.com/en-us/marketplace/partners/vte/slashdb-slashdb-azure/
SlashDB reflects the entire database as URL-linked structure so a lot of queries need not even be written. It supports SELECT, INSERT and UPDATE. In addition to that it allows for defining an API end-point to call a stored procedure or return results of a SQL query.
Disclosure: I am the founder and CEO of the company behind SlashDB, but you don't have to take my word it - just try it.
How can we get a list of start addresses added in a Sharepoint content source programatically ? The code which need to achieve this will be running on different server than where sharepoint is installed so i am looking for a solution where we can query sharepoint database to get this list.
Never query the sharepoint database directly. Even a simple select statement an cause a read lock, potentially creating havoc in your farm.
By using the database directly you are NO LONGER SUPPORTED, meaning that if it does go belly up and you call microsoft for support, they'll send someone from consulting ($$$$) or ask you to go back to the last known good configuration (i.e. a backup of your sharepoint databases from before the point you started accessing the database directly...)
Create a web service that uses the sharepoint object model to get this list, then deploy it to the sharepoint machine, then have your external app use that service.