How can I implement data masking in Synapse serverless SQL pool, as currently, it is only implemented in a Synapse dedicated SQL pool?
I am expecting to achieve masking in a serverless SQL pool.
As per a Microsoft document, it is clearly stated that Dynamic data masking is only available for Dedicated SQL Pool, not for Serverless SQL Pool. As serverless SQL pool does not support Tables, Materialized views, DDL statements, DML statements, it might the reason.
Also, as Nandan suggested, it's not supported on external tables either.
You can raise a feature request here.
Just because something is not implemented, does not mean you can not implement it yourself.
First, I thought it would be great to create a function. But the dedicated and server less pools only support in line table value functions.
Second, we can also create a view with masked data. Then revoke the user from having rights to see that base table. Lets implement that for the customer id key. The code below shows the view.
--
-- Create view with masked customer number
--
CREATE VIEW saleslt.vw_dim_masked_customer
AS
SELECT
'***' +
SUBSTRING(CAST([CustomerKey] AS VARCHAR(5)), len([CustomerKey]) - 2, 2) AS MASKED,
[CustomerKey],
[FirstName],
[MiddleName],
[LastName]
FROM [saleslt].[dim_customer]
GO
-- Test view
SELECT * FROM saleslt.vw_dim_masked_customer
GO
I have a database called mssqltips that contains the adventure works data as parquet data files exposed by external tables.
The output from the view shows that our data is masked. I did not get rid of the original column, Customer Key, since I wanted to do a comparison. Also, I would add some error handling for strings that are less than 2 characters long or null.
In short, dynamic data masking as a feature might not be supported. But you can easily mask data using custom logic and views. Just remember to revoke the user access to the base table.
Dynamic data masking is supported in actual physical tables and not supported on external tables.
Hence DDM is not supported in serverless pools
Related
It is generally known that ABAP memory (EXPORT/IMPORT) is used for passing data inside ABAP session across call stack, and SAP memory (SET/GET) is session independent and valid for all the ABAP sessions of user session.
The pitfall here is that SET PARAMETER supports only primitive flat types, otherwise it throws the error:
"LS_MARA" must be a character-type field (data type C, N, D or T). by
Global assignment like ASSIGN '(PrgmName)Globalvariable' TO FIELD-SYMBOLS(<lo_data>). is not always a way, for example if one wants to pass structure to some local method variable.
Creating SHMA shared memory objects seems like an overkill for simple testing tasks.
So far I found only this ancient thread were the issue was raised, but the solution from there is stupid and represents a perfect example of how you shouldn't write, a perfect anti-pattern.
What options (except DB) do we have if we want to pass structure or table to another ABAP session?
As usual Sandra has a good answer.
Export/Import To/From Shared buffer/Memory is very powerful.
But use it wisely and make sure you understand that is is on 1 App server and
is non persistent.
You can use rfc to call the other app servers to get the buffer from other servers if necessary. CALL FUNCTION xyz DESTINATION ''
See function TH_SERVER_LIST . ie what you see in SM59 Internal Connection.
Clearly the lack of persistency of shared buffer/memory is of key consideration.
But what is not immediately obvious until you read the docu carefully is how the shared buffer manager will abandon entries based on buffer size and avaliable memory. You can not assume shared buffer entry will be there when you go to access it. It most likely will be, but it can be "dropped", the server might be restarted etc. Use it as a performance helping tool but always assume the entry might not be there.
Shared memory as opposed to shared buffer, suffers from the upper limit issue, requiring other entries to be discarded before more can be added. Both have pros and cons.
In St02 , look for red entries here, buffer limits reached.
See the current parameters button that tells you which profile parameters need to be changed.
A great use of this language element is for logging or for high performance buffering of data that could be reconstructed . It is also ideal for scenarios in badis etc were you can not issue commits. You can "hold" info without issuing a commit or db commit.
You can also update / store your log without even using locking.
Using the simple principle the current workprocess no. is unique.
CALL FUNCTION 'TH_GET_OWN_WP_NO'
IMPORTING
wp_index = wp_index.
Use the index no as part of the key to your data .
if your kernel is 7.40 or later see class CL_OBJECT_BUFFER
otherwise see function SBUF_OBJ_SHOW_OBJECT
Have fun with Shared Buffers/Memory.
One major advantage of share buffers over share memory objects is the ABAP Garbage Collector. SAPSYS Garbage collection can bite you!
In the same application server, you may use EXPORT/IMPORT ... SHARED BUFFER/MEMORY ....
Probable statement for your requirement:
EXPORT mara = ls_mara TO SHARED BUFFER indx(zz) ID 'MARA'.
Between application servers, you may use ABAP Channels.
Most of our tables have one or more columns which are set by the database, either by a trigger, or we want to use the database default value (which requires not sending the field at all in the insert or update)
This includes transaction dates set in the dB (so all the times are times stamped very accurately by a single source, not relying on the accuracy of the time on an arbitrary server or pc.)
The second VERY common use case is say if a customer record has his address and a last logged in field. the last logged in field (and number of failed logins) is set by another part of the system (e.g. by a web site server). The current overly simplistic CRUD system which GORM provides would overwrite such a field when an operator or customer edits their address for example. This is because GORM includes in its update and insert statements every field, even if it's null, or if it has not been changed.
We need a way to scrub the field from inserts and updates, but still have it used in the read calls.
I.e. a true "read only" attribute.
We tried this:
failedLogins editable: false, attributes: [readonly:true]
Which has no effect on the SQL generated (and doesn't even affect the scaffolded UIs - its still editable in create and edit, in grails 2.4.4 at least, but thats another story)
When we do want to explicitly write one of these fields, such as number of failed logins, we would resort to using embedded SQL.
I saw this post: Read-Only columns
Which asks exactly the same question, but only gives one solution, which is this plugin:
extended GORM mappings
Unfortunately, this plugin has not been updated since 2010, and only works with 1.3. We need something which works with 2.4.4.
Any grails app which has multiple systems which edits independent fields needs something like this, or to do extensive locking (Which is usually out of the question).
E.g. an operator opens the customer details for editing, edits something editable (e.g. address), then the operator fails a login on the website (a different grails or non-grails app), then the operator saves the player details. If the saving included the numberOfFailedLogins field, the system would fail. If opening the player details for editing locked the player, then the player would not be able to login, as updating the "lastLoggedIn" or "numFailedLogins" would fail to be able to write due to the lock. The solution is VERY simple - read only columns. Another way would be to put each read only type field in their own tables, but this would be untenable (and result in hundreds of one field tables)
Or we go back to using MyBatis, which has no such issues, and full control. Sadly, there is no good mybatis plugin for grails.
You can use derived properties for string and number properties:
class Batch {
String name
Integer timesRun
static mapping = {
timesRun formula: 'times_run' //times_run is a column in the "batch" table
}
}
In the code above, timesRun would be read in from the database but ignored in inserts and updates as Hibernate considers the column a calculated one.
Updated the example because the original one may have been misleading
This probably doesn't specifically answer your question, but you can use dynamicUpdates to tell GORM to only update the properties of the domain object that have changed during the current session. So as long as you don't change the "read-only" property in your code it won't be set in the SQL update statement generated by Grails. For added safety you could override (and noop) the setter so that your code can never change that property.
https://grails.github.io/grails-doc/latest/ref/Database%20Mapping/dynamicUpdate.html
One of the downsides of dynamicUpdates is that it might make the Hibernate query cache less useful. However, it seems that some Grails/Hibernate experts recommend that you disable the query cache anyway (at least in older versions of Grails). Not sure if that's true of Grails 2.4+
http://grails.github.io/grails-howtos/en/performanceTuning.html
http://www.anyware.co.uk/2005/2012/11/12/the-false-optimism-of-gorm-and-hibernate/
http://tech.puredanger.com/2009/07/10/hibernate-query-cache/
Need to periodically read ~20K records from some external database (schema not under my control), and update/create respective instances in a the local schema (grails' main dataSource). The target is a single domain class.
I've mapped the external database as another dataSource. I'm thinking to use groovy.sql.Sql + raw SQL to bring-in all records, and generate domain instances as-required. Is that a reasonable path? Should I alternatively model-out the external schema, and use GORM end-to-end?
Assuming the first approach, considering testing: are there any useful tools I should look into for setting-up test data (I.E. an equivalent of build-test-data/fixtures for non-domain data)?
Thanks
Yes. Think this is reasonable given the data size and how often you are going to do this. Just dont forget to execute the sql by batch to save on resources.
Can any one tell me the difference between usage of Deep Insert and $batch - ChangeSet in the context of OData ? I have a scenario that requires creation of a Sales Order Header and Sales Order Items together.
I can either user Deep Insert (BTW is this standard OData spec ?) or
I can use a $batch (this is standard OData spec) call with these two entities specified as a part of the same ChangeSet, which would ensure that they get saved together as a part of a single LUW.
What are the pros / cons of using either of these approaches ? Any experiences ?
Cheers
Deep Insert is part of the OData specification, see http://docs.oasis-open.org/odata/odata/v4.0/os/part1-protocol/odata-v4.0-os-part1-protocol.html#_Toc372793718.
Deep Insert allows creating a tree of related entities in one request. It is insert only.
$batch allows grouping arbitrary requests into one request, and arbitrary modifying operations into LUWs (called change sets).
For insert-only cases Deep Insert is easier: you just POST the same format that you would GET with $expand.
Deep insert or deep update is not currently defined and supported by OData spec. However there are such feature requests, like this: https://data.uservoice.com/forums/72027-wcf-data-services-feature-suggestions/suggestions/4416931-odata-deep-update-request-support
If you decided to use a batch, then you have to do the next set of commands in your batch:
PUT SalesOrderItem
...
PUT SalesOrderItem
PUT SalesOrderHeader
PUT SalesOrderHeader/links$/SalesOrderItem
...
PUT SalesOrderHeader/links$/SalesOrderItem
See also here: How do I update an OData entity and modify its navigation properties in one request?
In our ASP.NET project we decided to go with CQRS pattern and use OData for Query requests and Web API for Commands. Talking in terms of your case we created Web API Controller with action CreateSalesOrder with parameter of class SalesOrderHeaderDto that contains array of SalesOrderItemDtos. Having the data on server you can easily develop insert the whole Order Sale in one transaction with its Order Items. Also there is just two command to be sent on server - ~/api/CreateSalesORder and ~/odata/SalesOrder with include=Items and filter by something... for example first command can return an Id of the Order...
Deep insert gives one operation that will insert all the items as one operation.
The same thing isn't possible in a $batch.
This is not automatic in a batch :
they get saved together as a part of a single LUW
The $batch needs to be in a single change set to expect atomicity.
According to OData 4.0 11.7.4 Responding to a Batch Request:
All operations in a change set represent a single change unit so a service MUST successfully process and apply all the requests in the change set or else apply none of them. It is up to the service implementation to define rollback semantics to undo any requests within a change set that may have been applied before another request in that same change set failed and thereby apply this all-or-nothing requirement. The service MAY execute the requests within a change set in any order and MAY return the responses to the individual requests in any order. The service MUST include the Content-ID header in each response with the same value that the client specified in the corresponding request, so clients can correlate requests and responses.
However, a single changeset is unordered. Given you are doing a deep insert, there is some realtionship between the entities, and given you are doing an insert, in either a contained navigation or a $ref navigation you can't perform both inserts or both inserts and the PUT / POST $ref in an unordered fashion.
A change set is an atomic unit of work consisting of an unordered group of one or more Data Modification requests or Action invocation requests.
I have a program in Delphi 2010 that uses a JET (mdb) database via ADO. I would like to be able to extract the definitions of some of the queries in the database and display them to the user. Is this possible either via SQL, some ADO interface, or by interrogating the database itself (I don't seem to have rights to MSysObjects).
Some of that information is available via ADOX calls. There is an overview of the api with some examples (unfortunately not in Delphi) on the MSDN website.
Basically what you will want to do is to is to import the ADOX type library, and then use the wrapper that is generated for you to access the underlying API. From there its as simple as navigating the hierarchy to get at the data you need.
You will need to access the specific View object, and from there get the command property.
Via DAO, it's pretty easy. You just extract the SQL property of each QueryDef. In DAO from within Access, that would be:
Dim db As DAO.Database
Dim qdf As DAO.QueryDef
Set db = DBEngine.OpenDatabase("[path/name of database]")
For Each qdf In db
Debug.Print qdf.SQL
Next qdf
Set qdf = Nothing
db.Close
Set db = Nothing
I don't know how to translate that, but I think it's the simplest method once you're comfortable with using DAO instead of ADOX.
I don't use ADO at all, but I'm guessing that it has a collection of views and the SQL property would work for SELECT queries. However, if you're interested in getting the SQL for all saved QueryDefs, you'd also need to look at the DML queries, so you'd have to look at the stored procedures. I would have to look up the syntax for that, but I'm pretty certain that's how you'd get to the information via ADO.