According to documents there are four transaction isolation levels in Firebird. However, as far as I know, there's no explicit isolation level selection in uib library (TUIBTransaction), but bunch of options for transactions. How I should use those? Is there documentation somewhere?
These bunch of options are what will change the isolation level. As #Arioch said in his compact comment, you can change the isolation level changing the property Options that is of type TTransParams. This is a set of TTransParam as below.
// Transaction parameters
TTransParam = (
{ prevents a transaction from accessing tables if they are written to by
other transactions.}
tpConsistency,
{ allows concurrent transactions to read and write shared data. }
tpConcurrency,
{ Concurrent, shared access of a specified table among all transactions. }
{$IFNDEF FB_21UP}
tpShared,
{ Concurrent, restricted access of a specified table. }
tpProtected,
tpExclusive,
{$ENDIF}
{ Specifies that the transaction is to wait until the conflicting resource
is released before retrying an operation [Default]. }
tpWait,
{ Specifies that the transaction is not to wait for the resource to be
released, but instead, should return an update conflict error immediately. }
tpNowait,
{ Read-only access mode that allows a transaction only to select data from tables. }
tpRead,
{ Read-write access mode of that allows a transaction to select, insert,
update, and delete table data [Default]. }
tpWrite,
{ Read-only access of a specified table. Use in conjunction with tpShared,
tpProtected, and tpExclusive to establish the lock option. }
tpLockRead,
{ Read-write access of a specified table. Use in conjunction with tpShared,
tpProtected, and tpExclusive to establish the lock option [Default]. }
tpLockWrite,
tpVerbTime,
tpCommitTime,
tpIgnoreLimbo,
{ Unlike a concurrency transaction, a read committed transaction sees changes
made and committed by transactions that were active after this transaction started. }
tpReadCommitted,
tpAutoCommit,
{ Enables an tpReadCommitted transaction to read only the latest committed
version of a record. }
tpRecVersion,
tpNoRecVersion,
tpRestartRequests,
tpNoAutoUndo
{$IFDEF FB20_UP}
,tpLockTimeout
{$ENDIF}
);
Since Interbase 6.0 code "opensourced", the documentation for the API hasn't changed much. So if you want an explanation about any of them the docs you are looking are in Interbase manuals.
You can get them here https://www.firebirdsql.org/en/reference-manuals/
Below I'm quoting Ann Harrison in this link to an quick explanation on the usual options used:
isc_tpb_consistency can cause performance problems due the fact that it's locking tables and possibly excluding concurrent access.
isc_tpb_concurrency is the design center for Firebird. Readers don't
block writers, writers don't block readers, and both get a consistent
view of the database.
isc_tpb_read_committed + isc_tpb_rec_version + isc_tbp_read_only give
inconsistent results and occasionally produces an error on a blob
read*, but unlike other modes, it does not block garbage collection so
it's a good mode for long running read transactions that don't have to
get the "right" answer.
isc_tpb_read_committeed + isc_tpb_rec_version has the same performance
as isc_tpb_concurrency, but gets inconsistent results - the same query
run twice in the same transaction may return different rows.
isc_tpb_read_committed + isc_tpb_no_rec_version + isc_tpb_wait is
slower than other modes because it will wait for a change to be
commited rather than reading the newest committed version. Like all
variants of isc_tpb_read_committed, it does not produce consistent
results.
isc_tpb_read_committed + isc_tpb_no_rec_version + isc_tpb_no_wait
gives lots and lots of deadlock errors because every time a reader
encounters a record that's being changed, it returns an error.
NOTE: I hope that you can see that, beside the parameters are not named equally, it's not that hard to understand if you remove the "isc_tpb_" part.
Related
Apache Beam has recently introduced state cells, through StateSpec and the #StateId annotation, with partial support in Apache Flink and Google Cloud Dataflow.
I cannot find any documentation on what happens when this is used with a GlobalWindow. In particular, is there a way to have a "state garbage collection" mechanism to get rid of states for keys that have not been seen for a while according to some configuration, while still maintaining a single all-time state for keys are that seen frequently enough?
Or, is the amount of state used in this case going to diverge, with no way to ever reclaim state corresponding to keys that have not been seen in a while?
I am also interested in whether a potential solution would be supported in either Apache Flink or Google Cloud Dataflow.
Flink and direct runners seem to have some code for "state GC" but I am not really sure what it does and whether it is relevant when using a global window.
State can be automatically garbage collected by a Beam runner at some point after a window expires - when the input watermark exceeds the end of the window by the allowed lateness, so all further input is droppable. The exact details depend on the runner.
As you correctly determined, the Global window may never expire. Then this automatic collection of state will not be invoked. For bounded data, including drain scenarios, it actually will expire, but for a perpetual unbounded data source it will not.
If you are doing stateful processing on such data in the Global window you can use user-defined timers (used through #TimerId, #OnTimer, and TimerSpec - I haven't blogged about these yet) to clear state after some timeout of your choosing. If the state represents an aggregation of some sort, then you'll want a timer anyhow to make sure your data is not stranded in state.
Here is a quick example of their use:
new DoFn<Foo, Baz>() {
private static final String MY_TIMER = "my-timer";
private static final String MY_STATE = "my-state";
#StateId(MY_STATE)
private final StateSpec<ValueState<Bizzle>> =
StateSpec.value(Bizzle.coder());
#TimerId(MY_TIMER)
private final TimerSpec myTimer =
TimerSpecs.timer(TimeDomain.EVENT_TIME);
#ProcessElement
public void process(
ProcessContext c,
#StateId(MY_STATE) ValueState<Bizzle> bizzleState,
#TimerId(MY_TIMER) Timer myTimer) {
bizzleState.write(...);
myTimer.setForNowPlus(...);
}
#OnTimer(MY_TIMER)
public void onMyTimer(
OnTimerContext context,
#StateId(MY_STATE) ValueState<Bizzle> bizzleState) {
context.output(... bizzleState.read() ...);
bizzleState.clear();
}
}
There is not automatic garbage collection of state if you use GlobalWindows. Only if you use some non-global window will state be garbage collected after the watermark passes the end of a window plus the allowed lateness.
What you can do if you must work with GlobalWindows is to manually keep as state the last update timestamp. Then you would periodically set a timer where you check this timestamp against the current time and delete state if necessary. You would set this timer when encountering a key for the first time (which you can see from the absence of your timestamp state) and then re-set it in the #OnTimer method.
Does this mean we can not call some thing like this via Java API?
I get error - "Caused by: org.neo4j.graphdb.QueryExecutionException: Cannot perform schema updates in a transaction that has performed data updates."
This happens when I call schema update from a procedure call via neo4j console.
try (Transaction tx = db.beginTx()) {
String query = "CREATE INDEX ON :" + lbl + "(" + name + ")";
db.execute(query);
tx.success();
}
The Cypher query calling the procedure is already executed in a transaction, and there are no nested transactions in Neo4j: when you call db.beginTx(), you're getting the existing transaction, and it's not actually necessary unless you need the Transaction object (e.g. to create locks).
Anyway, even though it's not explicitly documented, it's apparently not possible to manipulate the schema from Neo4j procedures. You could say that it fails the use case of
To provide access to functionality that is not available in Cypher, such as manual indexes and schema introspection.
I created a test procedure similar to yours:
public class IndexProcedure {
#Context
public GraphDatabaseService db;
#Procedure
#PerformsWrites
public void index(#Name("label") String label, #Name("property") String property) {
db.schema().indexFor(Label.label(label)).on(property).create();
}
}
and ran it from the shell in the simplest Cypher query:
CALL my.package.index('Node', 'name');
Without the #PerformsWrite annotation, I get the following (expected) exception:
WARNING: Failed to invoke procedure my.package.index: Caused by: org.neo4j.graphdb.security.AuthorizationViolationException: Schema operations are not allowed for READ transactions.
With the annotation, I get the same exception as you:
WARNING: Failed to invoke procedure my.package.index: Caused by: org.neo4j.graphdb.QueryExecutionException: Cannot perform schema updates in a transaction that has performed data updates.
I guess the rationale is that setting up the schema is mostly a one-time operation that doesn't really need a procedure: if you're going to execute some Cypher query to call the procedure, you might as well run the script which creates the constraints and indices.
There could also be technical constraints: index creation is asynchronous and probably doesn't participate in the transaction (can you rollback the creation of an index?).
Or maybe it's just a bug? We should get someone from Neo to confirm.
Update: it will supposedly be fixed in Neo4j 3.1 when it's released, per a discussion on SlackHQ.
Is it possible to have the DataFlow process maintain the state. There are log processing tools that allow for that by providing fast access (propriety / in-memory) files available for the real time process to keep track of the state on the logs while processing them.
A use case example would be with tracking registration steps taken by users. The registration steps would come in different logs and the data form those logs would be assembled by the real time process into one final database record (for each registered user) that is written to a database.
Can my DataFLow code keep track of the many registration steps (streaming input) by users and once user's registration steps are completed then have the DataFLow process write the records to the database (one record per user).
I don't know much about DataFlow architecture. It must be using some (proprietary / in-memory nosql) data storage for keeping track of things it needs to keep track of (ex. when it tries to produce top 100 customers). Is that fast access data storage also available to the DataFlow processes to use?
Thanks
As danielm said, state is not yet exposed. The good news is you may not need it for your use case.
If you have a PCollection<KV<UserId, LogEvent>> you can use a CombineFn and Combine.perKey to take all of the LogEvents for a specific UserId and combine them into a single output. The CombineFn tells Dataflow how to create an accumulator, update it by incorporating input elements, and then extract a final output. Transforms like Top actually use a CombineFn (with a Heap as the accumulator) rather than an actual state API.
If your events are of different types, you can still do something like this. For instance, if you have two logs, you can do:
PCollection<KV<UserId, LogEvent1>> events1 = ...;
PCollection<KV<UserId, LogEvent2>> events2 = ...;
// Create tuple tags for the value types in each collection.
final TupleTag<LogEvent1> tag1 = new TupleTag<LogEvent1>();
final TupleTag<LogEvent2> tag2 = new TupleTag<LogEvent2>();
//Merge collection values into a CoGbkResult collection
PCollection<KV<UserIf, CoGbkResult>> coGbkResultCollection =
KeyedPCollectionTuple.of(tag1, pt1)
.and(tag2, pt2)
.apply(CoGroupByKey.<UserId>create());
// Access results and do something.
PCollection<T> finalResultCollection =
coGbkResultCollection.apply(ParDo.of(
new DoFn<KV<K, CoGbkResult>, T>() {
#Override
public void processElement(ProcessContext c) {
KV<K, CoGbkResult> e = c.element();
// Get all LogEvent1 values
Iterable<LogEvent1> event1s = e.getValue().getAll(tag1);
// There will only be one LogEvent2
LogEvent2 event2 = e.getValue().getOnly(tag2);
... Do Something to compute T ....
c.output(...some T...);
}
}));
The above example was adapted from docs on CoGroupByKey which have information.
Dataflow does not currently expose the underlying state mechanism that it uses. However, this is definitely on the radar for a future update.
i'm displaying a server calculated value to the enduser by using propertyChanged event.
i was using breeze 1.4.8 and i'm using the productivity stack (ms sql, web api, ef)
It was working fine.
Recently i've updated to 1.4.12 and i recognized that this event doesn't get fired anymore.
The property "A_ProvisionTotal" gets calculated serverside only.
<snip>
var token = vm.transaction.entityAspect.propertyChanged.subscribe(propertyChanged);
function propertyChanged(propertyChangedArgs) {
var propertyName = propertyChangedArgs.propertyName;
if (vm.transaction.tblEmployees.CalculationMethod == "A" && propertyName == "A_ProvisionTotal")
logSuccess('Provision neuberechnet' + '<br/>' + 'Aktuell: ' + $filter('number')(vm.transaction.Provision, 2), true);
</snip>
Let me know if this is a known regression and if you need more snippets.
A couple of thoughts for how you could accomplish your desired functionality.
The entity could remember the last calculated value in a private field. Then whenever the recalculation gets triggered, you can compare the new value to the last calculated value and if there is no change, ignore the new calculated value.
Alternatively, you could define the properties involved in your calculation as ES5 properties in the entity ctor function and then trigger the calculation in the setter of the relevant properties, when they get set with a new value. More information here: http://www.breezejs.com/documentation/extending-entities#es5-property. ES5 properties are convenient if you want to build behavior such as your calculation into setters.
Update 3
This is not a bug - see the response to this post that describes this as a documented and deliberate behavior.
Update 2 June 2014
I overlooked a key fact in your question ... one that only became clear to me after I looked at the code you included in your comments. Let me extract the key pieces for other readers:
Your test issues a query, then saves an unrelated change to the server (where the property-of-interest is updated server-side), then checks if that telltale property-of-interest raises propertyChanged when the save result is merged back into cache.
var query = EntityQuery.from("Orders").where('id', 'eq', 10248);
query.using(em).execute().then(querySucceeded).then(checkPropertyChanged).fin(start);
// querySucceed receives order 10248, updates an unrelated property (so you can save),
// wires up a propertyChanged listener and saves, returning the saveChanges promise
function checkPropertyChanged(saveResults) {
var saved = saveResults.entities[0];
// this passes because the server-side change to `Freight` was returned
ok(saved && saved.Freight() === 1200.00,
"freight got changed serverside");
// this fails because Breeze did not notify you that the `Freight` had changed
ok(notifications[0].propertyName === "Freight",
"notified serverside change of Freight Property");
}
Summarizing, you expected that a property change on the server would trigger a propertyChanged notification on the client when the entity data are re-retrieved from the server as a by-product of saveChanges.
Do I have that right?
Our documentation was not clear on whether the merge of query, save, and import entity results would raise propertyChanged.
I discussed internally and confirmed that these operations SHOULD raise propertyChanged. I also wrote another (somewhat simpler) test that reveals the bug you discovered: that merged save results may not raise propertyChanged.
We'll look into it and tell you when we've fixed it. Thanks for discovering it.
Original
We have regression tests that show that the Breeze EntityAspect.propertyChanged event is raised in v.1.4.12. For example, you can see it at work in the DocCode sample, "basicTodoTests.js"; scroll to: "Breeze propertyChanged raised when any property changes".
Can you confirm that it really is a Breeze failure? Perhaps the property you are changing is not actually an entity property? Sometimes you think you are changing an entity (e.g, your Transaction entity) but the thing whose property you changed isn't actually an entity. Then the problem is that the data you thought would be mapped to a Transaction was not ... and you can start looking for that quite different problem.
In any case, I suggest that you write a small test to confirm your suspicion ... most importantly for yourself ... and then for us. That will help us discover what is different about your scenario from our scenarios. We'll fix it if you can find it. Thanks.
Actually, I'm not sure that this is a bug. Property change events DO get fired during a save merge but the property name parameter is documented as being 'null' when fired as a result of a save.
http://www.breezejs.com/sites/all/apidocs/classes/EntityAspect.html#event_propertyChanged
From the API Docs for the 'propertyName' parameter returned by EntityAspect.propertyChanged:
The name of the property that changed. This value will be 'null' for operations that replace the entire entity. This includes queries, imports and saves that require a merge. The remaining parameters will not exist in this case either.
What may have happened between 1.4.8 and 1.4.13 is that we actually implemented our design spec more carefully and probably introduced your breaking behavior. ( which we should have documented as such but likely missed).
Update by Ward
I updated the DocCode test which first confirmed the behavior described in your question and then confirmed the documented behavior.
We do regret that we apparently neglected to implement the documented behavior earlier and that we didn't mention the breaking change in our release notes (since updated).
Here's that test:
asyncTest("propertyChanged raised when merged save result changes a property", 3, function () {
var em = newTodosEm();
var todo = em.createEntity('TodoItem', {Description: "Saved description" });
em.saveChanges().then(saveSucceeded).catch(handleFail).finally(start);
ok(todo.entityAspect.isBeingSaved, "new todo is in the act of being saved");
// This change should be overwritten with the server value when the save result is returned
// even though the entity is in an Added state and the MergeStrategy is PreserveChanges
// because save expects to merge server values into an entity it is saving
todo.Description("Changed on client before save returns");
var descriptionChanged = false;
todo.entityAspect.propertyChanged.subscribe(function (changeArgs) {
// Watch carefully! The subscription is called twice during merge
// 1) propertyName === "Id" (assigned with permanent ID)
// 2) propertyName === null (WAT?)
// and not called with propertyName === "Description" as you might have thought.
// Actually 'null' means "merged a lot of properties"
// Documented: http://www.breezejs.com/sites/all/apidocs/classes/EntityAspect.html#event_propertyChanged
// The reason for this: don't want to fire a ton of events on whole entity load
// especially when merging many entities at the same time.
if (changeArgs.propertyName === null || changeArgs.propertyName === 'Description') {
descriptionChanged = true;
}
});
function saveSucceeded(saveResult) {
var saved = saveResult.entities[0];
// passes
equal(saved && saved.Description(), "Saved description",
"the merge after save should have restored the saved description");
// fails
ok(descriptionChanged,
"should have raised propertyChanged after merge/update of 'Description' property");
}
});
Background:
From another question here at SO I have a Winforms solution (Finance) with many projects (fixed projects for the solution).
Now one of my customers asked me to "upgrade" the solution and add projects/modules that will come from another Winforms solution (HR).
I really don't want to keep these projects as fixed projects on the existing finance solution. For that I'm trying to create plugins that will load GUI, business logic and the data layer all using MEF.
Question:
I have a context (DbContext built to implment the Generic Repository Pattern) with a list of external contexts (loaded using MEF - these contexts represent the contexts from each plugin, also with the Generic Repository Pattern).
Let's say I have this:
public class MainContext : DbContext
{
public List<IPluginContext> ExternalContexts { get; set; }
// other stuff here
}
and
public class PluginContext_A : DbContext, IPluginContext
{ /* Items from this context */ }
public class PluginContext_B : DbContext, IPluginContext
{ /* Items from this context */ }
and within the MainContext class, already loaded, I have both external contexts (from plugins).
With that in mind, let's say I have a transaction that will impact both the MainContext and the PluginContext_B.
How to perform update/insert/delete on both contexts within one transaction (unity of work)?
Using the IUnityOfWork I can set the SaveChanges() for the last item but as far as I know I must have a single context for it to work as a single transaction.
There's a way using the MSDTC (TransactionScope) but this approach is terrible and I'd reather not use this at all (also because I need to enable MSDTC on clients and server and I've had crashes and leaks all the time).
Update:
Systems are using SQL 2008 R2. Never bellow.
If it's possible to use TransactionScope in a way that won't scale to MSDTC it's fine, but I've never achieved that. All the time I've used TransactionScope it goes into MSDTC. According to another post on SO, there are some cases where TS will not go into MSDTC: check here. But I'd really prefer to go into some other way instead of TransactionScope...
If you are using multiple contexts each using separate connection and you want to save data to those context in single transaction you must use TransactionScope with distributed transaction (MSDTC).
Your linked question is not that case because in that scenario first connection do not modify data so it can be closed prior to starting the connection where data are modified. In your case data are concurrently modified on multiple connection which requires two-phase commit and MSDTC.
You can try to solve it with sharing single connection among multiple contexts but that can be quite tricky. I'm not sure how reliable the following sample is but you can give it a try:
using (var connection = new SqlConnection(connnectionString))
{
var c1 = new Context(connection);
var c2 = new Context(connection);
c1.MyEntities.Add(new MyEntity() { Name = "A" });
c2.MyEntities.Add(new MyEntity() { Name = "B" });
connection.Open();
using (var scope = new TransactionScope())
{
// This is necessary because DbContext doesnt't contain necessary methods
ObjectContext obj1 = ((IObjectContextAdapter)c1).ObjectContext;
obj1.SaveChanges(SaveOptions.DetectChangesBeforeSave);
ObjectContext obj2 = ((IObjectContextAdapter)c2).ObjectContext;
obj2.SaveChanges(SaveOptions.DetectChangesBeforeSave);
scope.Complete();
// Only after successful commit of both save operations we can accept changes
// otherwise in rollback caused by second context the changes from the first
// context will be already accepted = lost
obj1.AcceptAllChanges();
obj2.AcceptAllChanges();
}
}
Context constructor is defined as:
public Context(DbConnection connection) : base(connection,false) { }
The sample itself worked for me but it has multiple problems:
First usage of contexts must be done with closed connection. That is the reason why I'm adding entities prior to opening the connection.
I rather open connection manually outside of the transaction but perhaps it is not needed.
Both save changes successfully run and Transaction.Current has empty distributed transaction Id so it should be still local.
The saving is much more complicated and you must use ObjectContext because DbContext doesn't have all necessary methods.
It doesn't have to work in every scenario. Even MSDN claims this:
Promotion of a transaction to a DTC may occur when a connection is
closed and reopened within a single transaction. Because the Entity
Framework opens and closes the connection automatically, you should
consider manually opening and closing the connection to avoid
transaction promotion.
The problem with DbContext API is that it closes and reopens connection even if you open it manually so it is a opened question if API always correctly identifies if it runs in the context of transaction and do not close connection.
#Ladislav Mrnka
You were right from the start: I have to use MSDTC.
I've tried multiple things here including the sample code I've provided.
I've tested it many times with changed hare and there but it won't work. The error goes deep into how EF and DbContext works and for that to change I'd finally find myself with my very own ORM tool. It's not the case.
I've also talked to a friend (MVP) that know a lot about EF too.
We have tested some other things here but it won't work the way I want it to. I'll end up with multiple isolated transactions (I was trying to get them together with my sample code) and with this approach I don't have any way to enforce a full rollback automatically and I'll have to create a lot of generic/custom code to manually rollback changes and here comes another question: what if this sort of rollback fails (it's not a rollback, just an update)?
So, the only way we found here is to use the MSDTC and build some tools to help debug/test if DTC is enabled, if client/server firewalls are ok and all that stuff.
Thanks anyway.
=)
So, any chance this has changed by October 19th? All over the intertubes, people suggest the following code, and it doesn't work:
(_contextA as IObjectContextAdapter).ObjectContext.Connection.Open();
(_contextB as IObjectContextAdapter).ObjectContext.Connection.Open();
using (var transaction = new TransactionScope(TransactionScopeOption.Required, new TransactionOptions{IsolationLevel = IsolationLevel.ReadUncommitted, Timeout = TimeSpan.MaxValue}))
{
_contextA.SaveChanges();
_contextB.SaveChanges();
// Commit the transaction
transaction.Complete();
}
// Close open connections
(_contextA as IObjectContextAdapter).ObjectContext.Connection.Close();
(_contextB as IObjectContextAdapter).ObjectContext.Connection.Close();
This is a serious drag for implementing a single Unit of Work class across repositories. Any new way around this?
To avoid using MSDTC (distributed transaction):
This should force you to use one connection within the transaction as well as just one transaction. It should throw an exception otherwise.
Note: At least EF6 is required
class TransactionsExample
{
static void UsingExternalTransaction()
{
using (var conn = new SqlConnection("..."))
{
conn.Open();
using (var sqlTxn = conn.BeginTransaction(System.Data.IsolationLevel.Snapshot))
{
try
{
var sqlCommand = new SqlCommand();
sqlCommand.Connection = conn;
sqlCommand.Transaction = sqlTxn;
sqlCommand.CommandText =
#"UPDATE Blogs SET Rating = 5" +
" WHERE Name LIKE '%Entity Framework%'";
sqlCommand.ExecuteNonQuery();
using (var context =
new BloggingContext(conn, contextOwnsConnection: false))
{
context.Database.UseTransaction(sqlTxn);
var query = context.Posts.Where(p => p.Blog.Rating >= 5);
foreach (var post in query)
{
post.Title += "[Cool Blog]";
}
context.SaveChanges();
}
sqlTxn.Commit();
}
catch (Exception)
{
sqlTxn.Rollback();
}
}
}
}
}
Source:
http://msdn.microsoft.com/en-us/data/dn456843.aspx#existing