Can I implement a table lock on a sequence container?
My goal here is to be have N concurrent executions of the same package.
Here is a rough architecture of the package...
Is there a way I can table lock the sequence container so that my batches won't overlap?
I am setting the batch as
select top 10
from table
where isprocessed = 0 and batchID is null
order by [identity column]
Maybe a logical change in strategy?
UPDATE:
I made this work by changing the update from a dataflow to just an execute SQL update statement which SQL server automatically locks using UPDATE LOCK.
Related
We have TFS 2017.3 and the database it's huge - about 1.6 TB.
I want to try to clean up the space by running these 2 stored procedures:
prc_CleanupDeletedFileContent
prc_DeleteUnusedFiles
prc_DeleteUnusedContent
Is it safe to run it?
Is there a chance that it will delete important things I am currently using? (of course, I will do a backup before...)
What are the best values to put in these stored procedures?
Another thing - If I run this query:
SELECT A.[ResourceId]
FROM [Tfs_DefaultCollection].[dbo].[tbl_Content] As A
left join [Tfs_DefaultCollection].[dbo].[tbl_FileMetadata] As B on A.ResourceId=B.ResourceId
where B.[ResourceId] IS Null
I got result of 10681.
If I run this query:
SELECT A.[ResourceId]
FROM PTU_NICE_Coll.[dbo].[tbl_Content] As A
left join PTU_NICE_Coll.[dbo].tbl_FileReference As B on A.ResourceId=B.ResourceId
where B.[ResourceId] IS Null
I got result of 10896.
How can I remove this rows? and is it completely safe to remove them?
Generally we don't recommend to do actions against the DB directly as it may cause problems.
However if you have to do that, then you need to backup the DBs first.
You can refer to below articles to clean up and reduce the size of the TFS databases:
Control\Reduce TFS DB Size
Cleaning up and reduce the size of the TFS database
Clean up your Team Project Collection
Another option is to dive deep into the database, and run the cleanup
stored procedures manually. If your Content table is large:
EXEC prc_DeleteUnusedContent 1
If your Files table is large:
EXEC prc_DeleteUnusedFiles 1, 0, 1000
This second sproc may run for a long time, that’s why it has the third
parameter which defines the batch size. You should run this sprocs
multiple times, or if it completes quickly, you can increase the chunk
size.
we have oledbCommand Transformation for Update,delete pupose in SSIS it works fine but it will reduce the performance when we have huge amount of data so we can write a query in execute sql task in control flow pane in ssis for insert,update record i know this works but i don't know how to do so guys please help me.
You can achieve this in two ways using execute sql task.
1.Insert data into global temporary table,then use execute sql task update your target table with join temporary table and can also insert data to your target table using temporary table.
2. Insert data into in dummy table(database) then update and insert target table using this dummy table. You can use merge statement for insert and update.
I have a stored procedure which starts by creating a temp table, then it populates the temp table based on a query.
It then does a Merge statement, basically updating a physical table off of the temp table. Lastly it does an Update on the physical table. Pretty basic stuff.
It takes ~8 sec to run. My question is, at what point does it lock the physical table? Since the whole query is compiled prior to its run, is the physical table locked during the entire execution of the stored procedure, or does it wait until it gets to the statements that actually works with the physical table?
I'm not necessarily trying to resolve an issue, as much as make sure I don't cause one. We have other processes that need to be reworked to alleviate blocking, I don't want to create another.
OK, for SQL Server:
a stored procedure is "compiled" (e.g. an execution plan is determined) before it's first usage (and that execution plan is cached in the plan cache until it's either tossed out due to space constraints, or due to a restart). At the time an execution plan is determined, nothing happens on the table level - no locks, nothing
SQL Server will by default use row-level locks, e.g. a row is locked when it's read, inserted, updated or deleted - then and only then. So your procedure will place shared locks on the tables where it selects the data from to insert those rows into the temp table, it will put exclusive locks on the rows being inserted into the temp table (for the duration of the operation / transaction), and the MERGE will also place locks as and when needed
Most locks (not the shared locks, typically) in SQL Server are by default held until the end of the transaction. Unless you specifically handle transactions explicitly, each operation (MERGE, UPDATE, INSERT) runs inside its own implicit transaction, so any locks held will be released at the end of that transaction (e.g. that statement).
There are lots more aspects to locks - one could write entire books to cover all the details - but does this help you understand locking in SQL Server at least a little bit ?
I am just now starting to dig into Teradata's locking features and Google is fairly convoluted with explanations on this. Hopefully, I can get a very simple and streamlined answer from SE.
After encountering numerous issues with identity columns in Teradata, I've decided to create a mechanism that mimics Oracle's sequence. To do this, I am creating a table with two fields, one that holds a table name and the other that stores its last-used sequence. I am going to then create a stored procedure that takes a table name. Within the procedure, it will perform the following options:
Select the last-used sequence from the sequence table into a variable (select LastId from mydb.sequence where tablename = :tablename)
Add 1 to the variable's value, thus incrementing it
Update the sequence table to use the incremented value
Return the sequence variable to the procedure's OUT param so I can access the sequenced ID in my .NET app
While all of those operations are taking place, I need to lock the sequence table for all read and write access to ensure that other calls to the procedure do not attempt to sequence the table while it is currently in the process of being sequenced. This is obviously to keep the same sequence from being used twice for the same table.
If this were .NET, I'd use a Sync Lock (VB.NET) or lock (C#) to keep other threads from entering a block of code until the current thread was finished. I am wondering if there's a way to lock a table much in the same way that I would lock a thread in .NET.
Consider using an explicit locking mechanism for a rowhash lock for the transaction:
BEGIN TRANSACTION;
LOCKING ROW EXCLUSIVE
SELECT LastId + 1 INTO :LastID
FROM MyDB.SequenceCols
WHERE TableName = :TableName
AND DatabaseName = :DatabaseName;
UPDATE MyDB.SequenceCols
SET LastId = :LastID
WHERE TableName = :TableName
AND DatabaseName = :DatabaseName;
END TRANSACTION;
The rowhash lock will allow the procedure to be used by other processes against other tables. To ensure row level locking you must fully qualify the primary index of the SequenceCols table. In fact, the primary index of the SequenceCols table should be UNIQUE on DatabaseName and TableName.
EDIT:
The exclusive rowhash lock would prevent another process from reading the row until the END TRANSACTION is processed owner of the rowhash lock.
I am putting together a job on SQL Enterprise Manager 2000 to copy and delete records in a couple database tables. We've run a straight up mass copy and delete stored procedure, but it could be running it on millions of rows, and therefore hangs the server. I was interested in trying to run the service in 100-ish record chunks at a time, so the server doesn't grind to a halt (this is a live web database). I want this service to run once a night, which is why I've put it in an agent job. Is there any way to loop the calls to the stored procedures that actually do the copy and delete, and then "sleep" in between each call to give the server time to catch up? I know there is the WAITFOR command, but I'm unsure if this will hold the processor or let it run other queries in the meantime.
Thanks!
"Chunkifying" your deletes is the preferred way to delete excessive amounts of data without bloating up transaction log files. BradC's post is a reasonable example of this.
Managing such loops is best done within a single stored procedure. To spread such work out over time, I'd still keep it in the procedure. Inserting a WAITFOR in the loop will put a "pause" between each set of deletes, if you deem that necessary to deal with possible concurrency issues. Use a SQL Agent job to determine when the procedure start--and if you need to make sure it stops by a certain time, work that into the loop as well.
My spin on this code would be:
-- NOTE: This is a code sample, I have not tested it
CREATE PROCEDURE ArchiveData
#StopBy DateTime
-- Pass in a cutoff time. If it runs this long, the procedure will stop.
AS
DECLARE #LastBatch int
SET #LastBatch = 1
-- Initialized to make sure the loop runs at least once
WHILE #LastBatch > 0
BEGIN
WAITFOR DELAY '00:00:02'
-- Set this to your desired delay factor
DELETE top 1000 -- Or however many per pass are desired
from SourceTable
-- Be sure to add a where clause if you don't want to delete everything!
SET #LastBatch = ##rowcount
IF getdate() > #StopBy
SET #LastBatch = 0
END
RETURN 0
Hmm. Rereading you post implies that you want to copy the data somewhere first before deleting it. To do that, I'd set up a temp table, and inside the loop first truncate the temp table, then copy in the primary keys of the TOP N items, insert into the "archive" table via a join to the temp table, then delete the source table also via a join to the temp table. (Just a bit more complex than a straight delete, isn't it?)
Don't worry about waiting between loops, SQL server should handle the contention between your maintenance job and the regular activity on the server.
What really causes the problem in these types of situations is that the entire delete process happens all at once, inside a single transaction. This blows up the log for the database, and can cause the kinds of problems it sounds like you are experiencing.
Use a loop like this to delete in manageable chunks:
DECLARE #i INT
SET #i = 1
SET ROWCOUNT 10000
WHILE #i > 0
BEGIN
BEGIN TRAN
DELETE TOP 1000 FROM dbo.SuperBigTable
WHERE RowDate < '2009-01-01'
COMMIT
SELECT #i = ##ROWCOUNT
END
SET ROWCOUNT 0
You can use similar logic for your copy.
WAITFOR will let other processes 'have a go'. I've used this technique to stop large DELETE's locking up the machine. Create a WHILE loop, delete a block of rows, and then WAITFOR a few seconds (or less, whatever is appropriate).