I have an Analysis Services Cube in TEST and PROD.
We have recently started using branches in TFS.
After our development sprint and deployments, we will restore a backup of PROD over TEST, with the cubes coming through in their processed state.
However, recently, if we deploy the exact same project back to TEST (so exact same schema as what is deployed), all cubes become unprocessed.
I don't think it ever used to do this. For example, we tried changing a translation, not expecting to have to reprocess the cube and found that the whole thing required processing (which takes many hours).
Any idea why? It feels like SSAS is thinking that this is a new database and recreating it (although the created timestamps on the database are still from many months ago)
Steps to reproduce:
1. Create new branch called 'Branch A' from the master branch (master branch is currently deployed to our Analysis Server).
2. Redeploy this new Branch (even without changes)
3. All cubes now become Unprocessed (and deployed seemed to take longer than normal)
Try to go to the Warehouse Control Web Service, and check the Processing Status, then manually process the data warehouse relational database by following article Manually Process the Data Warehouse and Analysis Services Cube for Team Foundation Server.
To access the Warehouse Control Web Service:
Log on to the application-tier server.
Open Internet Explorer, type the following string in the Address bar, and then press ENTER:
-
http://localhost:8080/tfs/TeamFoundation/Administration/v3.0/WarehouseControlService.asmx
If manually process Data Warehouse doesn't work, try to rebuild the data warehouse by following article Rebuild the Data Warehouse and Analysis Services Cube.
Related
I wrote a silly amount on a task in a sprint. I filled in the value "1,000,000" for "Remaining Work". Now the Burndown chart has adjusted itself on the scale of a million. That means our piddly 80 hours of team work isn't showing on the chart at all :(.
Can I remove this value from remaining work somehow now that it has been added and saved?
Yes, just edit it to the value you want and resave.
Reporting in TFS makes use of Sql Server Analysis Server, or OLAP cube. This cube is by default refreshed every two hours. After you edit the value, you can manually process the TFS data warehouse and analysis services cube, then check the Burndown chat again.
Summary how to manually process the TFS data warehouse and analysis services cube:
Log on to the application-tier server and open the Warehouse Control Web Service by entering the following URL in a supported web browser:
http://localhost:8080/tfs/TeamFoundation/Administration/v3.0/WarehouseControlService.asmx
Choose ProcessWarehouse, click Invoke:
The service returns True when it successfully starts the processing of the warehouse and False if it is not successful. A value of False indicates that the warehouse is currently being processed.
To determine the processing status of the warehouse choose GetProcessingStatus to check the process status.
Warehouse processing is completed when the GetProcessingStatus service returns a value of Idle, and a value of Succeeded for Common Structures Warehouse Sync and for each warehouse for each team project collection.
Choose ProcessAnalysisDatabase. For processingType, type either Incremental or Full. Click Invoke:
The service returns True when it successfully starts the processing of the cube and False if it is not successful. A value of False indicates that the cube is currently being processed.
Cube processing is completed when the GetProcessingStatus service returns a value of Idle, and a value of Succeeded for the following jobs.
Full Analysis Database Sync
Incremental Analysis Database Sync
If manually processing still doesn't work, you can try to delete the Tfs_Analysis database and recreate it using TFS Admin Console. To delete the Tfs_Analysis database, you can connect to Analysis Service using Microsoft SQL Server Management Studio, expand the Databases node, and delete the Tfs_Analysis database.
To recreate the Tfs_Analysis database, please refer to the following steps:
Launch TFS Admin Console>>Application Tier>>Reporting, in the right panel, click Edit to open the Reporting windows.
Under the Analysis Services tab, input the Tfs_Analysis under Database textbox, and re-provide the required account under Account for accessing data source.
Click Ok to generate the Tfs_Analysis database in Analysis Services.
Click Start Jobs and Start Rebuild to rebuild warehouse and the Analysis Services database.
I'm working on a project in which we have two versions of an MVC App, the live, and the dev versions, I've made changes to the dev version and added tables and data, etc.
Is there any way to migrate these changes onto the live version without losing all data(i.e. just regenerating the database).
I've already tried just rebuilding the database but we lose all data that was previously stored( as obviously we are essentially deleting the old database and rebuilding it).
tl;dr
How do I migrate my dev version of an mvc app along with any new tables to the live version of an mvc app with missing models and tables.
Yes, it is possible to migrate your changes from your dev instance to your production instance; to do so you must create SQL scripts that update your production database with the changes. This may be accomplished by manually writing the scripts or by using tools to generate the scripts for you. However you go about it, you will need scripts to update your database (well, you could perform manual updates via the tooling of your database, but this is not desirable, as you want the updates to occur in a short time window, and you want this to happen reliably and repeatably).
The easiest way that I know of to do this is to use tools like SQL Compare (for schema updates) or SQL Data Compare (for data updates). These are from Redgate, but they cost a fair bit of money. They are well worth they price, and most companies I've worked with are happy to pay for licenses. However, you may not want to shell out for them personally. To use these tools, you connect them to source and destination databases, and they analyze the differences between the databases (schematically or data) and produce SQL scripts. These scripts may then be run manually or by the tools themselves.
Ideally, when you work on your application, you should be producing these scripts as you go along. That way when it comes time to update your environments, you may simply run the scripts you have. It is worth taking the time to include this in your build process, so database scripts get included in your builds. You should write your scripts so they are idempotent, meaning that you can run them multiple times and the end result will be the same (the database updated to the desired schema and data).
One may of managing this is creating a DBVersions table in your database. This table includes all your script updates. For example you could have a table like the following (this is SQL Server 2008 dialect):
CREATE TABLE [dbo].[DBVersions] (
[CaseID] [int] NOT NULL,
[DateExecutedOn] [datetime] NOT NULL,
CONSTRAINT [PK_DBVersions] PRIMARY KEY CLUSTERED (
[CaseID] ASC
)
) ON [PRIMARY]
CaseID refers to the case (or issue) number of the feature or bug that requires the SQL update. Your build process can check this table to see if the script has already been run. If not, it runs it. This is useful if you cannot write your scripts in a way that allows them to be run more than once. If all your scripts can be run an unbounded number of times, then this table is not strictly necessary, though it could still be useful to reduce the need to run a large number of script every time a deployment is done.
Here are links to the Redgate tools. There may be many other tools out there, but I've had a very good experience with these.
http://www.red-gate.com/products/sql-development/sql-compare/
http://www.red-gate.com/products/sql-development/sql-data-compare/
It depends on your deployment strategy and this is more of a workflow that your team need to embrace. If regenerating the live database from scratch it can take awhile depending how big the database size is. I don't see a need to do this in most scenarios.
You would only need to separate out database schema object and data row scripts. The live database version should have its database schema objects scripted out and stored in a repository. When a developer is working on a new functionality, he/she will need to make those changes against the database scripts in the repository. If there is a need to make changes to the database rows then the developer will also need to check in the data row scripts in the repository. On a daily deployment the live database version can be compared against what is checked in the repository and pushed to make it in sync.
On our side we use tools such as RedGate Schema Compare and Data Compare to do the database migration from the dev version to our intended target version.
Currently I'm working on a Rails project, where I keep on constantly switching between the deployable master branch, and then many other branches, where I implement new features.
The problem is, that usually these features add some tables to the database, which means every time I switch a branch, I have to drop the database, migrate and then populate it with some dummy data.
I can do this in about two to three steps, since I have a rake task that creates all the dummy data again, but it's not very fast (couple of minutes). It's not the worst wait time ever, but I'd like to know if there are any alternative solutions, where I don't have to recreate the database every time I checkout a branch.
I'm currently using MySQL on my development machine.
Why don't keep the databases for each branch and just switch the connection strings.
I am building an app that is fast moving into production and I am concerned about the possibility that due to hacking, some silly personal error (like running rake db:schema:load or rake db:rollback) or other circumstance we may suffer data loss in one database table or even across the system.
While I don't find it likely that the above will happen, I would be remiss in not being prepared in case it ever does.
I am using Heroku's PG Backups (which is to be replaced with something else this month), and I also run automated daily backups to S3: http://trevorturk.com/2010/04/14/automated-heroku-backups/, successfully generating .dump files.
What is the correct way to deal with data loss on a production app?
How would I restore the .dump file in case I need to? Can I do a selective restore if a small part of the system is hit?
In case a selective restore is not possible: assume one table loses data 4 hours after the last backup. Result => would fixing the lost table require rolling back 4 hours of users' activity? Any good solution to this?
What is the best way to support users through the inconvenience if something like this happens?
A full DR (disaster recovery) solution requires the following:
Multisite. If a fire, flood, Osama Bin Laden or whathaveyou strikes the Amazon (or is it Salesforce?) data center that Heroku uses, you want to be sure that your data is safe elsewhere.
On-going replication of the data to a separate site (or sites). That means that every transaction that's written to your database on one site, is replicated within seconds to the mirror database on the other site. Most RDBMS's have mechanisms to let you do a master-slave replication like that.
The same goes for anything you put on a filesystem outside of the database, such as images, XML configuration files etc. S3 is a good solution here - they replicate everything to multiple data centers for you.
I won't hurt to create periodic (daily or so) dumps of the database and store them separately (e.g. on S3). This helps you recover from data corruption that propagates to the slave DBs.
Automate the process of data recovery. You want this to just work when you need it.
Test everything. Ideally, you want to automate the test process and run it periodically to ensure that your backups can restore. Netflix Chaos Monkey is an extreme example of this.
I'm not sure how you'd implement all this on Heroku. A complete solution is still priced out of reach for most companies - we're running this across our own data centers (one in the US, one in EU) and it costs many millions. Work according to the 80-20 rule - on-going backup to a separate site, plus a well tested recovery plan (continuously test your ability to recover from backups) covers 80% of what you need.
As for supporting users, the best solution is simply to communicate timely and truthfully when trouble happens and make sure you don't lose any data. If your users are paying for your service (i.e. you're not ad-supported), then you should probably have an SLA in place.
About backups, you cannot be sure at 100 percent every time that no data will be lost. The best is to test it on another server. You must have at leat two types of backup :
A database backup, like pg-dump. A dump is uniquely SQL commands so you can use it to recreate the whole database, just a table, or just a few rows. You loose the data added in the meantime.
A code backup, for example a git repository.
in addition to Hartator's answer:
use replication if your DB offers it, e.g. at least master/slave replication with one slave
do database backups on a slave DB server and store them externally (e.g. scp or rsync them out of your server)
use a good version control system for your source code, e.g. Git
use a solid deploy mechanism, such as Capistrano and write your custom tasks, so nobody needs to do DB migrations by hand
have somebody you trust check your firewall setup and the security of your system in general
The DB-Dumps contain SQL-commands to recreate all tables and all data... if you were to restore only one table, you could extract that portion from a copy of the dump file and (very carefully) edit it and then restore with the modified dump file (for one table).
Always restore first to an independent machine and check if the data looks right. e.g. you could use one Slave server, take if offline, then restore there locally and check the data. Good if you have two slaves in your system, then the remaining system has still one master and one slave while you restore to the second slave.
To simulate a fairly simple "total disaster recovery" on Heroku, create another Heroku project and replicate your production application completely (except use a different custom domain name).
You can add multiple remote git targets to a single git repository so you can use your current production code base. You can push your database backups to the replicated project, and then you should be good to go.
The only step missing from this exercise verses a real disaster recovery is assigning your production domain to the replicated Heroku project.
If you can afford to run two copies of your application in parallel, you could automate this exercise and have it replicate itself on a regular basis (e.g. hourly, daily) based on your data loss tolerance.
I have started at a new site that is using .Net applications for the first time. As a developer I am used to VSS but this product is dying a death so we are using TFS (BASIC) instead.
I have been using TFS for source control up until now. But now we are having new servers installed for a live environment.
Now I am not sure what I should be doing. There are no books on TFS 2010 that I can find and I am wondering what tips you can give me. Does TFS need to be installed again, or should I use the existing installation? I am thinking I ought to set up a daily build for a test server. I have not been using TDD up until now, but for the next project this may change.
What must I absolutely get right, and what pitfuls should I avoid?
Without being there in your environment, it's hard to make appropriate recommendations. I've made some assumptions about what your installation based on what you said, but these may be wildly wrong.
You say you're using TFS (BASIC)-- I'm not sure what you mean by that, but if you are using TFS installed on one of the developers workstations, and you're starting to move towards a more robust development environment, I would recommend that you get a separate server (or servers) for your TFS installation.
It sounds like you're relatively small, so having your application tier and your data tier on the same machine shouldn't be that much of an issue. Just make sure that you have enough RAM on the machine to support both processes, and that you have enough disk space allocated for the growth of the database.
You talk about Test Driven Development (TDD), but what I think you're actually talking about is Continuous Integration (CI). When you have a CI environment set up, builds happen automatically based on either a schedule, or triggered by check-ins. Having this set up is never a bad idea, and would recommend that you get into the rhythm of CI builds as soon as possible.
If you're looking for a build server, you are probably going to be ok hosting the build agent on the combined application/data tier. If you find that you're getting performance hits when you do builds, you can move your builds to a different server without much effort.
You will also want to look at migrating your source code repository from your current environment to your future environment. The TFS installation wizard might be able to help you with that. If not, there are other options available, such as moving the database files to the new machine, or using the codeplex-based TFS Integration Platform.