SCDF project creates tables/sequences on the fly while starting the application. but our database is in shared environment, it doesn't have any create table permission. Is it possible to create tables/indices/sequences well before that are needed for Spring Cloud Data Flow project to run ?. if you have any SQL scripts, please provide me the link ?..
The SCDF and Skipper rely on Flyway for database schema migration between the releases. It is a strict requirement, so, unfortunately, there's no way around it right now.
You could work around this situation by letting Flyway create all the required schemas automatically in either dev/staging like environments and then manually you could export the schema/data using the approved database tooling to the other environments that have restricted access.
Once when the tables/data exists, Flyway wouldn't interfere or reattempt to create the schemas when SCDF bootstraps. I hope this helps.
Related
I'm using SCDF and i was wondering if there was any way to configure default properties for one application?
I got a task application registered in SCDF and this application gets some JDBC properties to access business database :
app.foo.export.datasource.url=jdbc:db2://blablabla
app.foo.export.datasource.username=testuser
app.foo.export.datasource.password=**************
app.foo.export.datasource.driverClassName=com.ibm.db2.jcc.DB2Driver
Do i really need to put this prop in a property file like this : (it's bit weird to define them during the launch)
task launch fooTask --propertiesFile aaa.properties
Also, we cannot use the rest API, credentials would appear in the url.
Or is there another way/place to define default business props for an application ? These props will be only used by this task.
The purpose is to have one place where OPS team can configure url and credentials without playing with the launch command.
Thank you.
Yeah, SCDF feels a bit weird in the configuration area.
As you wrote, you can register an application and create tasks, but all the configuration is passed at the first launch of the task. Speaking other way round, you can't fully install/configure a task without running it.
As soon as a task has run once, you can relaunch it without any configuration and it uses the configuration from before. The whole config is saved in the SCDF database.
However, if you try to overwrite an existing configuration property with a new value, SCDF seems to ignore the new value and continue to use the old one. No idea if this is intended by design or a bug or if we are doing something wrong.
Because we run SCDF tasks on Kubernetes and we are used to configure all infrastructure in YAML files, the best option we found was to write our own Operator for SCDF.
This operator works against the REST interface of SCDF and also compensates the weird configuration issues mentioned above.
For example the overwrite issue is solved by first deleting the configuration and recreate it with the new values.
With this operator we have reached what you are looking for: all our SCDF configuration is in a git repository and all changes are done through merge requests. Thanks to CI/CD, on the next launch, the new configuration is used.
However, a Kubernetes operator should be part of the product. Without it, SCDF on Kubernetes feels quite "alien".
I've got a Grails plugin that exports domain objects so that several applications can share the same schema. We have a few SQL scripts for setting up some complex triggers, views and other functions that just don't really belong in GORM/Hibernate, at least not elegantly. I'd like to store the scripts inside the same project. Is the "scripts" folder (the one containing _[Un]Install/Upgrade.groovy) the best place for this? I saw a StackOverflow answer that was building a catalog from scripts stored in grails-app/conf/sql. But I'm not actually trying to execute them from within the a project.
The absolute best solution for anything database related is to use the database migration plugin. This way you can ensure that any database your application is pointed to (dev, test, prod, etc.) will have the same information/schema/functions/procedures etc.
Personal preference. I usually add a 'database' dir for all that kind of stuff. The 'scripts' dir is for Grails scripts, at least in 1.x and 2.x. See Creating Gant Scripts or the create-script command for more on those. In Grails 3 these kind of scripts have been moved to src/main/scripts.
I'm currently working with the Grails tool suit with eclipse. I created an an application, defined a domain class and my app works great. My question is, when I deploy my war file how is the database stored? Do I link my data source file to an sql database url. If so upon running my app the first time does Grails create the database for you? You probably understand my question by now. How does this work?
I've looked at this documentation and cant find how grails goes about with creating the database I defined.
http://grails.org/doc/latest/guide/conf.html#dataSourcesAndEnvironments
First off, with the exception of H2 Grails does not setup your database. You will need to setup the database, and configure your datasource to connect to the database.
That said, Grails will manage (as best it can) the schema for your database based upon your Domain classes. This is the default behavior when dbCreate is set to "update" in your DataSource.groovy file.
I would recommend reading through the great online documentation regarding database configuration and settings.
You also have more advanced tools available to you such as the database migration plugin should you need that level of control and flexibility.
In DataSource.groovy (under the conf dir) you find the definition of a H2 db. You could configure a mysql db, oracle, mongodb and so on database.
You also could specify which database use in dev, test and prod enviroments.
when you run your default rails app. The grails environment creates a in memory database for your app. It is created every time you restart your project.
In case you want to have your persistent database like mysql, mongodb etc.
What you need to do is (mysql for example)
Add a mysql dependency in BuildConfig.groovy like runtime 'mysql:mysql-connector-java:5.1.27'
Add Database and driver settings in DataSource.groovy. Now you can have different databases for different environments i.e. prod, test and dev modes. You can do this by having global setting for database or by defining settings for each mode separately.
In order to view your database from your running app you can use link http://localhost:8080/app/dbconsole just enter your database password and username. You will be able to do all your db related queries here. In case you are using grails default in memory database just use hit enter the default values that are there in DataSource.groovy for database
I am trying to update existing orbeon installation with nightly build and use the existing MYSQL database, but the new version is not letting me connect to mysql databse. It is defaulted to eXist database.
I set the property oxf.fr.persistence.service.mysql.datasource to mysql:
<property as="xs:anyURI" name="oxf.fr.persistence.service.mysql.datasource" value="mysql"/>
but no use.
can some one please help me with this connection issue.
The way you configure which persistence layer to use changed after the 3.9 release to give you more flexibility, specifically to allow you to have different apps stored in different schemas of the same database. So most likely, you'll just need to change your properties in properties-local.xml. For more about this change, see Form Runner persistence layer configuration.
How does ASP.NET MVC, if at all, deal with or provide ways to create your application using multiple environments? For example:
Development environment (local machine, probably run via the built-in web server and talking to a local database)
Testing (runs against a preloaded databse with example data, although this part could be skipped and mocks could be used)
Production database on a real server with real data
Ruby on Rails has the concept of environments and "automagically" can deduce if you're in development or production, so you can specify your connection information (connection string) in a config file and the framework dynamically pulls the appropriate one. Is there a similar way of doing things with .NET MVC? If not then how are professional developers using .NET MVC handling different environments?
The only way I can think of is to manually add an "environment" global method (or use an enum, or something like that, maybe this is a use for something like the State pattern?) and store the different connection strings in the web.config file, and then create a base class which all data access classes derive from which provides a way to obtain the connection string for the current environment; this would then have to be set to production when the time comes to put the application live.
Is there another way? Most of the .NET MVC videos and articles I've seen don't even bother with separate environments but only use a development database and don't indicate how you do it in production.
I'd say this is really a question of your company's internal processes. Since every company is a little bit different it's hard to have a "right" generic way to support dev/test/alpha/production and/or other environments.
One way: Create a setup program that supplies the correct connection string based on the environment chosen during the setup process.
Another way: System Admin edits web.config file to supply correct connection string during install.
Yet ANother Way: Connection strings are stored in the system registry.
Even Another Odd Way: You have all your connection strings for all environments in web.config, then a setting in appSettings the tells you which one to use.
Depending on the client, I've done all of these. There are more but these are the more popular.
(One client wanted to store the connecting string in the data base itself. Really.)
You can use alias for your database. You just point these aliases to different servers in the different environments. Stored in the registry under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSQLServer\Client\Connect if i remember right. Then you use the alias in the connectionstring.
In response to Jason's response:
We use Enterprise Library Environments to configure the different environment paramters and via msbuild invoke the Merge Configuration Tool that generates the different configs for each environment. The deploy process picks the right config file depending on which environment to install.
I was able to solve a similar situation following these steps:
In your Visual Studio, access Build > Configuration Manager
Click in "new"
Choose a name for your configuration, and then copy settings from an existing config. After the configuration creation, it will be available for you to target as build configuration
Create a Web.{env-name-you-chose}.config in your application folder, along with the original Web.config file.
Open your .csproj file with Visual Studio or any text editor
Search for a section that looks like the following and add the highlighted lines, with the config file name you gave previously:
Open your Visual Studio, reload projects if it's required, and now you are able to choose your configuration via CLI or manual publish using Visual Studio.
There is a Publishing Wizard (in Visual Studio) wich let's you change parts of web.config for release build automaticaly. Wich happens to be the feature you are asking about. No magic thou.
What we have done is during our automated build process (Hudson), we alter values in web.config depending on which environment the build is for. Unfortunately there isn't a magical way to do this.
For deployment, which I assume that is what the op was asking about, one creates multiple configurations and in the publish, picks a different configuration. These are called transforms and they operate on the web.config. One would have at least three publish profiles, one for dev, test and prod. One can change more than just the connection string in this way. One can turn on custom errors, turn off debugging and change values of configuration variables. I highly recommend it.
I have a similar question. I have a log table reader. I want it to read log tables in the development, test and production databases. The major difficulty lies in my user account doesn't have permission to look at test and production. It's some silly security thing. The user that I'm impersonating in the application does have permission. I'm struggling trying to tell MVC to build the test and production models using the impersonated user.