I'm comparing two databases using liquibase integrated with ant. But the output it is generating is like generic format. It is not giving sql statements. Please can anyone tell me how compare two databases using liquibase integrated with ant or command line utility.
Obtaining the SQL statements, representing the diff between two databases, is a two step operation:
Generate the XML "diff" changelog
Generate SQL statements
Example
This example requires a liquibase.properties file (simplifies the command-line parameters):
classpath=/path/to/jdbc/jdbc.jar
driver=org.Driver
url=jdbc:db_url1
username=user1
password=pass1
referenceUrl=jdbc:db_url2
referenceUsername=user2
referencePassword=pass2
changeLogFile=diff.xml
Now run the following commands to create the SQL statements:
liquibase diffChangeLog
liquibase updateSQL > update.sql
A nice feature of liquibase is that it can also generate the rollback SQL:
liquibase futureRollbackSQL > rollback.sql
Update
Liquibase does not generate a data diff between databases, only the schema. However, it is possible to dump database data as a series of changesets:
liquibase --changeLogFile=data.xml --diffTypes=data generateChangeLog
One can use the data.xml file to migrate data contained in new tables.
Update 2:
Also possible to generate liquibase changesets using groovy.
import groovy.sql.Sql
import groovy.xml.MarkupBuilder
//
// DB connection
//
this.class.classLoader.rootLoader.addURL(new URL("file:///home/path/to/h2-1.3.162.jar"))
def sql = Sql.newInstance("jdbc:h2:db/db1","user","pass","org.h2.Driver")
//
// Generate liquibase changeset
//
def author = "generated"
def id = 1
new File("extract.xml").withWriter { writer ->
def xml = new MarkupBuilder(writer);
xml.databaseChangeLog(
"xmlns":"http://www.liquibase.org/xml/ns/dbchangelog",
"xmlns:xsi":"http://www.w3.org/2001/XMLSchema-instance",
"xsi:schemaLocation":"http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-2.0.xsd"
) {
changeSet(author:author, id:id++) {
sql.eachRow("select * from employee") { row ->
insert(tableName:"exmployee") {
column(name:"empno", valueNumeric:row.empno)
column(name:"name", value:row.name)
column(name:"job", value:row.job)
column(name:"hiredate", value:row.hiredate)
column(name:"salary", valueNumeric:row.salary)
}
}
}
}
}
Related
We are using a Python client binding for ZetaSQL GRPC local service in our application to analyze statements and extract referenced tables and output columns.
It is possible to extract referenced tables using the following simplified Python code and the local service:
import zetasql.local_service as zql
conn = zql.connect()
language_options = conn.GetLanguageOptions(
zql.pb2.LanguageOptionsRequest(maximum_features=True)
)
# Used to allow ZetaSQL parser to parse `CREATE TABLE AS` statments
language_options.supported_statement_kinds.pop()
req = zql.pb2.ExtractTableNamesFromStatementRequest(
sql_statement=sql, options=language_options
)
res = conn.ExtractTableNamesFromStatement(req)
return json.loads(MessageToJson(res))
However, from what I see here, the local service doesn't have the full functionalities of the Java client, mainly creating simple catalog with tables and columns to analyze any SQL statement. Also setting analyzer options doesn't seem to be possible.
Is it possible to analyze SQL statements using ZetaSQL with only the local service? If not, what should be the alternative approach to extract output columns?
I'm using apache beam to create a pipeline where basically reads an InputFile, Convert to Avro, write the AvroFile to a bucket and then Import these avro files to Spanner using Dataflow template
The problem that I'm facing is that the last step (Import the Avro files to the Database) is starting before the previous (write Avro Files to the bucket) is done.
I tried to add the Wait.on function but that only works if returns a PCollection, but when I write the files to avro it returns PDone.
Example of the Code:
// Step 1: Read Files
PCollection<String> lines = pipeline.apply("Reading Input Data exported from Cassandra",TextIO.read().from(options.getInputFile()));
// Step 2: Convert to Avro
lines .apply("Write Item Avro File",AvroIO.writeGenericRecords(spannerItemAvroSchema).to(options.getOutput()).withSuffix(".avro"));
// Step 3: Import to the DataBase
pipeline.apply( new ImportTransform(
spannerConfig,
options.getInputDir(),
options.getWaitForIndexes(),
options.getWaitForForeignKeys(),
options.getEarlyIndexCreateFlag()));
Again, the problem is because step 3 starts before Step 2 is done
any ideas?
This is a flaw in the API, see, e.g. a recent discussion on this on the beam dev list. The only solutions for now are to either fork AvroIO to return a PCollection or run two pipelines sequentially.
I have Jenkins with several hundreds of jobs, and I need to find job which runs specified gradle task. I see following solution
1.retrieve all jobs (Jenkins.instance.projects)
2.iterate over them
3.get xml config and verify presence of substring
The question is how to retrieve xml representation from hudson.model.FreeStyleProject? Or may be this data stored as map, than the question will be how it is called and how get it?
I did something similar before but using a shell script. Not sure if that approach is useful for you but just in case:
cd /var/lib/jenkins/jobs/
grep WORD */config.xml
Also the next groovy script that can list all the FreeStyleProject name and gradle tasks:
def builderFilter = { builder -> builder.class == hudson.plugins.gradle.Gradle.class }
jenkins.model.Jenkins.instance.getAllItems(hudson.model.FreeStyleProject.class).each{ job ->
job.getBuilders().findAll(builderFilter).each{ gradleStep ->
gradleStep.each { gradleItem ->
println(job.getDisplayName() + ' ' + gradleItem.getTasks())
}
}
}
After many tries I have concluded that the optimal way to transfer with SSIS data from AS400 (non-unicode) to SQL Server is:
Use native transfer utility to dump data to tsv (tab delimited)
Convert files from utf-8 to unicode
Use bulk insert to put them into SQL Server
In #2 step I have found a ready made code that does this:
string from = #"\\appsrv02\c$\bg_f0101.tsv";
string to = #"\\appsrv02\c$\bg_f0101.txt";
using (StreamReader reader = new StreamReader(from, Encoding.UTF8, false, 1000000))
using (StreamWriter writer = new StreamWriter(to, false, Encoding.Unicode, 1000000))
{
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
if (line.Length > 0)
writer.WriteLine(line);
}
}
I need to fully understand what is happening here with the encoding and why this is necessary.
How can I replace this script task with a more elegant solution?
I don't have much insight into exactly why you need the utf-8 conversion task, except to say that SQL server - I believe - uses UCS-2 as its native storage format, and this is similiar to UTF-16 which is what your task converts the file to. I'm surprised SSIS can't work with a UTF-8 input source though.
My main point is to answer the "How could I replace this script task with a more elegant solution?":
I have had a lot of success using HiT OLEDB/400 Server. It allows you to set up your AS/400 / iSeries / System i / whatever IBM are calling it this week as a linked server in SQL server, and you can then access the 400's data directly from the server its linked to using the standard 4 part SQL syntax, e.g. SELECT * FROM my400.my400.myLib.myFile.
Or even better, it's much more efficient as a passthrough query using EXEC...AT.
Using this you would not need SSIS at all, you'd just need a simple stored proc with that does an insert into your destination table direct from the 400 data.
I am using the quartz plugin with Grails 1.3.7. I have a need to load balance/cluster a server app that uses quartz jobs. Apparently this is supported but I am finding that all the google search results and links within documents are broken. I've found some raw Java examples but I would assume Grails has a more grailsy way to do this. All I need is a simple example to use as a template. I understand I need to somehow enable quartz to use JDBC to store the jobs and manage locking.
I think a link to a single sample would do it. But literally every time I've found something that looks promising it points to a broken link on terracotta's site. Pretty much every site eventually leads me here: http://www.opensymphony.com/quartz/wikidocs/TutorialLesson9.html but when I look on terracotta's site I see Java stuff but no grails. If Java is the only way to do this then so be it, but I feel like there has to be some grails expertise on this out there somewhere!
TIA.
To cluster the Quartz plugin in Grails, there are some files you need to include in your project. First, install the grails-app/conf/QuartzConfig.groovy and make sure jdbcStore is enabled.
quartz {
autoStartup = true
jdbcStore = true
waitForJobsToCompleteOnShutdown = true
}
Next, install the Hibernate configuration files relevant to the database to which you will be connecting. For example, with Oracle, the base Hibernate xml config at grails-app/conf/hibernate/hibernate.cfg.xml is:
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
'-//Hibernate/Hibernate Configuration DTD 3.0//EN'
'http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd'>
<hibernate-configuration>
<session-factory>
<mapping resource="Quartz.oracle.hbm.xml"/>
</session-factory>
</hibernate-configuration>
The actual Quartz-Hibernate SQL file for this example will be named Quartz.oracle.hbm.xml and will reside in the same directory. These files should be available at the Quartz plugin on GitHub (https://github.com/nebolsin/grails-quartz), under src/templates/sql. Note, that these scripts only seem to work for DataSource create and create-drop, so you'll need to manually create the Quartz tables on an update, if they don't already exist from a previous run.
Create a grails-app/conf/quartz/quartz.properties file, and edit is to fit your business needs:
/* Have the scheduler id automatically generated for
* all schedulers in a cluster */
org.quartz.scheduler.instanceId = AUTO
/* Don't let Quartz "Phone Home" to see if new versions
* are available */
org.quartz.scheduler.skipUpdateCheck = true
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
/* Configure Quartz for only one thread as the only job
* should run once per day */
org.quartz.threadPool.threadCount = 4
/* Give the thread a Thread.MIN_PRIORITY level*/
org.quartz.threadPool.threadPriority = 1
/* Allow a minute (60,000 ms) of non-firing to pass before
* a trigger is called a misfire */
org.quartz.jobStore.misfireThreshold = 60000
/* Handle only 2 misfired triggers at a time */
org.quartz.jobStore.maxMisfiresToHandleAtATime = 2
/* Check in with the cluster every 5000 ms*/
org.quartz.jobStore.clusterCheckinInterval = 5000
/* Use the Oracle Quartz Driver to communicate via JDBC */
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
/* Have Quartz handle its own transactions with the database */
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
/* Define the prefix for the Quartz tables in the database*/
org.quartz.jobStore.tablePrefix = QRTZ_
/* Tell Quartz it is clustered */
org.quartz.jobStore.isClustered = true
/* Tell Quartz that properties passed to the job call are
* NOT all String objects */
org.quartz.jobStore.useProperties = false
/* Detect the jvm shutdown and call shutdown on the scheduler */
org.quartz.plugin.shutdownhook.class = org.quartz.plugins.management.ShutdownHookPlugin
org.quartz.plugin.shutdownhook.cleanShutdown = true
/* Log the history of triggers and jobs */
org.quartz.plugin.triggerHistory.class = org.quartz.plugins.history.LoggingTriggerHistoryPlugin
org.quartz.plugin.jobHistory.class = org.quartz.plugins.history.LoggingJobHistoryPlugin
Note from the above properties, you can set org.quartz.plugins in the Log4j setup of Config.groovy to log relevant job and trigger triggering information. I think info level should suffice.
Edit, or create, scripts/_Events.groovy and add the following war modification closure. This fixes a known Quartz plugin bug to install the correct quartz.properties, instead of a blank one from the plugin, in to the final war file.
eventCreateWarStart = { warName, stagingDir ->
// Make sure we have the correct quartz.properties in the
// correct place in the war to enable clustering
ant.delete(dir:"${stagingDir}/WEB-INF/classes/quartz")
ant.copy(file:"${basedir}/grails-app/conf/quartz/quartz.properties",
todir:"${stagingDir}/WEB-INF/classes")
}
And you should be done...
P.S. If you are using an Oracle database, add the following to BuildConfig.groovy in the dependencies block, so that you have access to the Quartz-Oracle communication drivers:
runtime("org.quartz-scheduler:quartz-oracle:1.7.2") {
// Exclude quartz as 1.7.3 is included from the plugin
excludes('quartz')
}
P.P.S The sql files at the link above are just the SQL. To make it in to a hibernate file, just surround each individual SQL command with a Hibernate database-object node, like so (again w/ Oracle example):
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE hibernate-mapping PUBLIC
'-//Hibernate/Hibernate Mapping DTD 3.0//EN'
'http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd'>
<hibernate-mapping>
<database-object>
<create>
CREATE TABLE QRTZ_JOB_DETAILS (
JOB_NAME VARCHAR2(200) NOT NULL,
JOB_GROUP VARCHAR2(200) NOT NULL,
DESCRIPTION VARCHAR2(250) NULL,
JOB_CLASS_NAME VARCHAR2(250) NOT NULL,
IS_DURABLE VARCHAR2(1) NOT NULL,
IS_VOLATILE VARCHAR2(1) NOT NULL,
IS_STATEFUL VARCHAR2(1) NOT NULL,
REQUESTS_RECOVERY VARCHAR2(1) NOT NULL,
JOB_DATA BLOB NULL,
PRIMARY KEY (JOB_NAME,JOB_GROUP)
)
</create>
<drop>DROP TABLE QRTZ_JOB_DETAILS</drop>
<dialect-scope name='org.hibernate.SomeOracleDialect' />
</database-object>
...
<database-object>
<create>INSERT INTO QRTZ_LOCKS VALUES('TRIGGER_ACCESS')</create>
<drop></drop>
<dialect-scope name='org.hibernate.SomeOracleDialect' />
</database-object>
...
</hibernate-mapping>
The dialect-scope tells Hibernate with which Database dialects the create and drop nodes should be used. You can try leaving it out and see if it works, otherwise you may have to add the MySql dialect used by your Grails DataSource.