I want to user two data source in single environment.
dataSource {
dbCreate = "update" // one of 'create', 'create-drop','update'
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
username = "root"
password = ""
url = ""
dialect = "org.hibernate.dialect.MySQL5InnoDBDialect"
properties {
maxActive = 75
maxIdle = 10
minIdle = 2
initialSize = 2
maxWait = 30000
validationQuery="select 1"
testOnBorrow=true
testWhileIdle=true
timeBetweenEvictionRunsMillis=60000
}
}
Prior to Grails 2.0 multiple data source support is provided by the Datasources plugin.
Related
I have some tests written using Artillery + Playwright and I am using the publish-metrics plugin with type influxdb-statsd. I then have the following telegraf.config
[[outputs.influxdb_v2]]
urls = ["http://${INFLUX_DB2_HOST_ADDRESS}:8086"]
token = "${INFLUX_DB2_TOKEN}"
organization = "${INFLUX_DB2_ORGANIZATION}"
bucket = "${INFLUX_DB2_BUCKET}"
[[inputs.statsd]]
protocol = "udp"
max_tcp_connections = 250
tcp_keep_alive = false
service_address = ":8125"
delete_gauges = true
delete_counters = true
delete_sets = true
delete_timings = true
metric_separator = "_"
parse_data_dog_tags = true
datadog_extensions = true
datadog_distributions = false
Data from artillery is sent in this format to statsD
artillery.browser.page.FID.compliance-hub_dashboard.min:3.2|g
artillery.browser.page.FID.compliance-hub_dashboard.max:3.2|g
artillery.browser.page.FID.compliance-hub_dashboard.count:2|g
artillery.browser.page.FID.compliance-hub_dashboard.p50:3.2|g
I would like to set up a telegraf template so that in Influx DB
artillery.browser.page.FID.compliance-hub_dashboard is a measurement and min, max, count and p50 are fields.
How do I do that?
I tried:
templates = [
"measurement.measurement.measurement.measurement.measurement.field",
]
but it's not working. :(
What I see in InfluxDb is a measurement of artillery_browser_page_FID_compliance-hub_dashboard_min with a field of value = 3.2
I am following SE Thread to get some response to HTTP POST on an express node. But unable to get any response from kapacitor.
Environment
I am using Windows 10 via PowerShell.
I am connected to an InfluxDB internal Server which is mentioned in the kapacitor.conf and have a TICKscript to stream data via it.
kapacitor.conf
hostname = "134.102.97.81"
data_dir = "C:\\Users\\des\\.kapacitor"
skip-config-overrides = true
default-retention-policy = ""
[alert]
persist-topics = true
[http]
bind-address = ":9092"
auth-enabled = false
log-enabled = true
write-tracing = false
pprof-enabled = false
https-enabled = false
https-certificate = "/etc/ssl/kapacitor.pem"
https-private-key = ""
shutdown-timeout = "10s"
shared-secret = ""
[replay]
dir = "C:\\Users\\des\\.kapacitor\\replay"
[storage]
boltdb = "C:\\Users\\des\\.kapacitor\\kapacitor.db"
[task]
dir = "C:\\Users\\des\\.kapacitor\\tasks"
snapshot-interval = "1m0s"
[load]
enabled = false
dir = "C:\\Users\\des\\.kapacitor\\load"
[[influxdb]]
enabled = true
name = "DB5Server"
default = true
urls = ["https://influxdb.internal.server.address:8086"]
username = "user"
password = "password"
ssl-ca = ""
ssl-cert = ""
ssl-key = ""
insecure-skip-verify = true
timeout = "0s"
disable-subscriptions = true
subscription-protocol = "https"
subscription-mode = "cluster"
kapacitor-hostname = ""
http-port = 0
udp-bind = ""
udp-buffer = 1000
udp-read-buffer = 0
startup-timeout = "5m0s"
subscriptions-sync-interval = "1m0s"
[influxdb.excluded-subscriptions]
_kapacitor = ["autogen"]
[logging]
file = "STDERR"
level = "DEBUG"
[config-override]
enabled = true
[[httppost]]
endpoint = "kapacitor"
url = "http://localhost:1440"
headers = { Content-Type = "application/json;charset=UTF-8"}
alert-template = "{\"id\": {{.ID}}}"
The daemon runs without any problems.
test2.tick
dbrp "DBTEST"."autogen"
stream
|from()
.measurement('humid')
|alert()
.info(lambda: TRUE)
.post()
.endpoint('kapacitor')
Already defined the task .\kapacitor.exe define bc_1 -tick test2.tick
Enabled it .\kapacitor.exe enable bc_1
The status shows nothing:
.\kapacitor.exe show bc_1
ID: bc_1
Error:
Template:
Type: stream
Status: enabled
Executing: true
Created: 13 Mar 19 15:33 CET
Modified: 13 Mar 19 16:23 CET
LastEnabled: 13 Mar 19 16:23 CET
Databases Retention Policies: ["NIMBLE"."autogen"]
TICKscript:
dbrp "TESTDB"."autogen"
stream
|from()
.measurement('humid')
|alert()
.info(lambda: TRUE)
.post()
.endpoint('kapacitor')
DOT:
digraph bc_1 {
graph [throughput="0.00 points/s"];
stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stream0 -> from1 [processed="0"];
from1 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
from1 -> alert2 [processed="0"];
alert2 [alerts_inhibited="0" alerts_triggered="0" avg_exec_time_ns="0s" crits_triggered="0" errors="0" infos_triggered="0" oks_triggered="0" warns_triggered="0" working_cardinality="0" ];
}
The Daemon logs provide this for the task
ts=2019-03-13T16:25:23.640+01:00 lvl=debug msg="starting enabled task on startup" service=task_store task=bc_1
ts=2019-03-13T16:25:23.677+01:00 lvl=debug msg="starting task" service=kapacitor task_master=main task=bc_1
ts=2019-03-13T16:25:23.678+01:00 lvl=info msg="started task" service=kapacitor task_master=main task=bc_1
ts=2019-03-13T16:25:23.679+01:00 lvl=debug msg="listing dot" service=kapacitor task_master=main dot="digraph bc_1 {\nstream0 -> from1;\nfrom1 -> alert2;\n}"
ts=2019-03-13T16:25:23.679+01:00 lvl=debug msg="started task during startup" service=task_store task=bc_1
ts=2019-03-13T16:25:23.680+01:00 lvl=debug msg="opened service" source=srv service=*task_store.Service
ts=2019-03-13T16:25:23.680+01:00 lvl=debug msg="opening service" source=srv service=*replay.Service
ts=2019-03-13T16:25:23.681+01:00 lvl=debug msg="skipping recording, metadata is already correct" service=replay recording_id=353d8417-285d-4fd9-b32f-15a82600f804
ts=2019-03-13T16:25:23.682+01:00 lvl=debug msg="skipping recording, metadata is already correct" service=replay recording_id=a8bb5c69-9f20-4f4d-8f84-109170b6f583
But I get nothing on the Express Node side. The code is exactly the same as that in the above mentioned SE thread.
Any Help as to how to capture stream from Kapacitor on HTTP Post? I already have a live system that is pushing information into the dedicated database already
I was able to shift focus from stream to batch in the above query. I have documented the complete process on medium.com.
Some Files:
kapacitor.gen.conf
hostname = "my-windows-10"
data_dir = "C:\\Users\\<user>\\.kapacitor"
skip-config-overrides = true
default-retention-policy = ""
[alert]
persist-topics = true
[http]
bind-address = ":9092"
auth-enabled = false
log-enabled = true
write-tracing = false
pprof-enabled = false
https-enabled = false
https-certificate = "/etc/ssl/kapacitor.pem"
https-private-key = ""
shutdown-timeout = "10s"
shared-secret = ""
[replay]
dir = "C:\\Users\\des\\.kapacitor\\replay"
[storage]
boltdb = "C:\\Users\\des\\.kapacitor\\kapacitor.db"
[task]
dir = "C:\\Users\\des\\.kapacitor\\tasks"
snapshot-interval = "1m0s"
[load]
enabled = false
dir = "C:\\Users\\des\\.kapacitor\\load"
[[influxdb]]
enabled = true
name = "default"
default = true
urls = ["http://127.0.0.1:8086"]
username = ""
password = ""
ssl-ca = ""
ssl-cert = ""
ssl-key = ""
insecure-skip-verify = true
timeout = "0s"
disable-subscriptions = true
subscription-protocol = "http"
subscription-mode = "cluster"
kapacitor-hostname = ""
http-port = 0
udp-bind = ""
udp-buffer = 1000
udp-read-buffer = 0
startup-timeout = "5m0s"
subscriptions-sync-interval = "1m0s"
[influxdb.excluded-subscriptions]
_kapacitor = ["autogen"]
[logging]
file = "STDERR"
level = "DEBUG"
[config-override]
enabled = true
# Subsequent Section describes what this conf does
[[httppost]]
endpoint = "kap"
url = "http://127.0.0.1:30001/kapacitor"
headers = { "Content-Type" = "application/json"}
TICKScript
var data = batch
| query('SELECT "v" FROM "telegraf_test"."autogen"."humid"')
.period(5s)
.every(10s)
data
|httpPost()
.endpoint('kap')
Define the Task
.\kapacitor.exe define batch_test -tick .\batch_test.tick -dbrp DBTEST.autogen
I suspect the hostname was michieveous where it was set to localhost previously but I set it my machine's hostname and instead used the IP address 127.0.0.1 whereever localhost was mentioned
I have below datasource config in grails.
dataSource_staging_oracle {
dbCreate = "none"
pooled = true
url = "jdbc:oracle:thin:#//my-box-oracle.com:1521/DB1"
driverClassName = "oracle.jdbc.OracleDriver"
username = "USER_1"
password = "encryptedpassword"
passwordEncryptionCodec = PropertiesCodec
}
dataSource_prod_oracle {
dbCreate = "none"
pooled = true
autoReconnect = true
url = "jdbc:oracle:thin:#//my-oracle-prod-box.com:1521/DB2"
driverClassName = "oracle.jdbc.OracleDriver"
username = "user_2"
password = "encrypted_password"
passwordEncryptionCodec = PropertiesCodec
}
Application is able to fetch the DB connection and work fine. However after 1-2 hours of processing, I see connection closed exceptions.
ERROR util.JDBCExceptionReporter - Closed Connection
I believe this has to do with datasource config properties. What grails settings help me in refreshing the connections? I already have set autoReconnect and pooled to true
I use grails 2.3.3
Datasource properties will inform the database what the middleware server will expect from the Database. If your Oracle server does not agree with the settings Grails would like to use, expect sometimes cryptic error messages.
Using pooled connections tells Oracle to hold a number of connections open for the application. If the application is running 1-2 hours and continuously processing, I expect the pool may be running out of valid connections. For some background please view the links included below.
Check out the Grails configuration docs, section 4.4, to see advanced datasource configuration options. Do you have a validation query defined? (eg select 1 from dual;)
Check out this AskTom post for a clear explanation of connection pooling.
Previous StackOverflow question with MySQL but similar error.
Below config helped me fix my problem.
dataSource_staging {
dbCreate = "none"
pooled = true
autoReconnect = true
url = "jdbc:oracle:thin:#//my-box-oracle.com:1521/DB1"
driverClassName = "oracle.jdbc.OracleDriver"
username = "USER_1"
password = "encryptedpassword"
passwordEncryptionCodec = PropertiesCodec
properties {
validationQuery = "SELECT 1 FROM DUAL"
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
maxAge = 10 * 60000
maxWait = 10000
maxIdle = 25
maxActive = 50
initialSize = 5
}
}
I seem to randomly get the warning below in my Grails 2.2.4 application. It doesn't look like it is causing any issues, but it is still concerning.
I tried to prevent this warning by modifying my datasource properties in my DataSource.groovy file:
dataSource {
pooled = true
properties {
maxWait = 10000 // 10 seconds
minEvictableIdleTimeMillis = 1000 * 60 * 30 // 30 minutes
numTestsPerEvictionRun = 3
testOnBorrow = true
testOnReturn = false
testWhileIdle = false
timeBetweenEvictionRunsMillis = 1000 * 60 * 30 // 30 minutes
validationQuery = "SELECT 1"
}
}
And when that didn't work I tried to set the properties in my BootStrap.groovy file:
def init = { servletContext ->
def ctx = Holders.getApplicationContext()
def dataSource = ctx.dataSourceUnproxied
println "configuring database connection pool"
dataSource.setMinEvictableIdleTimeMillis(1000 * 60 * 30)
dataSource.setTimeBetweenEvictionRunsMillis(1000 * 60 * 30)
dataSource.setNumTestsPerEvictionRun(3)
dataSource.setTestOnBorrow(true)
dataSource.setTestWhileIdle(false)
dataSource.setTestOnReturn(false)
dataSource.setValidationQuery("SELECT 1")
}
Neither attempt prevents the warning. The author of this post said he had success setting the properties directly in the tomcat config, but I need a more generic solution that will work from the command line and in other servers.
2013-09-25 15:07:51,027 [http-bio-8080-exec-9] WARN jdbc.AbstractBatcher - exception clearing maxRows/queryTimeout
java.sql.SQLException: org.apache.commons.dbcp.DelegatingPreparedStatement with address: "com.mysql.jdbc.JDBC4PreparedStatement#13ed0db0: EXCEPTION: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after statement closed." is closed.
at org.apache.commons.dbcp.DelegatingStatement.checkOpen(DelegatingStatement.java:137)
at org.apache.commons.dbcp.DelegatingStatement.getMaxRows(DelegatingStatement.java:237)
at ace_2.DefsUploadController.upload(DefsUploadController.groovy:16)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
After connections to my MySQL database timed out I would see that exact exception as well. I updated the DataSource configuration using the suggestion in the Grails docs as a guideline and I have yet to see any exceptions from closed connections.
Here are my current settings:
properties {
initialSize=5
maxActive=50
minIdle=5
maxIdle=25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}
I could not find how connection pooling is activated in Grails.
In my DataSource.groovy I have the following:
url = "jdbc:mysql://localhost/myapp?useUnicode=yes&characterEncoding=UTF-8"
username = "root"
password = ""
dialect = org.hibernate.dialect.MySQL5InnoDBDialect
pooled = true
properties {
maxActive = -1
minEvictableIdleTimeMillis=1800000
timeBetweenEvictionRunsMillis=1800000
numTestsPerEvictionRun=3
testOnBorrow=true
testWhileIdle=true
testOnReturn=true
validationQuery = "select 1"
}
Is there something else I have to do to activate connection pooling than setting pooled = true?
No, the pooled attribute configures pooling. If it's set to false then you will create a new connection each time, and this can take hundreds of milliseconds, so it's not advised.
You would set pooled = false when using JNDI however, since the JNDI datasource does its own pooling, so there's no need to pool twice.
Below is a sample from Grails DataSource documentation page:
dataSource {
pooled = true
dbCreate = "update"
url = "jdbc:mysql://localhost:3306/my_database"
driverClassName = "com.mysql.jdbc.Driver"
dialect = org.hibernate.dialect.MySQL5InnoDBDialect
username = "username"
password = "password"
properties {
jmxEnabled = true
initialSize = 5
maxActive = 50
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}}
In addition to setting pooled to true, you can also tweak settings on how many connections are initialized up front (initialSize) and the maximum number of connections you would want to maintain when the load gets higher (maxActive).