Log management on Docker - docker

I want to send STDOUT log from Docker container to fluentd.
But, when one container outputs access logs and error logs, logs are mixed.
example
# rails access log
2017-04-07 12:10:01 +0000 6a51e389e724: {"log":"I, [2017-04-07T12:10:01.825923 #7] INFO -- : Started GET \"/users/new\" for 172.21.0.1 at 2017-04-07 12:10:01 +0000","container_id":"6a51e389e724c67be4e714402b69da192db4a304cbfdf638594de6cff9774c23","container_name":"/app","source":"stdout"}
# rails error log
2017-04-07 12:10:01 +0000 6a51e389e724: {"container_id":"6a51e389e724c67be4e714402b69da192db4a304cbfdf638594de6cff9774c23","container_name":"/app","source":"stdout","log":"E, [2017-04-07T12:10:01.830039 #7] ERROR -- : Invoke logger error"}
# rails access log
2017-04-07 12:10:03 +0000 6a51e389e724: {"log":"I, [2017-04-07T12:10:01.825923 #7] INFO -- : Started POST \"/users/create\" for 172.21.0.1 at 2017-04-07 12:10:01 +0000","container_id":"6a51e389e724c67be4e714402b69da192db4a304cbfdf638594de6cff9774c23","container_name":"/app","source":"stdout"}
Can I add a label for each log type?
Please tell me if you have a good solution.
Thanks you and best regards.

As your app's access and error logs are sent to STDOUT of container, then you have no way to separate them by logging driver. As solution you can send access log to STDOUT, error log to STDERR of container, then later differentiate logs by "source" field of each json message (this can be done if you connect fluentd to elasticsearch+kibana).

Related

Gerrit plugin-manager doesn't show plugins

I have a Gerrit v2.15 on a linux machine and I'm the administrator in Gerrit.
I installed the Gerrit plugin-manager under gerrit/plugins with command :
wget https://gerrit-ci.gerritforge.com/view/Plugins-stable-2.15/job/plugin-plugin-manager-bazel-master-stable-2.15/lastSuccessfulBuild/artifact/bazel-genfiles/plugins/plugin-manager/plugin-manager.jar
But nothing uploads in Gerrit UI in tab Plugins. I only see the Installed, as before.
In gerrit/logs/error.log I can read out the following:
[2018-06-15 08:54:26,366] [main] INFO com.google.gerrit.server.plugins.PluginLoader : Loaded plugin plugin-manager, version v2.15-1-g27c3d2935d
[2018-06-15 08:54:26,367] [plugin-manager-preloader] INFO com.googlesource.gerrit.plugins.manager.OnStartStop : Start-up: pre-loading list of plugins from registry
[2018-06-15 08:54:26,902] [main] INFO com.google.gerrit.server.git.GarbageCollectionRunner : Ignoring missing gc schedule configuration
[2018-06-15 08:54:26,902] [main] INFO com.google.gerrit.server.config.ScheduleConfig : accountDeactivation schedule parameter "accountDeactivation.interval" is not configured
[2018-06-15 08:54:26,902] [main] INFO com.google.gerrit.server.change.ChangeCleanupRunner : Ignoring missing changeCleanup schedule configuration
[2018-06-15 08:54:26,946] [main] INFO com.google.gerrit.sshd.SshDaemon : Started Gerrit SSHD-CORE-1.6.0 on *:29418
[2018-06-15 08:54:26,949] [main] INFO org.eclipse.jetty.server.Server : jetty-9.3.18.v20170406
[2018-06-15 08:54:28,040] [main] INFO com.googlesource.gerrit.plugins.gitiles.HttpModule : No /opt/gerrit/etc/gitiles.config; assuming defaults
[2018-06-15 08:54:28,847] [main] INFO org.eclipse.jetty.server.handler.ContextHandler : Started o.e.j.s.ServletContextHandler#62672ad5{/,null,AVAILABLE}
[2018-06-15 08:54:28,869] [main] INFO org.eclipse.jetty.server.AbstractConnector : Started ServerConnector#38503309{HTTP/1.1,[http/1.1]}{127.0.0.1:8081}
[2018-06-15 08:54:28,890] [main] INFO org.eclipse.jetty.server.Server : Started #20170ms
[2018-06-15 08:54:28,893] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.15.1 ready
[2018-06-15 08:54:44,819] [plugin-manager-preloader] INFO com.googlesource.gerrit.plugins.manager.OnStartStop : 45 plugins successfully pre-loaded
What have i missed?
Is it some permission settings that is wrong or configuration in gerrit.config that has to be done?
Directory gerrit/plugin has permissions set to: drwxr-xr-x
Note : I've tried restarting Gerrit.
Good to know: We are using nginx. That might be useful info.
I know very little about how nginx works.
What it looks like in the UI
The plugin-manager only works in the Old UI. If you're using the new PolyGerrit UI first click on "Switch to Old UI" link at the right/down on the page. Then the plugin-manager can be access at: Plugins > Manage menu.
If you're not able to see the "Manage" option in the Old UI check the following:
. Verify if you're logged in Gerrit with an Admin user (you need to have "Administrate Server" permission in All-Projects project). There're an specific permission to access Plugins > Installed item (called "View Plugins") but this permission doesn't work the same way to Plugins > Manage one.
. Verify if you have the following set in the GERRIT_SITE/etc/gerrit.config file:
[plugins]
allowRemoteAdmin = true
Maybe this can resolve your issue.

Apache nifi is not starting up

I am trying to start Apache nifi version 1.2.0 on window 8 machine. It used to start properly. After I restarted the system the nifi is not starting at all. I had check status Its keep getting "Apacha Nifi not running".
Below are logs from nifi.bootstrap.log file:-
2017-07-05 15:41:57,105 WARN [NiFi Bootstrap Command Listener]
org.apache.nifi.bootstrap.RunNiFi Failed to set permissions so that only the
owner can read pid file E:\softwares\nifi-1.2.0\bin\..\run\nifi.pid; this
may allows others to have access to the key needed to communicate with NiFi.
Permissions should be changed so that only the owner can read this file
2017-07-05 15:41:57,142 WARN [NiFi Bootstrap Command Listener]
org.apache.nifi.bootstrap.RunNiFi Failed to set permissions so that only the
owner can read status file E:\softwares\nifi-1.2.0\bin\..\run\nifi.status;
this may allows others to have access to the key needed to communicate with
NiFi. Permissions should be changed so that only the owner can read this
file
2017-07-05 15:41:57,168 INFO [NiFi Bootstrap Command Listener]
org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for
Bootstrap requests on port 50765
2017-07-05 15:43:12,077 ERROR [NiFi logging handler] org.apache.nifi.StdErr
Failed to start web server: Unable to start Flow Controller.
2017-07-05 15:43:12,078 ERROR [NiFi logging handler] org.apache.nifi.StdErr
Shutting down...
2017-07-05 15:43:14,501 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi
never started. Will not restart NiFi
Stack trace from nifi.app.log: -
2017-07-05 15:43:12,077 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:452)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:419)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:695)
at org.apache.nifi.NiFi.<init>(NiFi.java:160)
at org.apache.nifi.NiFi.main(NiFi.java:267)
Caused by: java.io.IOException: Expected to read a Sentinel Byte of '1' but got a value of '0' instead
at org.apache.nifi.repository.schema.SchemaRecordReader.readRecord(SchemaRecordReader.java:65)
at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeRecord(SchemaRepositoryRecordSerde.java:115)
at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:109)
at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:46)
at org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1096)
at org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459)
at org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:381)
at org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:712)
at org.apache.nifi.controller.StandardFlowService.initializeController(StandardFlowService.java:953)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:534)
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
... 28 common frames omitted
Thanks in advance
After Googling on this error "Caused by: java.io.IOException: Expected to read a Sentinel Byte of '1' but got a value of '0' instead" I found that this error indicates a partial write to the repos.
Here are a couple of things you can check/try to bring your Dataflow back online ;
check if your dsks are not full
Did you launch nifi with the same user ? Did you run it with administrator privileges ?
You can backup/move your repositories and try to start Nifi with empty repositories, you will still have your dataflows there but any file that was processing when you shutdown will be gone.
Could you please try that ?
I think the issue is with incompatible java version, use JAVA 8 version.
If you haven't set JAVA_HOME then set in environment variables with path Like "C:/program files/jdk1.8"
Jira addressing when NiFi run with java 9 version and the issue not resolved yet
https://issues.apache.org/jira/browse/NIFI-4419

Graphaware Framework and UUID not starting on Neo4j GrapheneDB

I am trying to get the Graphaware Framework and UUID running on a GrapheneDB instance. I have followed the instructions to zip the JAR and neo4j.properties files and uploaded using the GrapheneDB Web Interface but UUID's are not added when I create a new node.
neo4j.properties file
dbms.unmanaged_extension_classes=com.graphaware.server=/graphaware
com.graphaware.runtime.enabled=true
#UIDM becomes the module ID:
com.graphaware.module.UIDM.1=com.graphaware.module.uuid.UuidBootstrapper
#optional, default is uuid:
com.graphaware.module.UIDM.uuidProperty=uuid
#optional, default is false:
com.graphaware.module.UIDM.stripHyphens=true
#optional, default is all nodes:
#com.graphaware.module.UIDM.node=hasLabel('Label1') || hasLabel('Label2')
#optional, default is no relationships:
#com.graphaware.module.UIDM.relationship=isType('Type1')
com.graphaware.module.UIDM.relationship=com.graphaware.runtime.policy.all.IncludeAllBusinessRelationships
#optional, default is uuidIndex
com.graphaware.module.UIDM.uuidIndex=uuidIndex
#optional, default is uuidRelIndex
com.graphaware.module.UIDM.uuidRelationshipIndex=uuidRelIndex
Log Output
2017-03-02 10:20:40.184+0000 INFO Neo4j Server shutdown initiated by
request 2017-03-02 10:20:40.209+0000 INFO
[c.g.s.f.b.GraphAwareServerBootstrapper] stopped 2017-03-02
10:20:40.209+0000 INFO Stopping... 2017-03-02 10:20:40.982+0000 INFO
Stopped. 2017-03-02 10:20:43.402+0000 INFO Starting... 2017-03-02
10:20:43.820+0000 INFO Bolt enabled on 0.0.0.0:7475. 2017-03-02
10:20:45.153+0000 INFO [c.g.r.b.RuntimeKernelExtension] GraphAware
Runtime disabled. 2017-03-02 10:20:48.130+0000 INFO Started.
2017-03-02 10:20:48.343+0000 INFO
[c.g.s.f.b.GraphAwareServerBootstrapper] started 2017-03-02
10:20:48.350+0000 INFO Mounted unmanaged extension
[com.graphaware.server] at [/graphaware] 2017-03-02 10:20:48.724+0000
INFO Mounting GraphAware Framework at /graphaware 2017-03-02
10:20:48.755+0000 INFO Will try to scan the following packages:
{com..graphaware.,org..graphaware.,net..graphaware.}
2017-03-02 10:20:52.633+0000 INFO Remote interface available at
http://localhost:7474/
Messages.log Extract
2017-03-02 10:33:59.991+0000 INFO [o.n.k.i.DiagnosticsManager] ---
STARTED diagnostics for KernelDiagnostics:StoreFiles END ---
2017-03-02 10:34:01.846+0000 INFO [o.n.k.i.DiagnosticsManager] ---
SERVER STARTED START --- 2017-03-02 10:34:02.526+0000 INFO
[c.g.s.f.b.GraphAwareBootstrappingFilter] Mounting GraphAware
Framework at /graphaware 2017-03-02 10:34:02.547+0000 INFO
[c.g.s.f.c.GraphAwareWebContextCreator] Will try to scan the following
packages:
{com..graphaware.,org..graphaware.,net..graphaware.}
2017-03-02 10:34:06.100+0000 INFO [o.n.k.i.DiagnosticsManager] ---
SERVER STARTED END ---
It looks like the framework is not started but I have set enabled=true in the properties file.
Environment Setup
Neo4j Community Edition 3.1.1
graphaware-server-3.1.0.44
graphaware-uuid-3.1.0.44.13
Thanks

Rails + CarrierWave + RMagick not logging errors..?

Using carrier wave to upload a pdf document via CarrierWave. Then using RMagick to save the first page of the pdf as a thumbnail.jpg. I have this working correctly in my development environment (Mac OS X 10.9), however its failing in production (Nginx + Unicorn on Ubuntu 12.04 LTS).
It appears that the upload completes successfully, based on watching the upload percentage in the browser status bar. However, something errors in the processing, and I get Rails error page. The problem is, there is nothing in my production.log.
I get the following in unicorn.log:
I, [2013-12-12T01:22:55.232431 #2935] INFO -- : worker=1 ready
I, [2013-12-12T01:22:55.246992 #2932] INFO -- : worker=0 ready
E, [2013-12-12T01:26:00.666715 #2929] ERROR -- : worker=1 PID:2935 timeout (31s > 30s), killing
E, [2013-12-12T01:26:00.676250 #2929] ERROR -- : reaped #<Process::Status: pid 2935 SIGKILL (signal 9)> worker=1
I, [2013-12-12T01:26:00.676382 #2929] INFO -- : worker=1 spawning...
I, [2013-12-12T01:26:00.681087 #3502] INFO -- : worker=1 spawned pid=3502
I, [2013-12-12T01:26:00.681399 #3502] INFO -- : Refreshing Gem list
SECURITY WARNING: No secret option provided to Rack::Session::Cookie.
This poses a security threat. It is strongly recommended that you
provide a secret to prevent exploits that may be possible from crafted
cookies. This will not be supported in future versions of Rack, and
future versions will even invalidate your existing user cookies.
Called from: /home/deployer/apps/______/shared/bundle/ruby/1.9.1/gems/actionpack-3.2.9/lib/action_dispatch/middleware/session/abstract_store.rb:28:in `initialize'.
And this in my nginx error.log
2013/12/12 01:26:00 [error] 2597#0: *9 upstream prematurely closed connection while reading response header from upstream, client: ###.###.###.###, server: ___.org, request: "POST /admin/explores HTTP/1.1", upstream: "http://unix:/tmp/unicorn.sock:/admin/explores", host: "___.org", referrer: "https://___.org/admin/explores/new"
Not sure how to get a better error description of what's going on. Would appreciate any help.
Thanks!
Ruby 1.9.3 + Rails 3.2.9
The line E, [2013-12-12T01:26:00.666715 #2929] ERROR -- : worker=1 PID:2935 timeout (31s > 30s), killing from your unicorn.log tells you the problem: you have a timeout. Increase unicorn's timeout to 60 seconds by setting timeout 60 in your unicorn.rb. More information: http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-timeout

Apache with passenger gives error(Premature end of script headers) in development mode

I'm using Apache + passenger for my rails application, The application is running fine in production mode except when the browser displays error message after every 2-3 clicks in development mode which goes off when i refresh the page.I'm using metal in my rails application which i see could be the reason but not sure if it is the only reason.
Error message on browser:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, [no address given] and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Apache/2.2.16 (Ubuntu) Server at localhost Port 80
Apache log Error:
[Thu Dec 01 16:55:23 2011] [error] [client 127.0.0.1] Premature end of script headers: view_transactions, referer: http://localhost/admin_portal/1014978061/view_transactions
[ pid=660 thr=3078097552 file=ext/apache2/Hooks.cpp:822 time=2011-12-01 16:55:23.563 ]: The backend application (process 796) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application.
[ pid=796 thr=-608495398 file=utils.rb:176 time=2011-12-01 11:25:23.878 ]: *** Exception NoMethodError in application (undefined method `new' for Rails:Module) (process 796, thread #<Thread:0xb77631b4>):
from /home/brijendra/aptana_studio_3_workspace/wantsa/vendor/plugins/newrelic_rpm/lib/new_relic/control.rb:467:in `new_instance'
from /home/brijendra/aptana_studio_3_workspace/wantsa/vendor/plugins/newrelic_rpm/lib/new_relic/control.rb:54:in `instance'
from /home/brijendra/aptana_studio_3_workspace/wantsa/vendor/plugins/newrelic_rpm/lib/new_relic/agent.rb:167:in `logger'
I would be eager to post any other output if required and also suggestions to fix this issue.
I fixed this error by changing cache as true in my development.rb and it worked fine.

Resources