I am super new to docker and trying to configure single node hadoop using docker in ubuntu server. Here is what I have already done.
$ docker pull sequenceiq/hadoop-docker:2.7.1
......
$ docker run -it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash
Starting sshd: [ OK ]
18/06/27 12:59:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [cb46e163e0be]
cb46e163e0be: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-cb46e163e0be.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-cb46e163e0be.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-cb46e163e0be.out
18/06/27 12:59:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn--resourcemanager-cb46e163e0be.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-cb46e163e0be.out
bash-4.1# jps
532 ResourceManager
204 DataNode
118 NameNode
371 SecondaryNameNode
918 Jps
620 NodeManager
jps shows that resource manager is running. Now I tried to test the hadoop
bash-4.1# cd $HADOOP_PREFIX
bash-4.1# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'
18/06/27 13:02:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/06/27 13:02:25 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/06/27 13:02:27 INFO input.FileInputFormat: Total input paths to process : 31
18/06/27 13:02:27 INFO mapreduce.JobSubmitter: number of splits:31
18/06/27 13:02:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1530118774059_0001
18/06/27 13:02:28 INFO impl.YarnClientImpl: Submitted application application_1530118774059_0001
18/06/27 13:02:28 INFO mapreduce.Job: The url to track the job: http://cb46e163e0be:8088/proxy/application_1530118774059_0001/
18/06/27 13:02:28 INFO mapreduce.Job: Running job: job_1530118774059_0001
18/06/27 13:02:44 INFO mapreduce.Job: Job job_1530118774059_0001 running in uber mode : false
18/06/27 13:02:44 INFO mapreduce.Job: map 0% reduce 0%
18/06/27 13:05:56 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
18/06/27 13:05:57 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
18/06/27 13:05:58 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
18/06/27 13:05:59 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
18/06/27 13:06:00 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
18/06/27 13:06:01 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
^C
bash-4.1# jps
532 ResourceManager
204 DataNode
1671 Jps
371 SecondaryNameNode
bash-4.1#
Now I don't understand 2 things here.
This is kind of official image of hadoop so why it is not running correctly? Did I make any mistake? IF yes then what ?
When I did jps before running the example code you can see node manager and name node were returned by jps. But after running the example and quitting the example, these couple of objects were not returned by jps. Why is it so?
Please help. Thanks
Related
There is an error while run wdio test in Docker using Jenkins. I have no idea how to solve this problem :(
The same config run successfully on local env (windows + docker).
This is wdio config. I used default dockerOptions.
wdio.conf
import { config as sharedConfig } from './wdio.shared.conf'
export const config: WebdriverIO.Config = {
...sharedConfig,
...{
host: 'localhost',
services: ['docker'],
dockerLogs: './logs',
dockerOptions: {
image: 'selenium/standalone-chrome:4.1.2-20220131',
healthCheck: {
url: 'http://localhost:4444',
maxRetries: 3,
inspectInterval: 7000,
startDelay: 15000
},
options: {
p: ['4444:4444'],
shmSize: '2g'
}
},
capabilities: [{
acceptInsecureCerts: true,
browserName: 'chrome',
browserVersion: 'latest',
'goog:chromeOptions': {
args: [ '--verbose', '--headless', '--disable-gpu', 'window-size=1920,1800','--no-sandbox', '--disable-dev-shm-usage', '--disable-extensions'],
}
}]
}
}
After that, I try to run UI test via jenkins:
19:37:34 Run `npm audit` for details.
19:37:34 + npm run test:ci -- --spec ./test/specs/claim.BNB.spec.ts
19:37:34
19:37:34 > jasmine-boilerplate#1.0.0 test:ci
19:37:34 > wdio run wdio.ci.conf.ts
And got an error.
Logs attached:
wdio.log
2022-02-04T16:59:20.725Z DEBUG #wdio/utils:initialiseServices: initialise service "docker" as NPM package
2022-02-04T16:59:20.758Z INFO #wdio/cli:launcher: Run onPrepare hook
2022-02-04T16:59:20.760Z DEBUG wdio-docker-service: Docker command: docker run --cidfile /home/jenkins/workspace/tests_e2e1_configure_CI_CD/e2e/selenium_standalone_chrome_latest.cid --rm -p 4444:4444 -p 5900:5900 --shm-size 2g selenium/standalone-chrome:latest
2022-02-04T16:59:20.769Z WARN wdio-docker-service: Connecting dockerEventsListener: 6283
2022-02-04T16:59:20.772Z INFO wdio-docker-service: Cleaning up CID files
2022-02-04T16:59:20.834Z INFO wdio-docker-service: Launching docker image 'selenium/standalone-chrome:latest'
2022-02-04T16:59:20.841Z INFO wdio-docker-service: Docker container is ready
2022-02-04T16:59:20.841Z DEBUG #wdio/cli:utils: Finished to run "onPrepare" hook in 82ms
2022-02-04T16:59:20.842Z INFO #wdio/cli:launcher: Run onWorkerStart hook
2022-02-04T16:59:20.843Z DEBUG #wdio/cli:utils: Finished to run "onWorkerStart" hook in 0ms
2022-02-04T16:59:20.843Z INFO #wdio/local-runner: Start worker 0-0 with arg: run,wdio.ci.conf.ts,--spec,./test/specs/claim.BNB.spec.ts
2022-02-04T16:59:22.034Z DEBUG #wdio/local-runner: Runner 0-0 finished with exit code 1
2022-02-04T16:59:22.035Z INFO #wdio/cli:launcher: Run onComplete hook
2022-02-04T16:59:22.036Z INFO wdio-docker-service: Shutting down running container
2022-02-04T16:59:32.372Z INFO wdio-docker-service: Cleaning up CID files
2022-02-04T16:59:32.373Z INFO wdio-docker-service: Docker container has stopped
2022-02-04T16:59:32.374Z WARN wdio-docker-service: Disconnecting dockerEventsListener: 6283
2022-02-04T16:59:32.374Z DEBUG #wdio/cli:utils: Finished to run "onComplete" hook in 10339ms
2022-02-04T16:59:32.430Z INFO #wdio/local-runner: Shutting down spawned worker
2022-02-04T16:59:32.681Z INFO #wdio/local-runner: Waiting for 0 to shut down gracefully
wdio-0-0.log
2022-02-04T16:59:21.223Z INFO #wdio/local-runner: Run worker command: run
2022-02-04T16:59:21.513Z DEBUG #wdio/config:utils: Found 'ts-node' package, auto-compiling TypeScript files
2022-02-04T16:59:21.714Z DEBUG #wdio/local-runner:utils: init remote session
2022-02-04T16:59:21.717Z DEBUG #wdio/utils:initialiseServices: initialise service "docker" as NPM package
2022-02-04T16:59:21.828Z DEBUG #wdio/local-runner:utils: init remote session
2022-02-04T16:59:21.840Z INFO devtools:puppeteer: Initiate new session using the DevTools protocol
2022-02-04T16:59:21.841Z INFO devtools: Launch Google Chrome with flags: --enable-automation --disable-popup-blocking --disable-extensions --disable-background-networking --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-sync --metrics-recording-only --disable-default-apps --mute-audio --no-first-run --no-default-browser-check --disable-hang-monitor --disable-prompt-on-repost --disable-client-side-phishing-detection --password-store=basic --use-mock-keychain --disable-component-extensions-with-background-pages --disable-breakpad --disable-dev-shm-usage --disable-ipc-flooding-protection --disable-renderer-backgrounding --force-fieldtrials=*BackgroundTracing/default/ --enable-features=NetworkService,NetworkServiceInProcess --disable-features=site-per-process,TranslateUI,BlinkGenPropertyTrees --window-position=0,0 --window-size=1200,900 --headless --disable-gpu --window-size=1920,1800 --no-sandbox --disable-dev-shm-usage --disable-extensions
2022-02-04T16:59:21.911Z ERROR #wdio/runner: Error:
at new LauncherError (/home/jenkins/workspace/tests_e2e1_configure_CI_CD/e2e/node_modules/chrome-launcher/src/utils.ts:31:18)
at new ChromePathNotSetError (/home/jenkins/workspace/tests_e2e1_configure_CI_CD/e2e/node_modules/chrome-launcher/dist/utils.js:33:9)
at Object.linux (/home/jenkins/workspace/tests_e2e1_configure_CI_CD/e2e/node_modules/chrome-launcher/src/chrome-finder.ts:153:11)
at Function.getFirstInstallation (/home/jenkins/workspace/tests_e2e1_configure_CI_CD/e2e/node_modules/chrome-launcher/src/chrome-launcher.ts:182:61)
at Launcher.launch (/home/jenkins/workspace/tests_e2e1_configure_CI_CD/e2e/node_modules/chrome-launcher/src/chrome-launcher.ts:252:37)
at launch (/home/jenkins/workspace/tests_e2e1_configure_CI_CD/e2e/node_modules/chrome-launcher/src/chrome-launcher.ts:74:18)
at launchChrome (/home/jenkins/workspace/tests_e2e1_configure_CI_CD/e2e/node_modules/devtools/build/launcher.js:80:55)
at launch (/home/jenkins/workspace/tests_e2e1_configure_CI_CD/e2e/node_modules/devtools/build/launcher.js:179:16)
at Function.newSession (/home/jenkins/workspace/tests_e2e1_configure_CI_CD/e2e/node_modules/devtools/build/index.js:50:54)
at remote (/home/jenkins/workspace/tests_e2e1_configure_CI_CD/e2e/node_modules/webdriverio/build/index.js:67:43)
wdio-chromedriver.log
Starting ChromeDriver 97.0.4692.71 (adefa7837d02a07a604c1e6eff0b3a09422ab88d-refs/branch-heads/4692#{#1247}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
[1643987609.767][SEVERE]: bind() failed: Cannot assign requested address (99)
docker-log.txt
2022-02-04 16:59:21,482 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
2022-02-04 16:59:21,484 INFO supervisord started with pid 7
Trapped SIGTERM/SIGINT/x so shutting down supervisord...
2022-02-04 16:59:22,487 INFO spawned: 'xvfb' with pid 9
2022-02-04 16:59:22,489 INFO spawned: 'vnc' with pid 10
2022-02-04 16:59:22,491 INFO spawned: 'novnc' with pid 11
2022-02-04 16:59:22,492 INFO spawned: 'selenium-standalone' with pid 12
2022-02-04 16:59:22,493 WARN received SIGTERM indicating exit request
2022-02-04 16:59:22,493 INFO waiting for xvfb, vnc, novnc, selenium-standalone to die
Setting up SE_NODE_GRID_URL...
2022-02-04 16:59:22,501 INFO success: xvfb entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2022-02-04 16:59:22,501 INFO success: vnc entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2022-02-04 16:59:22,501 INFO success: novnc entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Selenium Grid Standalone configuration:
[network]
relax-checks = true
[node]
session-timeout = "300"
override-max-sessions = false
detect-drivers = false
max-sessions = 1
[[node.driver-configuration]]
display-name = "chrome"
stereotype = '{"browserName": "chrome", "browserVersion": "97.0", "platformName": "Linux"}'
max-sessions = 1
Starting Selenium Grid Standalone...
16:59:22.930 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
16:59:22.939 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
16:59:23.452 INFO [NodeOptions.getSessionFactories] - Detected 4 available processors
16:59:23.493 INFO [NodeOptions.report] - Adding chrome for {"browserVersion": "97.0","browserName": "chrome","platformName": "Linux","se:vncEnabled": true} 1 times
16:59:23.505 INFO [Node.<init>] - Binding additional locator mechanisms: name, id, relative
16:59:23.526 INFO [LocalDistributor.add] - Added node 150c2c05-2b08-4ba9-929a-45fef66bb193 at http://172.17.0.2:4444. Health check every 120s
16:59:23.540 INFO [GridModel.setAvailability] - Switching node 150c2c05-2b08-4ba9-929a-45fef66bb193 (uri: http://172.17.0.2:4444) from DOWN to UP
16:59:23.645 INFO [Standalone.execute] - Started Selenium Standalone 4.1.2 (revision 9a5a329c5a): http://172.17.0.2:4444
2022-02-04 16:59:26,091 INFO waiting for xvfb, vnc, novnc, selenium-standalone to die
2022-02-04 16:59:29,095 INFO waiting for xvfb, vnc, novnc, selenium-standalone to die
2022-02-04 16:59:32,097 INFO waiting for xvfb, vnc, novnc, selenium-standalone to die
Background: I have a system behind a proxy/firewall. I can access docker to pull images, but do not have a username/password to access any other sites. Therefore my docker container of sonarqube is essentially offline.
Question: The docker container starts fine the first time, but fails to restart. This happens in two instances, either a manually installed plugin presents an error that it fails to download the update-center url, or it simply starts shutting down immediately as it starts. Both fail the application which closes the container. I do not seem to be able (or understand how to) modify the sonar.properties to get the update-center disabled and need guidance.
I have inquired on the github for the container without much help: https://github.com/SonarSource/docker-sonarqube/issues/76#issuecomment-364563967 The '-Dsonar.updatecenter.activate=false' option does not work when I try it.
Simply shutting down
2018.02.09 21:45:38 INFO ce[][o.s.p.ProcessEntryPoint] Starting ce
2018.02.09 21:45:38 INFO ce[][o.s.ce.app.CeServer] Compute Engine starting up...
2018.02.09 21:45:39 INFO ce[][o.e.p.PluginsService] no modules loaded
2018.02.09 21:45:39 INFO ce[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
2018.02.09 21:45:39 INFO ce[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
2018.02.09 21:45:39 INFO ce[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2018.02.09 21:45:39 INFO ce[][o.e.p.PluginsService] loaded plugin org.elasticsearch.transport.Netty4Plugin]
2018.02.09 21:45:41 INFO ce[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9001]
2018.02.09 21:45:41 INFO ce[][o.sonar.db.Database] Create JDBC data source for jdbc:postgresql://pgsonar:5432/sonar
2018.02.09 21:45:43 INFO ce[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /opt/sonarqube
2018.02.09 21:45:43 INFO ce[][o.s.c.c.CePluginRepository] Load plugins
2018.02.09 21:45:45 INFO ce[][o.s.c.q.PurgeCeActivities] Delete the Compute Engine tasks created before Sun Aug 13 21:45:45 UTC 2017
2018.02.09 21:45:45 INFO ce[][o.s.ce.app.CeServer] Compute Engine is operational
2018.02.09 21:45:45 INFO app[][o.s.a.SchedulerImpl] Process[ce] is up
2018.02.09 21:45:45 INFO app[][o.s.a.SchedulerImpl] SonarQube is up
2018.02.09 21:47:12 INFO app[][o.s.a.SchedulerImpl] Stopping SonarQube
2018.02.09 21:47:13 INFO ce[][o.s.p.StopWatcher] Stopping process
2018.02.09 21:47:13 INFO ce[][o.s.ce.app.CeServer] Compute Engine is stopping...
2018.02.09 21:47:13 INFO ce[][o.s.c.t.CeProcessingSchedulerImpl] Waiting for workers to finish in-progress tasks
2018.02.09 21:47:14 INFO ce[][o.s.ce.app.CeServer] Compute Engine is stopped
2018.02.09 21:47:15 INFO app[][o.s.a.SchedulerImpl] Process [ce] is stopped
2018.02.09 21:47:15 INFO web[][o.s.p.StopWatcher] Stopping process
2018.02.09 21:47:18 INFO app[][o.s.a.SchedulerImpl] Process [web] is stopped
2018.02.09 21:47:18 INFO app[][o.s.a.SchedulerImpl] Process [es] is stopped
2018.02.09 21:47:18 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
2018.02.09 21:47:18 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
chown: cannot access '/opt/sonarqube/temp/README.txt': No such file or directory
Will update with the fail to download later (no access to logs at this exact moment)
Regarding the README.txt issue, you have to create a volume and mount the temp folder (note that I use the postgres setup from anorak:girl). You can then start and stop with no problems.
sudo docker volume create sonarqube-temp
sudo docker run -d --name sonarqube --link sonar-postgres:pgsonar -p 9000:9000 -e SONARQUBE_JDBC_USERNAME=sonar -e SONARQUBE_JDBC_PASSWORD='secure' -e SONARQUBE_JDBC_URL=jdbc:postgresql://pgsonar:5432/sonar -v sonarqube-temp:/opt/sonarqube/temp sonarqube:lts
Regarding the UpdateCenter issue, workaround is to specify a configuration with the run command (this is specific to Godin's docker container for sonarqube - through his run.sh script):
sudo docker run -d --name sonarqube --link sonar-postgres:pgsonar -p 9000:9000 -e SONARQUBE_JDBC_USERNAME=sonar -e SONARQUBE_JDBC_PASSWORD='secure' -e SONARQUBE_JDBC_URL=jdbc:postgresql://pgsonar:5432/sonar -v sonarqube-temp:/opt/sonarqube/temp sonarqube:lts -Dsonar.updatecenter.activate=false
I've installed a neo4j database on a Google Cloud Compute instance and am wanting to connect to the database from my laptop.
[1] I have neo4j running on Google Cloud
● neo4j.service - Neo4j Graph Database
Loaded: loaded (/lib/systemd/system/neo4j.service; disabled; vendor preset: enabled)
Active: active (running) since Sat 2017-09-30 09:33:39 UTC; 1h 3min ago
Main PID: 2099 (java)
Tasks: 41
Memory: 504.5M
CPU: 18.652s
CGroup: /system.slice/neo4j.service
└─2099 /usr/bin/java -cp /var/lib/neo4j/plugins:/etc/neo4j:/usr/share/neo4j/lib/*:/var/lib/neo4j/plugins/* -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+U
nlockExperimentalVMOptions -XX:+TrustFinalNonStaticFields -XX:+DisableExplicitGC -Djdk.tls.ephemeralDHKeySize=2048 -Dunsupported.dbms.udc.source=debian -Dfile.encoding=UTF-8 org.neo4j.server.Commu
nityEntryPoint --home-dir=/var/lib/neo4j --config-dir=/etc/neo4j
Sep 30 09:33:40 neo4j-graphdb-server neo4j[2099]: certificates: /var/lib/neo4j/certificates
Sep 30 09:33:40 neo4j-graphdb-server neo4j[2099]: run: /var/run/neo4j
Sep 30 09:33:40 neo4j-graphdb-server neo4j[2099]: Starting Neo4j.
Sep 30 09:33:42 neo4j-graphdb-server neo4j[2099]: 2017-09-30 09:33:42.948+0000 INFO ======== Neo4j 3.2.5 ========
Sep 30 09:33:42 neo4j-graphdb-server neo4j[2099]: 2017-09-30 09:33:42.988+0000 INFO Starting...
Sep 30 09:33:44 neo4j-graphdb-server neo4j[2099]: 2017-09-30 09:33:44.308+0000 INFO Bolt enabled on 127.0.0.1:7687.
Sep 30 09:33:47 neo4j-graphdb-server neo4j[2099]: 2017-09-30 09:33:47.043+0000 INFO Started.
Sep 30 09:33:48 neo4j-graphdb-server neo4j[2099]: 2017-09-30 09:33:48.160+0000 INFO Remote interface available at http://localhost:7474/
Sep 30 09:39:17 neo4j-graphdb-server neo4j[2099]: 2017-09-30 09:39:17.918+0000 WARN badMessage: 400 No URI for HttpChannelOverHttp#27d4a9b{r=0,c=false,a=IDLE,uri=-}
Sep 30 09:46:18 neo4j-graphdb-server neo4j[2099]: 2017-09-30 09:46:18.374+0000 WARN badMessage: 400 for HttpChannelOverHttp#6cbed0ca{r=0,c=false,a=IDLE,uri=-}
[2] I've created a firewall rule on Google Cloud to allow external access to the DB server
The network tag of "google-db-server" has been added to the Google Cloud Compute server.
My expectation is that the rule below will allow any external machine to connect to port 7474 on the Google Cloud Compute instance
user#machine:~/home$ gcloud compute firewall-rules create custom-allow-neo4j --action ALLOW --rules tcp:7474 --description "Enable access to the neo4j database" --direction IN --target-tags google-db-server
user#machine:~/home$ gcloud compute firewall-rules list --format json
[
{
"allowed": [
{
"IPProtocol": "tcp",
"ports": [
"7474"
]
}
],
"creationTimestamp": "2017-09-30T00:25:51.220-07:00",
"description": "Enable access to the neo4j database",
"direction": "INGRESS",
"id": "5767618134171383824",
"kind": "compute#firewall",
"name": "custom-allow-neo4j",
"network": "https://www.googleapis.com/compute/v1/projects/graphdb-experiment/global/networks/default",
"priority": 1000,
"selfLink": "https://www.googleapis.com/compute/v1/projects/graphdb-experiment/global/firewalls/custom-allow-neo4j",
"sourceRanges": [
"0.0.0.0/0"
],
"targetTags": [
"google-db-server"
]
},
[3] Running nmap from the Google Cloud server instance shows that port 7474 is available locally, and I can telnet to that port locally
google_user#google-db-server:~$ nmap -p 22,80,443,7474 localhost
Starting Nmap 7.01 ( https://nmap.org ) at 2017-09-30 10:46 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000081s latency).
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp closed https
7474/tcp open unknown
Nmap done: 1 IP address (1 host up) scanned in 0.03 seconds
google-user#google-db-server:~$ telnet localhost 7474
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
[4] However I am unable to connect from my laptop and nmap shows port 7474 as unavailable
user#machine:~/home$ nmap -p 22,80,443,7474 35.201.26.52
Starting Nmap 7.01 ( https://nmap.org ) at 2017-09-30 20:50 AEST
Nmap scan report for 52.26.201.35.bc.googleusercontent.com (35.201.26.52)
Host is up (0.28s latency).
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp closed https
7474/tcp closed unknown
Nmap done: 1 IP address (1 host up) scanned in 0.75 seconds
So despite the firewall rule being created to allow any IP address to connect to the Google Cloud Compute instance on tcp:7474, I'm still unable to access this port from my laptop.
Am I missing some additional steps?
It looks like neo4j is only listening on the loopback interface. This means it only accepts connections from the same machine. You can verify this by running sudo netstat -lntp. If you see 127.0.0.1:7474, it's only listening on loopback. It should be 0.0.0.0:7474.
you can fix this in the neo4j config by setting dbms.connector.bolt.listen_address to 0.0.0.0:7474. Your Linux distribution may also have a different place to set this configuration.
I have 3 CentOS VMs and I have installed Zookeeper, Marathon, and Mesos on the master node, while only putting Mesos on the other 2 VMs. The master node has no mesos-slave running on it. I am trying to run Docker containers so i specified "docker,mesos" in the containerizes file. One of the mesos-agents starts fine with this configuration and I have been able to deploy a container to that slave. However, the second mesos-agent simply fails when I have this configuration (it works if i take out that containerizes file but then it doesn't run containers). Here are some of the logs and information that has come up:
Here are some "messages" in the log directory:
Apr 26 16:09:12 centos-minion-3 systemd: Started Mesos Slave.
Apr 26 16:09:12 centos-minion-3 systemd: Starting Mesos Slave...
WARNING: Logging before InitGoogleLogging() is written to STDERR
[main.cpp:243] Build: 2017-04-12 16:39:09 by centos
[main.cpp:244] Version: 1.2.0
[main.cpp:247] Git tag: 1.2.0
[main.cpp:251] Git SHA: de306b5786de3c221bae1457c6f2ccaeb38eef9f
[logging.cpp:194] INFO level logging started!
[systemd.cpp:238] systemd version `219` detected
[main.cpp:342] Inializing systemd state
[systemd.cpp:326] Started systemd slice `mesos_executors.slice`
[containerizer.cpp:220] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
[linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
[provisioner.cpp:249] Using default backend 'copy'
[slave.cpp:211] Mesos agent started on (1)#172.22.150.87:5051
[slave.cpp:212] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="docker,mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher="linux" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --runtime_dir="/var/run/mesos" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"
[slave.cpp:541] Agent resources: cpus(*):1; mem(*):919; disk(*):2043; ports(*):[31000-32000]
[slave.cpp:549] Agent attributes: [ ]
[slave.cpp:554] Agent hostname: node3
[status_update_manager.cpp:177] Pausing sending status updates
[state.cpp:62] Recovering state from '/var/lib/mesos/meta'
[state.cpp:706] No committed checkpointed resources found at '/var/lib/mesos/meta/resources/resources.info'
[status_update_manager.cpp:203] Recovering status update manager
[docker.cpp:868] Recovering Docker containers
[containerizer.cpp:599] Recovering containerizer
[provisioner.cpp:410] Provisioner recovery complete
[group.cpp:340] Group process (zookeeper-group(1)#172.22.150.87:5051) connected to ZooKeeper
[group.cpp:830] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
[group.cpp:418] Trying to create path '/mesos' in ZooKeeper
[detector.cpp:152] Detected a new leader: (id='15')
[group.cpp:699] Trying to get '/mesos/json.info_0000000015' in ZooKeeper
[zookeeper.cpp:259] A new leading master (UPID=master#172.22.150.88:5050) is detected
Failed to perform recovery: Collect failed: Failed to run 'docker -H unix:///var/run/docker.sock ps -a': exited with status 1; stderr='Cannot connect to the Docker daemon. Is the docker daemon running on this host?'
To remedy this do as follows:
Step 1: rm -f /var/lib/mesos/meta/slaves/latest
This ensures agent doesn't recover old live executors.
Step 2: Restart the agent.
Apr 26 16:09:13 centos-minion-3 systemd: mesos-slave.service: main process exited, code=exited, status=1/FAILURE
Apr 26 16:09:13 centos-minion-3 systemd: Unit mesos-slave.service entered failed state.
Apr 26 16:09:13 centos-minion-3 systemd: mesos-slave.service failed.
Logs from docker:
$ sudo systemctl status docker
● docker.service - Docker Application Container Engine Loaded:
loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/docker.service.d
└─flannel.conf Active: inactive (dead) since Tue 2017-04-25 18:00:03 CDT;
24h ago Docs: docs.docker.com Main PID: 872 (code=exited, status=0/SUCCESS)
Apr 26 18:25:25 centos-minion-3 systemd[1]: Dependency failed for Docker Application Container Engine.
Apr 26 18:25:25 centos-minion-3 systemd[1]: Job docker.service/start failed with result 'dependency'
Logs from flannel:
[flanneld-start: network.go:102] failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
You have answer in your logs
Failed to perform recovery: Collect failed:
Failed to run 'docker -H unix:///var/run/docker.sock ps -a': exited with status 1;
stderr='Cannot connect to the Docker daemon. Is the docker daemon running on this host?'
To remedy this do as follows:
Step 1: rm -f /var/lib/mesos/meta/slaves/latest
This ensures agent doesn't recover old live executors.
Step 2: Restart the agent.
Mesos keeps it state/metadata on local disk. When it's restarted it try to load this state. If configuration changed and is not compatible with previous state it won't start.
Just bring docker to live by fixing problems with flannel and etcd and everything will be fine.
add the following flag while starting agent,
--reconfiguration_policy=additive
more details here: http://mesos.apache.org/documentation/latest/agent-recovery/
I'm facing issues when installing thingsboard using docker-compose on ubuntu
images are correctly pulled , container seems to be up but logs shows :
logs for thingsboard/application:1.2.2 :
thingsboard-db-schema container is still in progress. waiting until it
completed...
thingsboard-db-schema container is still in progress. waiting until it
completed...
thingsboard-db-schema container is still in progress. waiting until it
completed...
thingsboard-db-schema container is still in progress. waiting until it
completed...
thingsboard-db-schema container is still in progress. waiting until it
completed...
thingsboard-db-schema container is still in progress. waiting until it
completed...
logs for thingsboard/thingsboard-db-schema:1.2.2
Wait for Cassandra...
Failed to resolve "db".
WARNING: No targets were specified, so 0 hosts scanned.
Wait for Cassandra...
Failed to resolve "db".
WARNING: No targets were specified, so 0 hosts scanned.
Wait for Cassandra...
seems that the first container waiting cassandra to be up which is not the case
Any suggestions ?
Thanks in advance
Please check output of the DB container using command 'docker-compose logs -f db' and verify that cassandra is ready to accept client on 9042 port:
db_1 | INFO 11:02:07 Waiting for gossip to settle before accepting client requests...
db_1 | INFO 11:02:15 No gossip backlog; proceeding
db_1 | INFO 11:02:15 Netty using native Epoll event loop
db_1 | INFO 11:02:15 Using Netty Version: [netty-buffer=netty-buffer-4.0.39.Final.38bdf86, netty-codec=netty-codec-4.0.39.Final.38bdf86, netty-codec-haproxy=netty-codec-haproxy-4.0.39.Final.38bdf86, netty-codec-http=netty-codec-http-4.0.39.Final.38bdf86, netty-codec-socks=netty-codec-socks-4.0.39.Final.38bdf86, netty-common=netty-common-4.0.39.Final.38bdf86, netty-handler=netty-handler-4.0.39.Final.38bdf86, netty-tcnative=netty-tcnative-1.1.33.Fork19.fe4816e, netty-transport=netty-transport-4.0.39.Final.38bdf86, netty-transport-native-epoll=netty-transport-native-epoll-4.0.39.Final.38bdf86, netty-transport-rxtx=netty-transport-rxtx-4.0.39.Final.38bdf86, netty-transport-sctp=netty-transport-sctp-4.0.39.Final.38bdf86, netty-transport-udt=netty-transport-udt-4.0.39.Final.38bdf86]
db_1 | INFO 11:02:15 Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
Output should be like logs above.
Plus additionally verify that no errors happened during the cassandra start up.