I am a beginner in Hadoop.
Two big issues that I am trying to fix are :-
1.)While starting hadoop through cmd (start-all.cmd) my namenode is not starting. It is showing the below error .
SHUTDOWN_MSG :Shutting down NameNode at Admin/192.168.1.6
I tried to fix it by following the below steps :-
a) Firstly getting the host name of my computer. It can be obtained by running $hostname command. My hostname is Admin. Then adding 192.168.1.6 localhost hostname into the /etc/hosts file.
But I am not getting file named hosts in etc folder. I am confused on whether I have to create a file (.txt) in etc folder or I need to do anything else ?
2.) Also, on my analysis, I checked that while firing the command (start-all.cmd), three folders are being created automatically, i.e, hortonworks, USER and tmp. I am wondering about the reason. Please help.
/etc/hosts is a text file named exactly that as-is on most all Linux/Unix operating systems. Are you running a Linux OS ?
You can check if your Linux OS is using it by running
cat /etc/nsswitch.conf | grep host
Looks for a line in the output that says
hosts files dns
If you see the word files in the row with hosts your system should use it.
You can view its contents by running cat /etc/hosts.
You can run the following to add your entry to hosts the run cat /etc/hosts to check it.
echo "192.168.1.6 localhost Admin" >> /etc/hosts
Related
I am trying to run the puppet pupperware suite (all 3 servers/puppet server/puppet DB/DB server).
I am using the official Yaml file provided by puppetlabs for docker compose : https://github.com/puppetlabs/pupperware/blob/master/docker-compose.yml
When I run that Yaml file in docker compose however, I am running into the following error (from docker-compose logs):
postgres_1 | ls: cannot open directory '/docker-entrypoint-initdb.d/': Permission denied
And as a result, the build fails (only the puppet server comes up, but not the other ones).
My docker host is a Fedora 33 virtual machine running inside a Proxmox environment. Proxmox runs on the physical host.
I have disabled SELinux, and I am running docker (moby) rootless. My local user (uid 1000) can run docker without sudo.
I believe I need to set permission in the container (probably via a Dockerfile) but I am not sure how to change that and I am not sure how to use a Dockerfile and docker-compose simultaneously.
thank you for your help
The docker-compose file is from the Puppet 6 era. The docker images that the Pupperware setup currently pulls, are latest, which is Puppet 7.
I got my pre-existing setup functioning again by changing the image names to:
puppet/puppetserver:6.14.1
postgres:9.6
puppet/puppetdb:6.13.1
Maybe this works for you as well.
well, since it's been a month and you have no answers I will tell try to help you with what I know.
You should put a Dockerfile in the root of your project. It contains commands to be run by the docker daemon AND the commands run by the linux inside the container. Then it runs through the contents of your docker-compose.yml and runs the commands in there.
So to solve the permission problem you should add RUN, which executes the linux command in Bash and add data to the folder.
Also look at this answer
I have a series of containers created with docker-compose. Some of these containers communicate between each other with some rules defined in the docker-compose.yml file.
I need to move those containers from a serverA to serverB (same OS) but i'm having issues in understanding how this works.
I tried both with the export and the save methods following tutorials i've found on the web but I was not able to get the port configurations and networking rules after the export - import or save - load operations (there's a chance I didn't really get how they work...)
The only way I've found to succesfully do this is to copy the whole docker-compose folder and run docker-compose up in serverB.
The question:
Is there a way to preserve the whole configuration of the containers and move them from a server to another using the export or save function?
Thank you for any help you can provide
2 scenarios:
Copy via ssh
$ sudo docker save myImage:tag | ssh user#IPhost:/remote/dir docker load -
Copy via scp
#Host A
$ docker save Image > myImage.tar
$ scp myImage.tar IPhostB:/tmp/myImage.tar
# Host B
$ docker load -i /tmp/myImage.tar
And then you need to copy the docker-compose.yml to the host B too.
The containers only have the original build's own configurations, but they don't save the environment that we generate with the file docker-compose.yml
Bye
I am new to docker and containers. I have a container consisting of an MRI analysis software. Within this container are many other software the main software draws its commands from. I would like to run a single command from one of the softwares in this container using research data that is located on an external hard drive which is plugged into my local machine that is running docker.
I know there is a cp command for copying files (such as scripts) into containers and most other questions along these lines seem to recommend copying the files from your local machine into the container and then running the script (or whatever) from the container. In my case I need the container to access data from separate folders in a directory structure and copying over the entire directory is not feasible since it is quite large.
I honestly just want to know how I can run a single command inside the docker using inputs present on my local machine. I have run docker ps to get the CONTAINER_ID which is d8dbcf705ee7. Having looked into executing commands inside containers I tried the following command:
docker exec d8dbcf705ee7 /bin/bash -c "mcflirt -in /Volumes/DISS/FMRIPREP/sub-S06V1A/func/sub-S06V1A_task-compound_run-01_bold.nii -out sub-S06V1A_task-compound_run-01_bold_mcf_COMMAND_TEST.nii.gz -reffile /Volumes/DISS/FMRIPREP_TMP/sub-S06V1A_dof6_ver1.2.5/fmriprep_wf/single_subject_S06V1A_wf/func_preproc_task_compound_run_01_wf/bold_reference_wf/gen_ref/ref_image.nii.gz -mats -plots"
mcflirt is the command I want to run inside the container. I believe the exec command would do what I hope since if I run docker exec d8dbcf705ee7 /bin/bash -c "mcflirt" I will get help output for the mcflirt command which is the expected outcome in that case. The files inside of the /Volume/... paths are the files on my local machine I would like to access. I understand that the location of the files is the problem since I cannot tab complete the paths within this command; when I run this I get the following output:
Image Exception : #22 :: ERROR: Could not open image /Volumes/DISS/FMRIPREP/sub-S06V1A/func/sub-S06V1A_task-compound_run-01_bold
terminate called after throwing an instance of 'RBD_COMMON::BaseException'
Can anyone point me in the right direction?
So if I got you right, you need to execute some shell script and provide the context (like local files).
The way is straightforward.
Lets say your script and all needed files are located in /hello folder of your host PC (no matter really if they are stored together or not, just showing the technique).
/hello
- runme.sh
- datafile1
- datafile1
You mount this folder into your container to make the files accessible inside. If you dont need container to modify them, better mount in readonly mode.
You launch docker like this:
docker run -it -v /hello:/hello2:ro ubuntu /hello2/runme.sh
And that's it! Your script runme.sh gets executed inside container and it has access to nearby files. Thanks to -v /hello:/hello2:ro directive. It maps host's folder /hello into container's folder /hello2 in readonly ro mode.
Note you can have same names, I've just differed them to show the difference.
I am running the mqtt broker Mosqitto in a docker image.
I am using the following arguments
sudo docker run -d -p 1883:1883 -p 1884:1884 -v /home/mosquitto/apps/dev/mosquitto:/mosquitto --restart always -u mosquitto eclipse-mosquitto:1.4.
This should mount the host folder /home/mosquitto/apps/dev/mosquitto to the image folder /mosquitto
The problem is that the host user IDs (1001) and the docker user IDs (100) do not match.
If I do not specify -u mosquitto, the application complains about not being able to write to /mosquitto/logs/mosquitto.log
So I thought I'd specify -u mosquitto, to make the application inside the image run as user 1001, and therefore have write access to the mounted files.
Which worked.
But then, the Mosquitto application made a new database file on exit. That file was made with the 101 user as owner..
What exactly happens when I specify -U to docker.
How come it kind of did what I was expecting (allowed writing to host files) and kind of didn't do what I was expecting(still made files with the original image user id)
Maybe this is something to do with this specific docker image .. it runs some script internally that switches user?
How about making write access to log path for any user? It may be less secure. But if it is just logs, lets see application inside docker can write to it.
Or think about bootstrap some commands to the container to make permission changes inside.
If you are using Linux or OSX for your Docker location, most likely it is a security or file permissions issue. Go to this bug report Permission denied for directories created automatically by Dockerfile ADD command #1295 and jump to the end...there are several links to sub-bug reports where you can most likely find your solution. I had a very similar issue, and it turned out to be a selinux misconfiguration.
I'm new to Docker, I come from Vagrant.
I'm using Docker (1.9.1) inside my "D:/Works/something/DockerFirstTime" folder.
Now I create the machine with
docker-machine create first
and simple Dockerfile:
FROM ruby:2.2-onbuild
and simple Gemfile:
source 'https://rubygems.org'
gem 'rails'
Now with this command I want to use a shared folder like in Vagrant in the same hard drive of my Dockerfile:
docker run -it -v //d/Works/something/DockerFirstTime:/usr/src/app -w /usr/src/app ruby:2.2 bundle install
But it doesn't works.
How to do this?
I know that Docker only shares the /c/User/folder, is that right?
How can I use the folder with the files and modify my files with editor in Windows and then restart server like in a normal shell on a single PC or like in Vagrant?
This question and this question have a similar root problem, mounting a non C:/ drive folder in boot2docker. I wrote an in-depth answer to the other question that provide the same information that is in the first half of #VonC's answer.
From Docker Docs:
All other paths come from your virtual machine’s filesystem. [...] In
the case of VirtualBox you need to make the host folder available as a
shared folder in VirtualBox. Then, you can mount it using the Docker
-v flag.
To get your folder mounted in a container:
This mounts your entire D:\ drive, you can simply change the file paths to be more granular and specific.
Share the directory with VBox:
This only needs to be done once.
In windows CMD:
VBoxManage sharedfolder add "boot2docker-vm" --name "d-share" --hostpath "D:\"
Mount the shared directory in your VM:
This will need to be done each time you restart the VM.
In the Boot2Docker VM terminal:
mount -t vboxsf -o uid=1000,gid=50 d-share /d
To see sources and explanation for how this works see my full answer to the other similar question
After this you can use the -v/--volume flag in Docker to mount this folder or any sub-folders or files into containers. If you mounted your whole D:\ drive you can use that exact docker run command from your question and it should now work. If you mounted a specific part of you drive you will have to change the paths to match.
To edit in windows, run in docker:
Also from Docker Docs:
Mounting a host directory can be useful for testing. For example, you
can mount source code inside a container. Then, change the source code
and see its effect on the application in real time.
As a VBox shared directory you should be able to see changes made from the Windows side reflected in the boot2docker vm.
You may need to restart containers to see the changes actually appear, this depends on how the program running inside the container, in your case ruby, uses the files. If the files are compiled into an app when the container starts, for example, you will definitely need to restart the container to see the changes.
Note:
Beware the CR LF vs. LF line ending difference when writing files in Windows and reading them in Linux. Make sure your text editor is saving files with Unix line endings or else you may start to see errors caused by '^M' appended to the end of all your lines.
I know that Docker only shares the /c/User/folder, is that right?
It does, and it is able to do so because the VirtualBox VM used for providing a Linux host for docker is sharing C:\Users.
For docker to see another folder, you would need to:
use VBoxmanage sharedfolder add "VM name" --name "sharename" --hostpath "D:\Works"
then mount /D/Works within a VM session, as mentioned in "share windows folder (other than c/Users/) with docker container (using docker windows client)", and mentioned in boot2docker:
mount -t vboxsf -o uid=1000,gid=50 sharename /some/mount/location
The issue with that last alternative is described in "
Introduction to boot2docker" (scroll down to the "Shared folders" section)
The main issue with vboxsf is that it does not do any sort of caching sort of caching so when you are attempting to share a large amount of small files (big git repo’s) or anything that is filesystem read heavy (grunt) performance becomes a factor.
The best solution I have come up with so far is using vagrant with a customized version of boot2docker with NFS support enabled, which has very little “hacking” to get working which is nice.
And a good enough selling point for me is the speed increase by using NFS instead of vboxsf, it’s pretty staggering actually.
This is the project that I have been using https://vagrantcloud.com/yungsang/boxes/boot2docker.
The magic sauce in the volume sharing is in this line.
config.vm.synced_folder ".", "/vagrant", type: "nfs"
Which tells Vagrant to share your current directory in to the boot2docker VM in the /vagrant directory, using NFS.
However, that project seems quite old and would need to be adapted in order to include the latest boot2docker.iso (docker 1.9.1).