Some tools like Hadoop need to explicitly especify the name of workers (section Slaves File in docs), but when deploys with Docker Swarm it assigns automatic container names, so workers file doesn't work anymore as the names in it don't exist. Is there any way to avoid this file or, at least, assign aliases for containers (independently of container name) to make it work?
Maybe I can't use docker-compose.yml file and I must create the services manually over the cluster... Any kind of light on the subject would be really appreciated
Well, Hadoop documentation sucks... Apparently if you set the alias of master node in the core-site.xml file you can omit the workers file. These are the step I followed:
Customized the core-site.xml file (in my docker-compose.yml file I put my master service with the name nodemaster). This file must be in master and workers nodes:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://nodemaster:9000</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://nodemaster:9000</value>
</property>
</configuration>
</configuration>
Now when you run:
start-dfs.sh
start-yarn.sh
I'll connect to the master automatically
Related
I'm trying to use a Maven Docker image to build a project. The pom of the project has it's parent in a Nexus repository. When I build the project, it says that the parent pom is not resolvable since it doesn't look for it in Nexus. How do I make the image look in Nexus? I am able to build the same project locally on my machine using the Maven installation.
This is what my Dockerfile looks like:
FROM maven:3.6.3-amazoncorretto-11
COPY settings.xml /usr/share/maven/ref/
CMD ["mvn","-v"]
You should put a section in your settings file telling it where the nexus repo is. This is done with the servers and mirrors sections and can also be repeated in individual poms if you wish to override it.
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
<servers>
<server>
<id>nexus-snapshots</id>
<username>admin</username>
<password>admin123</password>
</server>
<server>
<id>nexus-releases</id>
<username>admin</username>
<password>admin123</password>
</server>
</servers>
<mirrors>
<mirror>
<id>central</id>
<name>central</name>
<url>http://your-host:8081/repository/maven-group/</url>
<mirrorOf>*</mirrorOf>
</mirror>
</mirrors>
</settings>
This example is taken from a sonatype blog entry from a couple of years ago, but I think it still works.
I have a project that is built via maven, its a dockerized project for a node application.
I want to be able to customize my CMD/EntryPoint based on the maven build arguments.
I know that when we do docker run and provide it the arguments it is accepted and that works fine.
but I want to do the same thing from maven commandline.
Is there a way to let docker run know the argument passed in the maven commandline?
or even better can I edit the dockerfile and read commandline args of maven and use in the dockerfile ENTRYPOINT?
Thanks in advance,
Minakshi
Based on this, you can either use maven-resources-plugin to replace instances of ${...} with the values set in maven before you build the docker file.
Example:
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
<executions>
<execution>
<id>filter-dockerfile</id>
<phase>generate-resources</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}</outputDirectory>
<resources>
<resource>
<directory>src/main/docker</directory>
<filtering>true</filtering>
</resource>
</resources>
</configuration>
</execution>
</executions>
This assume your docker file is under src/main/docker/ path. The replaced docker file will be copied on ${project.build.directory} path.
Or based on this comment, you could pass arguments to docker file.
Example:
On your maven docker plugin
<configuration>
<buildArgs>
<artifactId>${project.artifactId}</artifactId>
<groupId>${project.groupId}</groupId>
</buildArgs>
</configuration>
Then access those properties as ARGs on docker file
ARG artifactId
ARG groupId
ENV ARTIFACT_ID=${artifactId} GROUP_ID=${groupId}
Hope this help answer you question.
Thank you for the responses
I used the resource filtering in maven to solve my problem:
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<userdefvariable></userdefvariable> // variable that you want to pass along
</properties>
<build>
<resources>
<resource>
<directory>src/main/resource</directory> // path to the file (can be anything)
<filtering>true</filtering> // must be true this is what does replacement
</resource>
</resources>
</build>
add to maven commands: "resources:resources -Duserdefvariable=value"
//this setup generates a file in target folder after running the mvn commands, where the variable is injected with the value given by the user.
in Dockerfile now you can instead put in a command to run the file:
CMD ["sh", "path to the script in target folder"]
// in this script should be the commands that you want to use
I'm setting up an HBase cluster in fully distributed mode, with 1 HMaster node and 3 Regionservers node
My hbase-site.xml file contain
<configuration>
<property>
<name>hbase.master</name>
<value>hbase.master:60000</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://hadoop.master:9000/data/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/tmp</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>hbase.master</value>
</property>
</configuration>
My hadoop cluster is running normally.
I run Zookeeper on the same machine with hbase master, and the configuration file zoo.cfg as it default value.
When I start cluster and view the Hbase Master web UI, all the Regionserver appear in the list, but when I try to create table or something else command such as hbase>status they always show Exeception:
ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1889)
at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:695)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42406)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2033)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
So what's wrong with my cluster?
I found out that I'm using Docker version 1.8.2
After fully remove Docker then install the older one (1.7.0) then my script run normally
We are using Wildfly 8.0.0 Final but are in the process of moving to Wildfly 8.2. We are using Arquillian to run our unit tests in the container. I have noticed that Arquillian always seems to use the Wildfly standalone.xml.
It would be useful to be able to tell Arquillian what configuration to use when starting Wildfly. Wildfly comes with several different configuration files. It would be useful to have able to have Arquillian run wildfly with a specific configuration or even tell Arquillian what configuration to use for a test.
We use the Wildfly CLI to configure wildfly properties. This configuration is stored in the configuration file. If we could specify which configuration to use in starting Wildfly for a test we could then test our different configurations.
This seems reasonable, but I haven't found a way to do this.
The Wildfly configuration file is specified by the startup parameter, --server-config.
As John wrote, you can. Adding an example of /arquillian.xml:
(This is for WFly 10.x but it's been the same since AS 7 I think.)
<arquillian xmlns="http://jboss.org/schema/arquillian"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://jboss.org/schema/arquillian
http://jboss.org/schema/arquillian/arquillian_1_0.xsd">
<container qualifier="jbossas-managed" default="true">
<configuration>
<property name="jbossHome">target/wildfly-10.1.0.Final</property>
<property name="serverConfig">standalone-full.xml</property>
<property name="javaVmArguments">-Xms64m -Xmx2048m -Dorg.jboss.resolver.warning=true -Djboss.socket.binding.port-offset=100</property>
<property name="managementPort">10090</property>
<!--<property name="javaVmArguments">-Xms64m -Xmx2048m -Dorg.jboss.resolver.warning=true -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=y</property>-->
</configuration>
</container>
</arquillian>
Yes, just specify serverConfig in your arquillian.xml. By default it will be standalone.xml
I have a problem with a build where I have to resolve non-standard artifacts through Apache Ivy.
Problem:
I have dependencies on two artifacts (a.jar and a-lib.jar).
The two dependencies come only as part of a single installer (a_installer.jar).
The installer can be downloaded, the embedded artifacts themselves not.
It's possible to manipulate the installer to unpack the needed dependencies.
Requirements:
I have to resolve/download the artifacts during the build (I cannot keep the installer or the extracted artifacts with my code).
I cannot use a repository to store the extracted artifacts.
Subclassing/Extending Ivy/whatever is perfectly fine.
Has anyone solved a similar problem, or some helpful information to share?
Or maybe I'm approaching the problem in the wrong way? From what I found so far on the web, people seem to use Ivy just to download files and post-process them manually (with Ant/whatever) after the fact, and not actually resolving more complicated dependencies within Ivy.
Thanks
PS: I don't care whether the installer is also put into the ivy download cache, but I would like to download the installer only one time (and not for both dependencies).
The problem with a call to "ivy:retrieve" is that you need to also add an "artifact" tag in your ivy.xml (complete with URL) in order to retrieve a dependency not found in a Maven respository...
I don't like this for two reasons
The ivy.xml should just declare your dependencies, not their locations.
Need additonal custom logic in the build.xml to handle the 3rd party package
Ideally it should be your repository settings that decide how to download the various jars, that is why I like the packager resolver. Even if the library I want is not in Maven, I can configure ivy to handle it.
The following is an example of turning the jreleaseinfo project into an ivy dependency (hosted in sourceforge, I couldn't find it in Maven)
ivy.xml:
<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="ivy_packager"/>
<dependencies>
<dependency org="ch.oscg" name="jreleaseinfo" rev="1.3.0"/>
</dependencies>
</ivy-module>
Declare two resolvers. Default is Maven2, the other is a packager configured to look locally for instructions. (See also the Ivy Roundup project)
ivysettings.xml
<ivysettings>
<settings defaultResolver="maven2"/>
<resolvers>
<ibiblio name="maven2" m2compatible="true"/>
<packager name="repackage" buildRoot="${user.home}/.ivy2/packager/build" resourceCache="${user.home}/.ivy2/packager/cache">
<ivy pattern="file:///${basedir}/repository/[organisation]/[module]/[revision]/ivy.xml"/>
<artifact pattern="file:///${basedir}/repository/[organisation]/[module]/[revision]/packager.xml"/>
</packager>
</resolvers>
<modules>
<module organisation="ch.oscg" name="jreleaseinfo" resolver="repackage"/>
</modules>
</ivysettings>
The magic is containing in the "packager" file. At resolve time this will be used to generate an ANT script that both downloads and extracts the required jars.
(No need to put this logic into your build.xml)
repository/ch.oscg/jreleaseinfo/1.3.0/packager.xml
<packager-module version="1.0">
<property name="name" value="${ivy.packager.module}"/>
<property name="version" value="${ivy.packager.revision}"/>
<property name="zipname" value="${name}-${version}"/>
<resource dest="archive" url="http://sourceforge.net/projects/jreleaseinfo/files/jreleaseinfo/jreleaseinfo%201.3.0/jreleaseinfo-1.3.0.zip/download" sha1="9386d92758e627d04c2480b820731fd538b13a3f" type="zip"/>
<build>
<move file="archive/${zipname}/${zipname}.jar" tofile="artifacts/jars/${name}.jar"/>
</build>
</packager-module>
To reduce the number of files I omitted the module's ivy.xml. This appears to be optional unless you want to declare it's licence and other attributes that should be present in a public repository.
I think this is very straightforward: 'ivy:retrieve' a_installer and then unzip a.j and a-lib into your lib directory (or wherever you want it). This should be easily to do with ant?
I have to wonder if there is some complication you haven't mentioned that prevents you from doing this.