I am setting up a Java Spark application and am following the Datastax documentation on getting started with the Java API. I've added
<dependencies>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
<version>1.1.1</version>
</dependency>
...
</dependencies>
and (a previously installed dse.jar to my local Maven repository)
<dependency>
<groupId>com.datastax</groupId>
<artifactId>dse</artifactId>
<version>version number</version>
</dependency>
. Next step in the guide is to do
SparkConf conf = DseSparkConfHelper.enrichSparkConf(new SparkConf())
.setAppName( "My application");
DseSparkContext sc = new DseSparkContext(conf);
. However, the class SparkConf can't be resolved. Should it? Am I missing some additional Maven dependency? Which?
The class is org.apache.spark.SparkConf which is in the spark-core_scala version artifact.
So your pom.xml might look like this:
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
<version>1.5.0-M2</version>
</dependency>
<dependency>
<groupId>com.datastax</groupId>
<artifactId>dse</artifactId>
<version>*version number*</version>
</dependency>
</dependencies>
The spark-core JAR is also located in:
dse_install/resources/spark/lib/spark_core_2.10-version.jar (tarball)
or:
/usr/share/dse/spark/lib/spark_core_2.10-version.jar (package installs)
Related
I was able to create docker based bitnami stand alone spark instance and run spark jobs on it. However I'm not able not able to write data to snowflake from the the spark dataframe.
I created a Dockerfile to copy the snowflake jar to the image but it still doesn't find the snowflake plugin. However if I check the jars folder the jar file is in there. I get the following error:
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 8.0 failed 4 times, most recent failure:
Lost task 0.3 in stage 8.0 (TID 10) (172.19.0.3 executor 0): java.lang.ClassNotFoundException:
net.snowflake.spark.snowflake.io.SnowflakeResultSetPartition
Here's my Dockerfile:
FROM docker.io/bitnami/spark
USER root
COPY *.jar /opt/bitnami/spark/jars
What other settings should I be setting to get it to snowflake plugin to be recognized?
Here are my maven dependencies:
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.test</groupId>
<artifactId>test-spark</artifactId>
<version>1.0.0</version>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bom</artifactId>
<version>1.11.837</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.2.1</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>net.snowflake</groupId>
<artifactId>spark-snowflake_2.12</artifactId>
<version>2.10.0-spark_3.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>3.3.1</version>
</dependency>
</dependencies>
It is a dependancy issue based on versions. You should use the spark docker related to the version you are referencing and not spark docker latest.
I configured log4j2 as described at https://quarkus.io/guides/logging, but the logs that I printed with org.apache.logging.log4j.LogManager never shows up.
I've tested using org.jboss.logging.Logger, it works. Although I can use jboss logger in this project instead of log4j2, some of the dependencies of my other base projects are using log4j2. Those logs are missing. So I hope to use log4j2 in quarkus, but not sure if it's possible.
pom dependencies
<properties>
<quarkus-plugin.version>1.9.2.Final</quarkus-plugin.version>
<quarkus.platform.group-id>io.quarkus</quarkus.platform.group-id>
<quarkus.platform.artifact-id>quarkus-universe-bom</quarkus.platform.artifact-id>
<quarkus.platform.version>1.9.2.Final</quarkus.platform.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>${quarkus.platform.group-id}</groupId>
<artifactId>${quarkus.platform.artifact-id}</artifactId>
<version>${quarkus.platform.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-resteasy</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-junit5</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>rest-assured</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-resteasy-jackson</artifactId>
</dependency>
<dependency>
<groupId>org.jboss.logmanager</groupId>
<artifactId>log4j2-jboss-logmanager</artifactId>
</dependency>
<dependency>
<groupId>com.rxnetworks.commons</groupId>
<artifactId>rxn-commons-utility</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.rxnetworks.commons</groupId>
<artifactId>rxn-commons-time</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>org.influxdb</groupId>
<artifactId>influxdb-java</artifactId>
<version>2.20</version>
</dependency>
<dependency>
<groupId>com.rxnetworks.commons</groupId>
<artifactId>rxn-commons-timeseries</artifactId>
<version>1.0.5</version>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>3.5.13</version>
<scope>test</scope>
<type>jar</type>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>28.2-jre</version>
<type>jar</type>
</dependency>
</dependencies>
Code example:
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class LogTest
{
private static final Logger LOG = LogManager.getLogger(); // With log4j2 Logger, no logs show up
// private static final org.jboss.logging.Logger LOG = org.jboss.logging.Logger.getLogger(LogTest.class); // If change to jboss Logger, it works
public void test() {
LOG.info("test log4j2");
}
}
What is the log level at which your application is?
If it is greater than INFO, the info logs will not log.
For reference, you can go through the below link.
https://logging.apache.org/log4j/log4j-2.3/manual/customloglevels.html
I am trying scdf and its monitoring with prometheus and grafana. I followed the documentation available and able to deploy the sample stream and able to see the metrics in the grafana.
I have created a stream with some custom stream app (other than the supplied rabbit mq starter apps).
Stream:
htt | participant | log
But am not able see the "participant" application metrics in gafana. But able to see the metrics of http and log apps.
Added below properties in application.properties.
management.endpoint.metrics.enabled=true
management.endpoints.web.exposure.include=*
management.endpoint.prometheus.enabled=true
management.metrics.export.prometheus.enabled=true
spring.cloud.streamapp.security.enabled=false
Added below dependencies:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<!--<!– https://mvnrepository.com/artifact/org.springframework.cloud.stream.app/app-starters-common –>-->
<!--<dependency>-->
<!--<groupId>org.springframework.cloud.stream.app</groupId>-->
<!--<artifactId>app-starters-common</artifactId>-->
<!--<version>2.1.1.RELEASE</version>-->
<!--<type>pom</type>-->
<!--</dependency>-->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-rabbit</artifactId>
</dependency>
After adding app-starters-common:org.springframework.cloud.stream.app dependency localhost:< port >/ opens a login page.
I think you need app-starters-micrometer-common dependency which adds some of the micrometer tags to your app. This dependency is intended to be used by the Spring cloud stream app starters and I believe you can use it in your custom application as well.
I came across this question while looking answers for the same.
Here is my working code snippet:-
Pom.xml
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>app-starters-micrometer-common</artifactId>
<version>2.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-jmx</artifactId>
</dependency>
application property change to enable prometheus and disable security(login page)
management.endpoints.web.exposure.include=*
management.metrics.export.prometheus.enabled=true
-- this one to remove security (login page), which was automatically added by app-starter-micrometre-common dependency.
spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration, org.springframework.boot.actuate.autoconfigure.security.servlet.ManagementWebSecurityAutoConfiguration
or
exclude the dependency, I excluded config-client and few other since I don't need them in my application.
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>app-starters-micrometer-common</artifactId>
<version>2.1.2.RELEASE</version>
<exclusions>
<exclusion>
<artifactId>spring-security-config</artifactId>
<groupId>org.springframework.security</groupId>
</exclusion>
<exclusion>
<artifactId>spring-cloud-services-starter-config-client</artifactId>
<groupId>io.pivotal.spring.cloud</groupId>
</exclusion>
<exclusion>
<artifactId>*</artifactId>
<groupId>org.springframework.boot</groupId>
</exclusion>
</exclusions>
</dependency>
With later Data Flow 2.3.x, you need to add the following dependencies to your processor:
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>app-starters-micrometer-common</artifactId>
<version>2.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer.prometheus</groupId>
<artifactId>prometheus-rsocket-spring</artifactId>
<version>0.9.0</version>
</dependency>
The app-starters-micrometer-common injects DataFlow specific tags, such as stream.name, application.name, application.type all used by the dashboard to aggregate the required metrics.
In addition you can follow instructions in the sample projects, showing how to build custom Source, Processor and Sink apps with enabled prometheus monitoring:
https://github.com/spring-cloud/spring-cloud-dataflow-samples/tree/master/monitoring-samples/stream-apps
We are trying to validate a JSON response to see if it matches a schema placed in the Eclipse ClassPath. This is designed to be a Maven project and I believe we have all the dependencies in place. But we are always getting the following error:
java.lang.NoClassDefFoundError: com/github/fge/jsonschema/main/JsonSchemaFactory
at com.jayway.restassured.module.jsv.JsonSchemaValidatorSettings.<init>(JsonSchemaValidatorSettings.java:58)
at com.jayway.restassured.module.jsv.JsonSchemaValidator$JsonSchemaValidatorFactory.createSettings(JsonSchemaValidator.java:277)
at com.jayway.restassured.module.jsv.JsonSchemaValidator$JsonSchemaValidatorFactory.create(JsonSchemaValidator.java:289)
at com.jayway.restassured.module.jsv.JsonSchemaValidator.matchesJsonSchema(JsonSchemaValidator.java:166)
at com.jayway.restassured.module.jsv.JsonSchemaValidator.matchesJsonSchemaInClasspath(JsonSchemaValidator.java:117)
at com.macys.xapi.ProductServices.TestProductServices.ohGodImDoomed(TestProductServices.java:76)
Caused by: java.lang.ClassNotFoundException: com.github.fge.jsonschema.main.JsonSchemaFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
We have the following dependencies in place:
<dependency>
<groupId>com.jayway.restassured</groupId>
<artifactId>rest-assured</artifactId>
<version>2.9.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.jayway.restassured</groupId>
<artifactId>json-schema-validator</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>com.github.fge</groupId>
<artifactId>jackson-coreutils</artifactId>
<version>1.8</version>
</dependency>
<dependency>
<groupId>com.github.fge</groupId>
<artifactId>json-schema-core</artifactId>
<version>1.2.5</version>
</dependency>
Not sure what we are missing here - can someone please help?
Vishnu, Just add following dependencies and remove all above dependencies.
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>rest-assured</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>json-schema-validator</artifactId>
<version>3.0.1</version>
</dependency>
I am using following code to try and retrieve the list of feeds from the YouTube API in a project that I build with maven.
YouTubeService service = new YouTubeService("dev","123456");
String feedUrl = "http://gdata.youtube.com/feeds/api/standardfeeds/most_viewed";
VideoFeed videofeed = service.getFeed(new URL(feedUrl), VideoFeed.class);
List<VideoEntry> videos = videofeed.getEntries();
When I do "mvn package" I get following error:
found : com.google.gdata.data.IFeed
required: com.google.gdata.data.youtube.VideoFeed
Is there anything that I am doing wrong?
My import looks like this:
import com.google.gdata.client.youtube.*;
import com.google.gdata.client.*;
import com.google.gdata.data.*;
import com.google.gdata.data.youtube.*;
import com.google.gdata.data.extensions.*;
import com.google.gdata.util.*;
import com.google.gdata.data.media.*;
import com.google.gdata.data.media.mediarss.*;
POM file. I added the artifactId using following method:
mvn install:install-file -DgroupId=youtube-12 -DartifactId=youtube-12 -Dversion=1.47.1 -Dpackaging=jar -Dfile=/usr/share/gdata/java/lib/gdata-calendar-2.0.jar
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>3.0.5.RELEASE</version>
<optional>false</optional>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>2.5</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
<version>1.2</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>youtube-1</groupId>
<artifactId>youtube-1</artifactId>
<version>1.47.1</version>
</dependency>
<dependency>
<groupId>youtube-2</groupId>
<artifactId>youtube-2</artifactId>
<version>1.47.1</version>
</dependency>
<dependency>
<groupId>youtube-3</groupId>
<artifactId>youtube-3</artifactId>
<version>1.47.1</version>
</dependency>
<dependency>
<groupId>javax.mail</groupId>
<artifactId>mail</artifactId>
<version>1.4.5</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>12.0</version>
</dependency>
</dependencies>
Any idea what I might be doing wrong? Thank you.