Cannot set pagecache size with embedded Neo4J/Gremlin - neo4j

I am writing a Java program that uses an embedded Neo4j graph with TinkerPop. Here's the relevant section of my pom
<dependency>
<groupId>org.apache.tinkerpop</groupId>
<artifactId>neo4j-gremlin</artifactId>
<version>3.3.1</version>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-tinkerpop-api-impl</artifactId>
<version>0.7-3.2.3</version>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-tinkerpop-api</artifactId>
<version>0.1</version>
</dependency>
I want to add a configuration option to set the page cache size when initializing the graph. My PS Old Gen heap space is filling up.
Currently, I'm opening the Neo4j graph by passing it a org.apache.commons.configuration.Configuration object. I’m trying to set two properties, the directory and the pagecache. When I run my program, the "gremlin.neo4j.directory" property is processed, but the "dbms.memory.pagecache.size" is not, according to the graph's log file. The log file's first line is this:
2019-03-20 14:38:36.155+0000 WARN [o.n.i.p.PageCache] The
dbms.memory.pagecache.size setting has not been configured. It is
recommended that this setting is always explicitly configured, to
ensure the system has a balanced configuration. Until then, a computed
heuristic value of 8310519808 bytes will be used instead.
Using jvisualvm and jconsole, I can see that the memory in the PS Old gen is filling up with objects related to page caching so I'm trying to throttle how much data is cached by Neo4j.
Here's my code:
Configuration configuration = new BaseConfiguration();
configuration.addProperty("gremlin.neo4j.directory", "tmp/mygraph");
configuration.addProperty("dbms.memory.pagecache.size", "500m");
myGraph = Neo4jGraph.open(configuration);
Any idea what I'm doing wrong?

I think that you need to prefix your configuration keys that are Neo4j specific with gremlin.neo4j.conf thus:
Configuration configuration = new BaseConfiguration();
configuration.addProperty("gremlin.neo4j.directory", "tmp/mygraph");
configuration.addProperty("gremlin.neo4j.conf.dbms.memory.pagecache.size", "500m");
myGraph = Neo4jGraph.open(configuration);

Related

Add jaeger trace id and span id to log4j2 logs

I want to inject x-b3-traceid and x-b3-spanid in logs with pattern as shown-
property name="PATTERN" value="%h %l %u [%date{dd/MMM/yyyy:HH:mm:ss.SSS}] "%r" %s %b "%i{Referer}" "%i{User-Agent}" [trace=%responseHeader{X-B3-TraceId},span=%i{X-B3-SpanId}] %D"
For zipkins, there are libraries available like
brave-context-log4j2 –
(https://github.com/openzipkin/brave/tree/master/context/log4j2)
Spring cloud sleuth. (https://cloud.spring.io/spring-cloud-sleuth/)
How can I add that while using jaeger?
The best way to move forward in order to use Jaegar is NOT TO USE JAEGAR CLIENT! Jaegar has the ability to collect Zipkin spans.
https://www.jaegertracing.io/docs/1.8/getting-started/#migrating-from-zipkin
You should take advantage of this and use the below Sleuth+Zipkin dependency and exclude Jaegar agent jars in your spring boot app.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>
The above will send Zipkin spans to http://localhost:9411 by default. You can override this in your Spring Boot app to point to your Jaegar server easily by overriding the zipkin base URL.
spring.zipkin.base-url=http://your-jaegar-server:9411
Sleuth will do all the heavy lifting and the default logging will log the span and traceIds.
In the log4j2.xml file, all you have to mention is
[%X]
I'll be uploading a working example of this approach into my GitHub and sharing the link.
EDIT 1:
You can find the sample code here:
https://github.com/anoophp777/spring-webflux-jaegar-log4j2

How to override 'org.apache.cxf.stax.maxChildElements' property value inside a TomEE container?

I've got a JAX-WS web service endpoint configured purely via annotations running in TomEE 7 environment. Basically, the method being called has to return a List<String> of all node names contained in a graph data structure. The response of such a request can contain more thank 50k elements.
With CXF 2.6.x this worked fine. However, when I call the WS-method under CXF 3.x (bundled in TomEE 7.x), the following exception is thrown on the server side:
org.apache.cxf.interceptor.Fault: Unmarshalling Error: Maximum Number of Child Elements limit (50000) Exceeded
at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:906)
at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:712)
at org.apache.cxf.jaxb.io.DataReaderImpl.read(DataReaderImpl.java:179)
at org.apache.cxf.wsdl.interceptors.DocLiteralInInterceptor.handleMessage(DocLiteralInInterceptor.java:109)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:801)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1680)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1559)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1356)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:653)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:514)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:423)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:324)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:277)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:139)
at com.sun.proxy.$Proxy51.getAllNodeNames(Unknown Source)
Caused by: javax.xml.bind.UnmarshalException
- with linked exception:
[javax.xml.stream.XMLStreamException: Maximum Number of Child Elements limit (50000) Exceeded]
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.handleStreamException(UnmarshallerImpl.java:485)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:417)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:394)
at org.apache.cxf.jaxb.JAXBEncoderDecoder.doUnmarshal(JAXBEncoderDecoder.java:855)
at org.apache.cxf.jaxb.JAXBEncoderDecoder.access$100(JAXBEncoderDecoder.java:102)
at org.apache.cxf.jaxb.JAXBEncoderDecoder$2.run(JAXBEncoderDecoder.java:894)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:892)
... 21 more
Caused by: javax.xml.stream.XMLStreamException: Maximum Number of Child Elements limit (50000) Exceeded
at com.ctc.wstx.sr.InputElementStack.push(InputElementStack.java:340)
at com.ctc.wstx.sr.BasicStreamReader.handleStartElem(BasicStreamReader.java:2951)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2839)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1073)
at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:196)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:415)
... 27 more
Error: Maximum Number of Child Elements limit (50000) Exceeded
So far, I've read the official CXF documentation on this issue, checked a HowTo at the TomEE website and read many related, yet older posts in forums.
I tried to set the properties - as advised by the TomEE documentation - via openejb-jar.xml in the webservice's WEB-INF folder:
<?xml version="1.0" encoding="UTF-8"?>
<openejb-jar>
<ejb-deployment ejb-name="MyWebService">
<properties>
org.apache.cxf.stax.maxChildElements = 100000
</properties>
</ejb-deployment>
</openejb-jar>
I also tried with the shorter property cxf.stax.maxChildElements to check whether this would be accepted, yet without success.
For testing/debugging, I start the TomEE instance via the tomee-maven-plugin, Therefore, I tried the set the maxChildElement property as an environment property like so:
<plugin>
<groupId>org.apache.tomee.maven</groupId>
<artifactId>tomee-maven-plugin</artifactId>
<version>${tomee.plugin.version}</version>
<configuration>
<tomeeVersion>${tomee.version}</tomeeVersion>
<tomeeClassifier>plus</tomeeClassifier>
<debug>false</debug>
<tomeeHttpPort>8181</tomeeHttpPort>
<debugPort>5005</debugPort>
<args>-Dfoo=bar</args>
<skipCurrentProject>true</skipCurrentProject>
<webapps>
<webapp>my.ws:${webservice.artifact.name}:${webservice.artifact.version}?name=ws-endpoint</webapp>
</webapps>
<libs>
<!-- Third party libraries needed in the global lib folder of TomEE -->
<lib>log4j:log4j:${log4j.version}</lib>
</libs>
<systemVariables>
<!--
special property needed to allow for more childElements in StAX Parser
-->
<org.apache.cxf.stax.maxChildElement>100000</org.apache.cxf.stax.maxChildElement>
</systemVariables>
</configuration>
</plugin>
Sadly, it has no effect on the runtime configuration of CXF/StAX (Woodstox).
Question
How can we override the maxChildElements property via a configuration in openejb-jar.xml or as an external property at TomEE startup.
Finally, I got it working with the help of Romain Manni-Bucau (credits to him for pointing me into the right direction). Yet, his original answer is not the final solution. Therefore, I give the working configuration here.
1.) Put the following openejb-jar.xml to the WEB-INF folder:
<?xml version="1.0" encoding="UTF-8"?>
<openejb-jar>
<ejb-deployment ejb-name="MyWebService">
<properties>
cxf.jaxws.properties = cxfConfig
</properties>
</ejb-deployment>
</openejb-jar>
2.) Provide a new (or add to an existing) resources.xml file, again via WEB-INF:
<?xml version="1.0" encoding="UTF-8"?>
<resources>
<Service id="cxfConfig" class-name="org.apache.openejb.config.sys.MapFactory" factory-name="create">
org.apache.cxf.stax.maxChildElements = 100000
</Service>
</resources>
Note well the configuration link via the MapFactory object with the id cxfConfig.
3.) Configure JAX-WS clients to set corresponding property as well. For instance, given a Spring client, this can be configured like so:
<bean id="wsClientProxy" class="org.apache.cxf.jaxws.JaxWsProxyFactoryBean">
<property name="serviceClass" value="com.acme.ws.jaxb.MyWebservice"/>
<property name="address" value="${ws.endpoint.url}"/>
<property name="properties">
<map>
<entry key="org.apache.cxf.stax.maxChildElements" value="100000" />
</map>
</property>
</bean>
In general, this might also be useful for people trying to set other CXF-related properties as listed in the XML section of the CXF security guideline, in particular to increase or decrease conservative default values.
I tested the above configuration steps successfully under a TomEE 7.0.3 and 8.0.9 environment, yet this should also reliably work with all 7.0.x and 8.0.x releases.
For other use cases, this blog post by Romain might also be worth reading, as it covers basic configuration concepts quite well.
Hope this helps others.
You can try
-Dorg.apache.cxf.stax.maxChildElements=100000
It should also work
I think you need to define a resources.xml with a Service of type (class-name) java.util.Properties and the properties inside:
openejb-jar.xml would get this property:
cxf.jaxws.properties = cxfConfig
resources.xml would get
org.apache.cxf.stax.maxChildElements = 1
This test does it programmatically: https://github.com/apache/tomee/blob/master/server/openejb-cxf/src/test/java/org/apache/openejb/server/cxf/MaxChildTest.java

Collect dependency information from pom.xml

I'd like to print a file containing listing of all the dependencies and their versions in my pom.xml. (no need to go inside of each dependency)
My ultimate goal is to have the dependency + version information listing in a file that can be read by the application at runtime and be displayed via "version info" link on a web page.
I found out there is a maven dependecy plugin | dependency:list, which I understand, should be doing pretty much what I want. I also manage it to print an output file, but it contains mainly gibberish. What I can make sense of it, it just has some kind of listing of the packages in the project. (no version info)
Could someone please clarify how to use this plugin correctly, or if it even does what I need it to do.
My configuration is same as in their usage instructions, but if try to use any optional parameters it fails always.
This looks like it is doing the right thing now. A version file is printed in 'target/classes' and I can find it when looking from the app at runtime.
In pom.xml (in build/plugins):
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.10</version>
<executions>
<execution>
<id>createVersionInfo</id>
<phase>compile</phase>
<goals>
<goal>list</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>com.company.project</groupId>
<artifactId>app-name</artifactId>
<version>${project.version}</version>
<type>war</type>
<overWrite>true</overWrite>
</artifactItem>
</artifactItems>
<sort>true</sort>
<includeScope>compile</includeScope>
<outputFile>target/classes/dependency_versions.txt</outputFile>
</configuration>
</execution>
</executions>
</plugin>
The '<excludeScope>test</excludeScope>' does not work as supposed. It excludes everything. I think the explanation for includeScope explains this somehow.
An Empty string indicates all scopes (default).
The scopes being interpreted are the scopes as Maven sees them,
not as specified in the pom. In summary:
runtime - scope gives runtime and compile dependencies,
compile - scope gives compile, provided, and system dependencies,
test - (default) scope gives all dependencies,
provided - scope just gives provided dependencies,
system - scope just gives system dependencies.
And this piece of code on the java side will do pretty much what is needed:
InputStream in = this.getClass().getResourceAsStream("/dependency_versions.txt"); // Finds the file from classpath
String content = "";
// Java7 AutoCloseable
try (Scanner scan = new Scanner(in)) {
scan.useDelimiter("\\Z"); // Found this from another SO-thread, enables to read in the whole file at once
content = scan.next();
}
// Iterate the content and collect the information
// \n = line change
for (String row : content.split("\n")) {
row = row.trim();
// Basically removes the 1st two lines of the file that are not wanted
if (StringUtils.isEmpty(row) || row.endsWith(":")) {
continue;
}
StringTokenizer tokenizer = new StringTokenizer(row, ":");
String package = tokenizer.nextToken();
String dependency = tokenizer.nextToken();
String type = tokenizer.nextToken();
String version = tokenizer.nextToken();
String scope = tokenizer.nextToken();
}
I am actually populating a list of my own TO's there for showing the info in JSF, but I omit that part for simplicity/readability.
... Actually made me think why do I want to read in the whole file at once and not iterate it and read it one row at a time, but I'll leave it for the daily WTF. ;)

"Can not initialize the default wsdl from..." -- Why?

My pom.xml contains the following to auto generate a client for a working web service having the WSDL specified below:
<plugin>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-codegen-plugin</artifactId>
<version>2.3.1</version>
<executions>
<execution>
<id>generate-sources</id>
<configuration>
<sourceRoot>${basedir}/target/generated/src/main/java</sourceRoot>
<wsdlOptions>
<wsdlOption>
<wsdl>${basedir}/src/main/wsdl/myclient.wsdl</wsdl>
<extraargs>
<extraarg>-client</extraarg>
<extraarg>-verbose</extraarg>
</extraargs>
<wsdlLocation>wsdl/myclient.wsdl</wsdlLocation>
</wsdlOption>
</wsdlOptions>
</configuration>
<goals>
<goal>wsdl2java</goal>
</goals>
</execution>
</executions>
</plugin>
The project builds fine, without any errors or warnings and I can see the the file myclient.wsdl in the JAR file right under a wsdl folder.
But when I try running that JAR:
java -Xmx1028m -jar myclient-jar-with-dependencies.jar
It complains that "Can not initialize the default wsdl from wsdl/myclient.wsdl"
Why?
What am I missing?
How can I find out what path that wsdl/myclient.wsdl in pom.xml translates into, that makes the client's JAR complain at run time?
Update: I am aware of some solutions/workarounds that involve modifying the auto-generated code:
Pass "null" for the wsdl URL and then use the ((BindingProvider)port).getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, "http://example.com/....") to set the address.
load the WSDL as a Java resource and pass its location into your service's constructor.
But I am more interested in a solution that requires entering the right values into the pom.xml like the classpath approach (but unfortunately classpath didn't work for me for some reason).
Any ideas what I should be typing there instead? Apparently this is a very simply case of figuring out the correct path rules for that particular plugin, but I am missing something and I don't know what it is.
The error comes from the static initializer of your generated service class (which is annotated by #WebServiceClient). It tries to load the wsdl file as resource. The generator uses the value which you have provided by the parameter wsdlLocation. You should leave away the "wsdl/" prefix:
<wsdlLocation>myclient.wsdl</wsdlLocation>
because the wsdl is located directly in the root of the classpath folder.
BTW: If you omit the parameter <wsdlLocation> the value of the param <wsdl> is used (which is not correct at runtime in your case, but would be correct if the provided URL would be a remote URL address, i.e. directly fetched from the webservice server).
BTW2: Your workaround 2 is in fact +/- what the generated code of the service class does if you use the parameterless constructor.
I notice the cfx examples use slightly different locations for sourceRoot, wsdl and wsdlLocation.
Remember that typically, files in src/main/resources are included in the produced artifact. In order for files in src/main/wsdl to be included, it needs to be added as a resource in the pom.xml:
<resources>
<resource>
<directory>src/main/wsdl</directory>
</resource>
</resources>
Tips:
Set the paths you suspect to known bad paths and see if you get the same error-message.
Unzip the produced *.jar-file(s) and check if the wsdl is included, and what the path is.

not able to see data in stax Events while debugging

I Started working on Stax Parser for past three months. I used to see data or text in the stax events while debugging. This used to help me a lot while working on my task. But from past 2days,there is weird behavior. When i debugged the project, i can only see events like this...[Stax Event #1], [Stax Event #4], [Stax Event #1], [Stax Event #4]
This is giving me hard time debugging. I am woodStox stax and java 1.6.
These are dependencies i am using
<dependency>
<groupId>javax.xml</groupId>
<artifactId>jsr173</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.codehaus.woodstox</groupId>
<artifactId>wstx-asl</artifactId>
<version>4.0.6</version>
</dependency>
<dependency>
<groupId>stax</groupId>
<artifactId>stax-api</artifactId>
<version>1.0.1</version>
</dependency>
<dependency>
<groupId>com.sun.xml.stream</groupId>
<artifactId>sjsxp</artifactId>
<version>1.0.2</version>
</dependency>
Do i need to change my settings to get back to normal behavior.
You have two StAX implementations: sjsxp and woodstox, so it's kind of random which one is actually used. Most likel you'll want to remove the dependency to sjsxp.
You also have two StAX APIs: jsr173 and stax-api. Definitely avoid the former, it's buggy! With Java 6 or later you may/should also remove the latter.
What code do you use to print output statements? Stax API always allows you to access any data events have; but it may not work by simply doing event.toString().

Resources