I need some suggestions or guidelines on Wildfly version which is in production recently.
We see heap memory spikes very often and garbage collection is not working and heap gets filled in and reaches to max 2 GB (this is our allocation of heap) and we need to restart the application to be back on normal. When we analysed the memory dump we couldn't find any issues on application code, but we have found one class is taking more memory. That class is related with to ActiveMQ & WildFly integration which application uses MDB and we are on EJB 3.0. And please more that we use Amazon Corretto with WildFly. Please any one have observed this issue regarding the below object from WildFly.
When we analysed the stack traces and objects in memory, the below object holds more memory and it is from WildFly class. Someone please share any thoughts or idea on this.
We are using wildfly-17.0.1.Final and Corretto-8.212.04.2 (build 1.8.0_212-b04).
And below is out JVM configuration
<jvms>
<jvm name="default">
<heap size="2048m" max-size="2048m"/>
<jvm-options>
<option value="-server"/>
<option value="-XX:MetaspaceSize=256m"/>
<option value="-XX:MaxMetaspaceSize=256m"/>
</jvm-options>
</jvm>
Heap stack trace as in below
**`**Who reference 1.25gb (71.7%) of byte[]?`**
1,196,563K **(65.3%) of byte[]**
Java Static org.wildfly.extension.messaging.activemq.broadcast.CommandDispatcherBroadcastEndpointFactory.BROADCAST_MANAGERS
{java.util.concurrent.ConcurrentHashMap}.values
org.wildfly.extension.messaging.activemq.broadcast.QueueBroadcastManager.broadcasts
{java.util.concurrent.LinkedBlockingDeque}**
We are running our J2EE application on Wildfly Cluster environment with the profile "full-ha" and please find our ACTIVEMQ configration
<subsystem xmlns="urn:jboss:domain:messaging-activemq:7.0">
<server name="default">
<cluster password="password"/>
<statistics enabled="${wildfly.messaging-activemq.statistics-enabled:${wildfly.statistics-enabled:false}}"/>
<security-setting name="#">
<role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
</security-setting>
<address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>
<http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
<http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
<param name="batch-delay" value="50"/>
</http-connector>
<in-vm-connector name="in-vm" server-id="0">
<param name="buffer-pooling" value="false"/>
</in-vm-connector>
<http-acceptor name="http-acceptor" http-listener="default"/>
<http-acceptor name="http-acceptor-throughput" http-listener="default">
<param name="batch-delay" value="50"/>
<param name="direct-deliver" value="false"/>
</http-acceptor>
<in-vm-acceptor name="in-vm" server-id="0">
<param name="buffer-pooling" value="false"/>
</in-vm-acceptor>
<broadcast-group name="bg-group1" jgroups-cluster="activemq-cluster" connectors="http-connector"/>
<discovery-group name="dg-group1" jgroups-cluster="activemq-cluster"/>
<cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="bg-group1"/>
<jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
<jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
<jms-queue name="CSVProcessPendingMessageBean" entries="java:/jms/queue/CSV_PROCESS_PENDING_Q java:jboss/exported/jms/queue/CSV_PROCESS_PENDING_Q"/>
<jms-queue name="CSVUploadQMessageBean" entries="java:/jms/queue/CSV_UPLOAD_Q java:jboss/exported/jms/queue/CSV_UPLOAD_Q"/>
<jms-queue name="InboundQMessageBean" entries="java:/jms/queue/CSV_INBOUND_Q java:jboss/exported/jms/queue/CSV_INBOUND_Q"/>
<jms-queue name="OutboundQMessageBean" entries="java:/jms/queue/CSV_OUTBOUND_Q java:jboss/exported/jms/queue/CSV_OUTBOUND_Q"/>
<jms-queue name="OutboundMessageBean" entries="java:/jms/queue/VFOREQUEST_Q java:jboss/exported/jms/queue/VFOREQUEST_Q"/>
<jms-queue name="AutoManualInboundMessageBean" entries="java:/jms/queue/VFOAUTOMANUALRESPOSE_Q java:jboss/exported/jms/queue/VFOAUTOMANUALRESPOSE_Q"/>
<jms-queue name="InboundMessageBean" entries="java:/jms/queue/VFORESPONSE_Q java:jboss/exported/jms/queue/VFORESPONSE_Q"/>
<connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
<connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>
<pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa"/>
</server>
</subsystem>
Had the same issue. This was the problem, because we moved to Azure which did not support multi-cast:
<broadcast-group name="bg-group1" jgroups-cluster="activemq-cluster" connectors="http-connector"/>
Related
I have dotnetcore service which is running in Kubernetes. For logging, I have used the log4net. I am trying to send the logs of service to application-insights but the logs are not showing there.
I have added the below configuration in my service for application insights.
Runtime version: netcoreapp3.1 version-2.31.0.1
Hosting environment: Azure-cluster
app-service.csproj
<ItemGroup>
<PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.16.0" />
<PackageReference Include="Microsoft.ApplicationInsights.Log4NetAppender" Version="2.16.0" />
...
<ItemGroup>
log4net.config
...
<appender name="aiAppender" type="Microsoft.ApplicationInsights.Log4NetAppender.ApplicationInsightsAppender, Microsoft.ApplicationInsights.Log4NetAppender">
<InstrumentationKey name="AppInsightsKey" value="abcdefgh-abcd-abcd-abcd-abcdefghijkl" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%message%newline"/>
</layout>
</appender>
startup.cs
public Startup(IConfiguration configuration)
{
Configuration = configuration;
Logger.ConfigureLog4Net("./logs/app.log", Configuration)
...
}
public void ConfigureServices(IServiceCollection services)
{
// The following line enables Application Insights telemetry collection.
services.AddApplicationInsightsTelemetry();
services.AddSingleton<ITelemetryInitializer, CloudRoleNameTelemetryInitializer>();
// This code adds other services for the application.
services.AddMvc();
...
}
CloudRoleNameTelemetryInitializer.cs
using System.Net;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.Extensibility;
public class CloudRoleNameTelemetryInitializer : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
// set custom role name here
telemetry.Context.Cloud.RoleName = "app-service";
}
}
I have also added the APPINSIGHTS_INSTRUMENTATIONKEY environment variable for service. I am able to see the request information for service but not logs. Only 2 logs are showing in the log section of application-insights but it is not from the application(log4net).
No XML encryptor configured. Key {XXX-XXX-XXX-XXX-XXX} may be persisted to storage in unencrypted form.
Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when the container is destroyed.
Below are some links I have explored.
https://learn.microsoft.com/en-us/azure/azure-monitor/app/asp-net-trace-logs
https://medium.com/#muralimohan.mothupally/application-insights-integration-with-log4net-e1bef68fe3c8
https://blog.ehn.nu/2014/11/using-log4net-for-application-insights/
https://jan-v.nl/post/using-application-insights-in-your-log4net-application/
But still, logs are not adding in the application insights. Is there anything I am missing here? Also is it possible to add Console.WriteLine logs to application-insights as like in nodejs app we can send it by enabling the application-insights module's configuration? Can you please help me?
Thanks...
Thanks for Peter Drier's answer, I thinks below code he provided is useful to you.
For more details, you can refer the orgin post.
Log4Net and Application Insights-no data is coming through
<appender name="aiAppender" type="Microsoft.ApplicationInsights.Log4NetAppender.ApplicationInsightsAppender, Microsoft.ApplicationInsights.Log4NetAppender">
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%message%newline" />
</layout>
<threshold value="INFO" />
<InstrumentationKey value="12345678-7946-1234-1234-8b330fbe1234" />
</appender>
Below is the sample used in log4j 1.x. I am not getting any example to convert the same in log4j2.
<appender name="CoalescingStatisticsAppender"
class="org.perf4j.log4j.AsyncCoalescingStatisticsAppender">
<!--
The TimeSlice option is used to determine the time window for which
all received StopWatch logs are aggregated to create a single
GroupedTimingStatistics log. Here we set it to 10 seconds, overriding
the default of 30000 ms
-->
<param name="TimeSlice" value="30000" />
<appender-ref ref="perf4jFileAppender" />
</appender>
The Appender won't work as is in Log4j 2. It would have to be rewritten.
You may be interested to know that Log4j 2 supports nanoTime timestamps in PatternLayout. This, in combination with the low overhead Async Loggers, allow you to use Log4j as a rough profiling tool.
I am trying to add monitoring for a hazelcast map to Nagios. Now I face the issue that the IMap entry in JMX is not available before the first usage of this map (get/set values)
Is there any option to initialize this during hazelcast startup.
The map is configured as follows
<map name="myMap">
<backup-count>1</backup-count>
<time-to-live-seconds>0</time-to-live-seconds>
<max-idle-seconds>0</max-idle-seconds>
<eviction-policy>LRU</eviction-policy>
<max-size policy="USED_HEAP_SIZE">256</max-size>
<eviction-percentage>25</eviction-percentage>
<merge-policy>com.hazelcast.map.merge.PutIfAbsentMapMergePolicy</merge-policy>
<map-store enabled="true" initial-mode="EAGER" />
</map>
I already tried to add map-store eager loading but this also did not fixed the issue:
<map name="myMap">
...
<map-store enabled="true" initial-mode="EAGER" />
</map>
The JMX path which I try to access is
com.hazelcast:instance=_hzInstance_1_dev,type=IMap,name=myMap
How can I configure the map in such a way that the JMX values are always available.
Our Webflow (2.3.1) application is claiming a lot of memory for each new flow that we open through the browser.
The screenshot below shows our application's memory use. When the application starts it takes an initial 400 Mb. After that we open 4 individual, identical Webflow TEST pages in the browser which each claim about 90Mb of extra memory..
Each test page is started from its own simple flow definition:
<?xml version="1.0" encoding="UTF-8"?>
<flow xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.springframework.org/schema/webflow"
xsi:schemaLocation="http://www.springframework.org/schema/webflow
http://www.springframework.org/schema/webflow/spring-webflow-2.0.xsd" start-state="start">
<view-state id="start" view="test/test1">
</view-state>
<end-state id="end"/>
<bean-import resource="../flow-beans.xml"/>
</flow>
The JSP test pages are also very simple, just empty with one line of text.
When we currently set the JVM memory to 1.5Gb the application eventually crashes on the server with OutOfMemoryExceptions after opening about 15 different flows. 1.5 Gb seems a bit much, regarding the low complexity of our screens..
We are wondering if the amount of memory Webflow seems to claim for these simple flows/pages is expected and if we should therefore just assign more memory to the server JVM. If not, we would like to know how we can decrease this memory usage.
Below is our entire webflow configuration.
We have tried adding a flow-execution-repository tag and played around with the max-executions-snapshots and max-executions values, but even the most conservative settings don't change the memory usage we are seeing.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:webflow="http://www.springframework.org/schema/webflow-config"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/webflow-config
http://www.springframework.org/schema/webflow-config/spring-webflow-config-2.3.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx-3.0.xsd">
<!-- Launches new flow executions and resumes existing executions. -->
<webflow:flow-executor id="flowExecutor" flow-registry="flowRegistry">
</webflow:flow-executor>
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="classpath:our.properties" />
<property name="placeholderPrefix" value="$xxxx"></property>
</bean>
<tx:annotation-driven transaction-manager="$xxxx{txManager}" />
<!-- Creates the registry of flow definitions for this application -->
<webflow:flow-registry id="flowRegistry" flow-builder-services="flowBuilderServices">
<webflow:flow-location-pattern value="classpath:flows/**/*-flow.xml" />
</webflow:flow-registry>
<bean id="viewFactoryCreator" class="org.springframework.webflow.mvc.builder.MvcViewFactoryCreator">
<property name="viewResolvers" ref="viewResolver" />
</bean>
<bean id="expressionParser" class="org.springframework.expression.spel.standard.SpelExpressionParser">
<constructor-arg name="configuration">
<bean class="org.springframework.expression.spel.SpelParserConfiguration">
<constructor-arg name="autoGrowCollections" value="true" />
<constructor-arg name="autoGrowNullReferences" value="false" />
</bean>
</constructor-arg>
</bean>
<bean id="webflowExpressionParser" class="org.springframework.webflow.expression.spel.WebFlowSpringELExpressionParser">
<constructor-arg name="expressionParser" ref="expressionParser" />
</bean>
<webflow:flow-builder-services id="flowBuilderServices" view-factory-creator="viewFactoryCreator" validator="validator" expression-parser="webflowExpressionParser"/>
<bean id="validator" class="org.springframework.validation.beanvalidation.LocalValidatorFactoryBean" />
<bean id="projectVersion" class="our.company.versions.ProjectVersionUtil">
<property name="xxxxVersion" value="$xxxx{xxxx.version}" />
<property name="systemConfigurationDao">
<ref bean="SystemConfigurationDao"/>
</property>
</bean>
</beans>
When Spring Web Flow starts a new flow it basically constructs a new BeanFactory which loads the xml file and imports any additional xml files. The newly constructed BeanFactory has the context of the DispatcherServlet as its parent.
Now the problem with this is that a the bean factory constructs instances of all the beans even those defined in imported XML files.
<bean-import resource="../flow-beans.xml"/>
If there are a lot of beans in there those will be duplicated for each flow instance. In general you don't want your all of your beans duplicated and stored in the users sessions.
Remove the singleton beans from the flow-beans.xml and put them in the normal application context, they are still referable from within the flow definition. Or you could simply add the flow-beans.xml to the list of files loaded at startup of your application.
I've used Log4Net in multiple applications for a while. It has been working fine, but recently I noticed that the applications were suddenly not logging anymore. Turns out this issue is the same for all my applications, and they all suddenly stopped logging some months ago. The strange thing is that the logging works when I access the applications directly on the server (http://localhost/myApp), while nothing is logged when I access the application from another PC. My first thought was that it must be related to file/folder permissions, but allowing "Everyone" (Windows user group) full access to the log folder did not help.
They are all ASP.Net MVC 4 applications running on IIS7 (Windows 2008 R2 Enterprise OS), and the application pool is using "ApplicationPoolIdentity". Log4Net version is 1.2.10.0 and I am using a custom CompositeRollingFileAppender. I thought it may have been something wrong with the custom appender, but the problem remained the same when I tried switching to the standard RollingFileAppender. I've seen the issue on multiple servers.
Has anyone seen something similar? Please share your thoughts, as I cannot see why there should be any difference in accessing the applications locally or remotly.
Here is the log4net section in one of my applications' web.config:
<log4net>
<appender name="RollingFileAppender" type="[mynamespace].CompositeRollingFileAppender">
<file value="Logs/ApplicationLog.log" />
<appendToFile value="true" />
<rollingStyle value="Composite" />
<maxSizeRollBackups value="20" />
<maximumFileSize value="10MB" />
<datePattern value="_yyyy-MM-dd" />
<staticLogFileName value="true" />
<preserveLogFileNameExtension value="true" />
<layout type="log4net.Layout.PatternLayout,log4net">
<param name="ConversionPattern" value="%utcdate;%property{ErrorCode};%property{Severity};%property{ErrorName};%property{Module};%m%n" />
</layout>
</appender>
<root>
<priority value="ALL" />
<appender-ref ref="RollingFileAppender" />
<level value="Warn" />
</root>
<logger name="NHibernate">
<level value="OFF" />
</logger>
<logger name="NHibernate.SQL">
<level value="OFF" />
</logger>
</log4net>
Turns out the issue was not directly related to log4net, but to the way ASP.Net MVC3 and newer handles exceptions by default. Some months ago we updated our applications from MVC2 to MVC4, and because of this code which is executed by default from global.asax.cs.Application_Start(), it "bypassed" our exception handling module when CustomErrors where set to RemoteOnly or On:
public class FilterConfig
{
public static void RegisterGlobalFilters(GlobalFilterCollection filters)
{
filters.Add(new HandleErrorAttribute());
}
}
This thread put me on the right track. I ended up removing the filters.Add(..) line, and now it seems to be working fine!