Isolating cause of Erlang and RabbitMQ crashes - erlang

We have been trying to make use of the RabbitMQ Service Bus (v3.3.4) but the central bus keeps crashing. At the moment we are not using any clustering and its hosted on Windows Server 2008 R2. We'd like to isolate the root cause but the below error is the only one we can find. Can anyone shed some light on what; if anything; we can do to find the root cause of this?
Note: There are roughly 20 consumers with roughly the same number of Topic subscriptions. Also, all the clients are .NET 4.5 using the 3.3.4 Rabbit client libraries.
Version=1
EventType=APPCRASH
EventTime=130658038736577295
ReportType=2
Consent=1
ReportIdentifier=7f93ccd8-9cbe-11e4-ae00-000c29c08139
IntegratorReportIdentifier=7f93ccd7-9cbe-11e4-ae00-000c29c08139
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=erl.exe
Sig[1].Name=Application Version
Sig[1].Value=0.0.0.0
Sig[2].Name=Application Timestamp
Sig[2].Value=5343035d
Sig[3].Name=Fault Module Name
Sig[3].Value=MSVCR100.dll
Sig[4].Name=Fault Module Version
Sig[4].Value=10.0.30319.1
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=4ba220dc
Sig[6].Name=Exception Code
Sig[6].Value=40000015
Sig[7].Name=Exception Offset
Sig[7].Value=00000000000760d9
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.1.7600.2.0.0.272.7
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1033
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=8d79
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=8d79a00078e92d9c3d5d79d4324254fe
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=9af5
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=9af5b20633c279dbf44b04a614c6a1f6
UI[2]=C:\Program Files\erl6.0\erts-6.0\bin\erl.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=erl.exe stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=C:\Program Files\erl6.0\erts-6.0\bin\erl.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\system32\kernel32.dll
LoadedModule[3]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\system32\MSVCR100.dll
LoadedModule[5]=C:\Program Files\erl6.0\erts-6.0\bin\erlexec.dll
LoadedModule[6]=C:\Windows\system32\USER32.dll
LoadedModule[7]=C:\Windows\system32\GDI32.dll
LoadedModule[8]=C:\Windows\system32\LPK.dll
LoadedModule[9]=C:\Windows\system32\USP10.dll
LoadedModule[10]=C:\Windows\system32\msvcrt.dll
LoadedModule[11]=C:\Windows\system32\IMM32.DLL
LoadedModule[12]=C:\Windows\system32\MSCTF.dll
LoadedModule[13]=C:\Windows\system32\apphelp.dll
LoadedModule[14]=C:\Program Files\erl6.0\erts-6.0\bin\beam.dll
LoadedModule[15]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[16]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[17]=C:\Windows\system32\RPCRT4.dll
LoadedModule[18]=C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7600.16661_none_fa62ad231704eab7\COMCTL32.dll
LoadedModule[19]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[20]=C:\Windows\system32\COMDLG32.dll
LoadedModule[21]=C:\Windows\system32\SHELL32.dll
LoadedModule[22]=C:\Windows\system32\WS2_32.dll
LoadedModule[23]=C:\Windows\system32\NSI.dll
LoadedModule[24]=C:\Windows\system32\IPHLPAPI.DLL
LoadedModule[25]=C:\Windows\system32\WINNSI.DLL
LoadedModule[26]=C:\Windows\system32\mswsock.dll
LoadedModule[27]=C:\Windows\System32\wshtcpip.dll
LoadedModule[28]=C:\Windows\system32\NLAapi.dll
LoadedModule[29]=C:\Windows\system32\DNSAPI.dll
LoadedModule[30]=C:\Windows\System32\winrnr.dll
LoadedModule[31]=C:\Windows\system32\napinsp.dll
LoadedModule[32]=C:\Windows\System32\wship6.dll
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=erl.exe
AppPath=C:\Program Files\erl6.0\erts-6.0\bin\erl.exe

Related

PySNMP No Response Received Before Timeout--Even with Documentation Examples

I have installed PySNMP (version 4.4.6) and am attempting to call the following function (part of a larger class):
def walk(self):
#Walks OID to scavenge for information.
oids={}
for (errorIndication,
errorStatus,
errorIndex,
varBinds) in nextCmd(SnmpEngine(),
CommunityData(self.cs),
UdpTransportTarget((self.device.split(".")[0], 161), timeout=60, retries=0),
ContextData(),
ObjectType(ObjectIdentity("1.3.6.1.4.1.14179.1.2.5.5"))):
print((errorIndication,
errorStatus,
errorIndex,
varBinds))
and repeatedly receive the following output:
(RequestTimedOut('No SNMP response received before timeout',),
I have triple-checked to make sure that my community string, SNMP version, device name, port, and OID are correct. I have altered the Transport timeout variable 60 seconds with no luck. My first thought was that it might be a firewall issue, but then I tried to run the simple example included in the documentation:
from pysnmp.hlapi import *
g = getCmd(SnmpEngine(),
CommunityData('public'),
UdpTransportTarget(('demo.snmplabs.com', 161)),
ContextData(),
ObjectType(ObjectIdentity('SNMPv2-MIB', 'sysDescr', 0)))
print(next(g))
And receive the exact same output. Does anyone know what might be causing this? Unfortunately I'm on a server with few admin rights, so I can't go digging through module files changing anything. Thanks in advance for the help!
Your code looks good. For as long as you get the right hostname/IP out of self.device.split(".")[0].
May be you are not getting response from your SNMP agent for one reason or the other. I'd try public SNMP agent:
$ snmpwalk -v2c -c public demo.snmplabs.com 1.3.6
SNMPv2-MIB::sysDescr.0 = STRING: Linux zeus 4.8.6.5-smp #2 SMP Sun Nov 13 14:58:11 CDT 2016 i686
...
To make sure you do not have connectivity/firewall problems. Consider trying snmpwalk against your local SNMP agent as well. If you can't have Net-SNMP installed, consider trying snmpclitools for conducting this experiment.

MySQL connection pool in python?

I'm trying to process large amount of data using Python and maintaining processing status in MySQL. However, I'm surprised there is no standard connection pool for python-mysql (like HikariCP in Java).
I initially started with PyMySQL, things were great until the program ran for first few hours. After few hours, things started to fail. I was getting lot of errors like:
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' ([Errno 99] Cannot assign requested address)")
Moreover, lot of ports were stuck in TIME_WAIT state because I'm opening and closing connections too frequently because of lack of connection pooling
/d/p/950 ❯❯❯ netstat -nt | wc -l
84752
Per this and this, I tried to set tcp_fin_timeout and ip_local_port_range, but hardly anything improved.
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 15000 65000 > /proc/sys/net/ipv4/ip_local_port_range
Then I found out that MySQL provides mysql.connector which comes with pooling functionality. After doing all that performance actually deteriorated. More processes started to get failed. I'm using Python's multiprocessing module to simultaneously run 29 processes(multiprocessing.Pool picked this no by default) on a 24 core machine. Following was the code, of course I was using .my.cnf to pass all the credential to avoid committing them to git :
import mysql.connector
from mysql.connector import pooling
conn_pool = pooling.MySQLConnectionPool(pool_name="mypool1",
pool_size=pooling.CNX_POOL_MAXSIZE,
option_files=MYSQL_CONFIG,
option_groups=MYSQL_GROUP_NODE1,
allow_local_infile=True)
conn = conn_pool.get_connection()
Finally, reverted back to old code. Still using PyMySQL and though errors are less frequent it is still causing a major problem. I looked at SQLAlchemy and couldn't really found much of a documentation around pooling.
I'm wondering how's everyone else dealing with mysql-python connection pooling issue? I really believe there should be something out there so that I don't have to reinvent the wheel.
Any pointers are much appreciated.
DBUtils implements MySQL (and generally claims to support abritrary DB-API 2 compliant database interfaces) user-sized connection pool PooledDB, thead-mapped pool PersistentDB and SteadyDB (see functionality section). The latter should fit your case where multiprocessing.Pool creates worker processes with managed persistent database connection each. It is described as:
DBUtils.SteadyDB is a module implementing "hardened" connections to a database, based on ordinary connections made by any DB-API 2 database module. A "hardened" connection will transparently reopen upon access when it has been closed or the database connection has been lost or when it is used more often than an optional usage limit.
You can use it with PyMySQL like:
import pymysql
from DBUtils.SteadyDB import connect
db = connect(
creator = pymysql, # the rest keyword arguments belong to pymysql
user = 'guest', password = '', database = 'name',
autocommit = True, charset = 'utf8mb4',
cursorclass = pymysql.cursors.DictCursor)
Also see this related question for more examples.

configuration maxSemaphores for zuul server

I am trying to do load test for zuul version 1.1.2.
However I am keep getting following issue after few a minute for running load test.
Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: book could not acquire a semaphore for execution and no fallback available.
at com.netflix.hystrix.AbstractCommand$21.call(AbstractCommand.java:783) ~[hystrix-core-1.5.3.jar:1.5.3]
My question is how can I increase maxSemaphores via confiugration.
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds= 20000000
zuul.hystrix.command.default.execution.isolation.strategy= SEMAPHORE
zuul.hystrix.command.default.execution.isolation.semaphore.maxConcurrentRequests= 10
zuul.hystrix.command.default.fallback.isolation.semaphore.maxConcurrentRequests= 10
zuul.semaphore.maxSemaphores=3000
zuul.eureka.book.semaphore.maxSemaphore=30000
I have tried search many option on Intenet but one of those works for me
Please advise
it turns out I am using old version. For later version we could set semaphores at Zuul level. below is an example to set the maxSemaphores 3000 as default for routing to every proxied service
zuul.semaphore.maxSemaphores=3000
The actual property is max-semaphores (this would be with yaml config):
zuul:
semaphore:
#com.netflix.hystrix.exception.HystrixRuntimeException: "microservice" could not acquire a semaphore for execution and no fallback available.
max-semaphores: 2000

Equinox Weaving doesn't work in Virgo tomcat

I'm using equinox weaving to do aspect across bundles in Virgo tomcat, I followed the weaving sample. It worked fine in eclipse target platform. But when I deployed into virgo, it didn't work.
I modified config.ini like below
……
osgi.framework.extensions=org.eclipse.equinox.weaving.hook_1.0.200.I20130319-1000
osgi.bundles=org.eclipse.equinox.weaving.aspectj#2\:start,org.aspectj.weaver#2\:start,org.aspectj.runtime#2\:start
aj.weaving.verbose=true
org.aspectj.weaver.showWeaveInfo=true
org.aspectj.osgi.verbose=true
……
No exception occrred when starting Virgo, and bundles' state were correct
id State Bundle
0 ACTIVE org.eclipse.osgi_3.8.1.v20120830-144521
Fragments=1, 34
1 RESOLVED org.eclipse.equinox.weaving.hook_1.0.200.I20130319-1000
Master=0
2 ACTIVE org.eclipse.equinox.weaving.aspectj_1.0.300.I20130319-1000
3 ACTIVE org.aspectj.weaver_1.7.3.20130613144500-a
4 ACTIVE org.aspectj.runtime_1.7.3.20130613144500-a
I found some difference in consoles. When starting in eclipse target platform, next tow statements displayed in consoles
[org.eclipse.equinox.weaving.hook] info adding AspectJ hooks ...
[org.eclipse.equinox.weaving.aspectj] info Starting AspectJ weaving service ...
But when starting in Virgo, just the second one displayed. I considered maybe the hook didn't start correctly, but i'm not sure. Anyone can help me?

Work progress report in weekly basis by the users in jira

How can I view the task done/in progress in weekly basis by the users in jira??
Thanks in advance.
Your question isn't very clear, what do you mean by done/in progress ? the status of the issue? and by saying How can I view what exactly do you mean? see them from Jira? send a weekly mail?
Anyway, in case that by saying done/in progress you mean that the issue is closed/unclosed, and you are looking to the right JQL query, than:
Closed last week:
project = Development and status = Closed and updated >= "-7d"
Worked on during last week, but not closed:
project = Development and status != Closed and updated >= "-7d"
Opened last week, but not closed:
project = Development and status != Closed and created >= "-7d"
and so on.. For more queries option visit JIRA Advanced Searching. If you have more questions feel free to ask.
I wrote a simple cli tool jira-report, that queries your jira and prints weekly report to console:
$ jira-report
Jira site address: https://jira.company.com
Username for 'https://jira.company.com': admin
Password for 'https://jira.company.com':
Connecting to 'https://jira.company.com'. Pls wait...
What was [admin] doing:
Created: 2
WFM-7180 - Provide static context for log property in BasicHashAnalyzer
TST-5862 - Unable to install Nginx on HP-UX with Java 6
Resolved: 8
GSM-364 - Migration of existing scenario
WFM-5865 - NullPointerException while finding categories
TST-5864 - Some NGinx installation improvements
TST-5863 - NGinx minimal dependency
SDK-7139 - Move common interfaces and classes from into individual jar
SDK-7138 - Move common interfaces and classes from into individual dll
TST-7111 - Event.getDonotNotify doesn't indicate about agent's state
TST-6985 - TST classes should have static Log fields
Reopened: 0
Closed: 5
TST-6943 - Remove redundant org.apache.log4j dependency from common part
TST-5862 - Unable to install NGinx on HP-UX with Java 6
TST-5857 - Put back support for Jdk 1.6
TST-5840 - NGinx fails to handle interaction initiated
GSM-364 - Migration of existing units
Enjoy it!

Resources