While debugging a dropped channel_layer message issue I turned up daphne verbosity and occasionally see the warnings like the ones below in the log file. Real issues or can I ignore them?
[ec2-user#ip-aaa-bb-c-dd ~]$ grep SyncToAsync /var/log/daphne/error_log | head
2018-11-16 11:51:03,898 daphne.server WARNING /opt/python/run/venv/local/lib/python3.6/site-packages/channels_redis/core.py:282: builtins.RuntimeWarning: coroutine 'SyncToAsync.__call__' was never awaited
2018-11-16 11:51:17,025 daphne.server WARNING /usr/lib64/python3.6/logging/__init__.py:277: builtins.RuntimeWarning: coroutine 'SyncToAsync.__call__' was never awaited
2018-11-16 11:51:28,523 daphne.server WARNING /usr/lib64/python3.6/asyncio/selector_events.py:316: builtins.RuntimeWarning: coroutine 'SyncToAsync.__call__' was never awaited
2018-11-16 11:51:40,559 daphne.server WARNING /usr/lib64/python3.6/selectors.py:191: builtins.RuntimeWarning: coroutine 'SyncToAsync.__call__' was never awaited
2018-11-16 11:51:51,824 daphne.server WARNING /usr/lib64/python3.6/asyncio/tasks.py:313: builtins.RuntimeWarning: coroutine 'SyncToAsync.__call__' was never awaited
2018-11-16 11:52:03,066 daphne.server WARNING /opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/query.py:862: builtins.RuntimeWarning: coroutine 'SyncToAsync.__call__' was never awaited
2018-11-16 11:53:20,962 daphne.server WARNING /opt/python/run/venv/local/lib64/python3.6/site-packages/twisted/web/http.py:244: builtins.RuntimeWarning: coroutine 'SyncToAsync.__call__' was never awaited
2018-11-16 11:56:51,245 daphne.server WARNING /opt/python/run/venv/local/lib64/python3.6/site-packages/twisted/web/http.py:2994: builtins.RuntimeWarning: coroutine 'SyncToAsync.__call__' was never awaited
2018-11-16 11:57:16,160 daphne.server WARNING /usr/lib64/python3.6/threading.py:347: builtins.RuntimeWarning: coroutine 'SyncToAsync.__call__' was never awaited
2018-11-16 11:57:21,295 daphne.server WARNING /opt/python/run/venv/local/lib64/python3.6/site-packages/twisted/internet/tcp.py:196: builtins.RuntimeWarning: coroutine 'SyncToAsync.__call__' was never awaited
For reference, here are the versions I'm using:
aioredis==1.2.0
asgiref==2.3.2
async-timeout==3.0.1
attrs==18.2.0
autobahn==18.11.1
Automat==0.7.0
channels==2.1.5
channels-redis==2.3.1
constantly==15.1.0
daphne==2.2.3
Django==2.1.3
django-health-check==3.8.0
django-redis-cache==1.8.0
hiredis==0.2.0
hyperlink==18.0.0
idna==2.7
incremental==17.5.0
msgpack==0.5.6
mysqlclient==1.3.13
PyHamcrest==1.9.0
pytz==2018.7
redis==2.10.6
six==1.11.0
Twisted==18.9.0
txaio==18.8.1
zope.interface==4.6.0
This is all running on AWS/EBS with ElastiCache providing redis. I'm happy to provide more detail if it's useful and/or worth investigating more.
Thanks,
Kevin
Yes, the warnings are real issues alright, but with my buggy code.
I was missing some async_to_sync wrappers in my WebsocketConsumer derived classes, specifically for channels.auth.get_user() which is decorated with #database_sync_to_async.
Related
Almost done upgrading to grails 5. I need to support grails 4 for 1 more month or so. Unfortunately Grails 4 can't get past the compileGroovy . Depending on increase of jvm settings (which I have tried very high values out of curiosity), it either
errors out with GC limit exceeded,
it hangs after a certain period of time. CPU usage drops off and it basically idles endlessly.
This is odd since the last commit was very light. I can control failure or success by removing a single domain class.
debug from GC error:
14:27:38.336 [DEBUG]
[org.gradle.api.internal.classloading.ClassInfoCleaningGroovySystemLoader] Removed ClassInfo from org.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry loaded by VisitableURLClassLoader(api-groovy-compile-loader)
14:27:40.570 [LIFECYCLE] [org.gradle.process.internal.health.memory.MemoryManager]
14:27:40.570 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting OS memory status event {Total: 16719904768, Free: 9322885120}
14:27:40.570 [DEBUG] [org.gradle.launcher.daemon.server.health.LowMemoryDaemonExpirationStrategy] Received memory status update: {Total: 16719904768, Free: 9322885120}
14:27:40.570 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting JVM
memory status event {Maximum: 1908932608, Committed: 571473920}
14:27:39.421 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger]
14:27:39.421 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :compileGroovy FAILED
14:27:40.914 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Completing Build operation 'compiler daemon'
14:27:40.914 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Build operation 'compiler daemon' completed
14:27:40.914 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Completing Build operation 'Execute compile for :compileGroovy'
14:27:40.914 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Build operation 'Execute compile for :compileGroovy' completed
14:27:40.915 [DEBUG] [org.gradle.api.internal.changedetection.changes.TaskFingerprintUtil] Fingerprinting property destinationDir (Output) for task ':compileGroovy'
14:27:40.928 [DEBUG] [org.gradle.internal.execution.impl.steps.CacheStep] Not storing result of task ':compileGroovy' in cache because the execution failed
Endless cyclical message when it hangs is just an emission of a memory update event
14:41:54.361 [DEBUG] [org.gradle.api.internal.classloading.ClassInfoCleaningGroovySystemLoader] Removed ClassInfo from [Ljava.beans.PropertyDescriptor; loaded by null
14:41:54.361 [DEBUG] [org.gradle.api.internal.classloading.ClassInfoCleaningGroovySystemLoader] Removed ClassInfo from org.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry loaded by VisitableURLClassLoader(api-groovy-compile-loader)
14:41:56.633 [LIFECYCLE] [org.gradle.process.internal.health.memory.MemoryManager]
14:41:56.633 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting OS memory status event {Total: 16719904768, Free: 8231411712}
14:41:56.633 [DEBUG] [org.gradle.launcher.daemon.server.health.LowMemoryDaemonExpirationStrategy] Received memory status update: {Total: 16719904768, Free: 8231411712}
14:41:56.633 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting JVM memory status event {Maximum: 7635730432, Committed: 492306432}
14:42:01.633 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting OS memory status event {Total: 16719904768, Free: 8210542592}
14:42:01.633 [DEBUG] [org.gradle.launcher.daemon.server.health.LowMemoryDaemonExpirationStrategy] Received memory status update: {Total: 16719904768, Free: 8210542592}
14:42:01.633 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting JVM memory status event {Maximum: 7635730432, Committed: 492306432}
//repeated endlessly
[org.gradle.launcher.daemon.server.health.LowMemoryDaemonExpirationStrategy] Received memory status update: {Total: 16719904768, Free: 8210542592}
14:42:01.633 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting JVM memory status event {Maximum: 7635730432, Committed: 492306432}
Using grails 4.1.2, java 8.0.302. Anyone encountered an issue like this with big grails projects? Not an issue with grails 5.
The logs for my Cloud Dataflow job contain multiple messages of the form:
[GC (Allocation Failure) [PSYoungGen: 72352K->160K(71168K)]
120584K->48392K(158720K), 0.0022739 secs] [Times: user=0.00 sys=0.00,
real=0.00 secs]
What does this mean? What should I do about it?
The "Allocation Failure" message is a completely normal part of Java memory management; it just means the JVM has run out of memory and needs to trigger a GC. Unless there is already a reason to suspect memory problems, the "Java GC" log is generally not worth looking at for Dataflow pipelines.
Please see Java GC (Allocation Failure) for additional information.
I have a problem with running automation tests on iOS platform.
Here is an error I get i logs. Will be very grateful for any solution.
Here is a error.
debug] [XCUITest] Waiting for WebDriverAgent server to finish loading...
[debug] [WebDriverAgent] Device: Nov 7 10:44:00 iPhone5cGreen XCTRunner[226] <Notice>: Running tests...
[debug] [XCUITest] Waiting for WebDriverAgent server to finish loading...
[Xcode] 2016-11-07 10:44:02.389 xcodebuild[4398:148129] Error Domain=IDETestOperationsObserverErrorDomain Code=5 "Early unexpected exit, operation never finished bootstrapping - no restart will be attempted" UserInfo={NSLocalizedDescription=Early unexpected exit, operation never finished bootstrapping - no restart will be attempted}
[Xcode]
Testing failed:
[Xcode] Test target WebDriverAgentRunner encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted)
** TEST FAILED **
[XCUITest] xcodebuild exited with code '0' and signal 'null'
I am following the instructions in https://github.com/mapbox/mapbox-android-sdk#building-from-source and am currently using Android Studio. I used to use Eclipse for this Android project and I just plain gave up after spending all of yesterday trying to make integrating mapbox-android-sdk to work with my android project in Eclipse. I wish the instructions were clearer on mapbox-android-sdk--I am delighted to help them if I knew how.
I imported my old Android project into Android Studio, and then went into File->Import Project and chose the build.gradle inside MapboxAndroidSDK in the directory where I downloaded the git source tree in my system.
Then, while my android project needing maps supported was selected in Android Studio, I went to File->Import Module in Android Studio to add the *.aar file that was generated in the MapboxAndroidSDK directory.
Anyway, when I now run my app, I get the following errors. Any help is appreciated.
09-29 11:30:07.061 1111-1111/com.username.someapp D/﹕ HostConnection::get() New Host Connection established 0xb7b6bdb8, tid 1111
09-29 11:30:07.091 1111-1111/com.username.someapp W/EGL_emulation﹕ eglSurfaceAttrib not implemented
09-29 11:30:07.101 1111-1111/com.username.someapp D/OpenGLRenderer﹕ Enabling debug mode 0
09-29 11:30:16.431 1111-1111/com.username.someapp D/dalvikvm﹕ GC_FOR_ALLOC freed 104K, 5% free 3282K/3448K, paused 20ms, total 24ms
09-29 11:30:16.481 1111-1111/com.username.someapp W/dalvikvm﹕ VFY: unable to find class referenced in signature (Ljava/nio/file/Path;)
09-29 11:30:16.481 1111-1111/com.username.someapp W/dalvikvm﹕ VFY: unable to find class referenced in signature ([Ljava/nio/file/OpenOption;)
09-29 11:30:16.491 1111-1111/com.username.someapp I/dalvikvm﹕ Could not find method java.nio.file.Files.newOutputStream, referenced from method okio.Okio.sink
09-29 11:30:16.491 1111-1111/com.username.someapp W/dalvikvm﹕ VFY: unable to resolve static method 10776: Ljava/nio/file/Files;.newOutputStream (Ljava/nio/file/Path;[Ljava/nio/file/OpenOption;)Ljava/io/OutputStrea
m;
09-29 11:30:16.491 1111-1111/com.username.someapp D/dalvikvm﹕ VFY: replacing opcode 0x71 at 0x000a
09-29 11:30:16.491 1111-1111/com.username.someapp W/dalvikvm﹕ VFY: unable to find class referenced in signature (Ljava/nio/file/Path;)
09-29 11:30:16.491 1111-1111/com.username.someapp W/dalvikvm﹕ VFY: unable to find class referenced in signature ([Ljava/nio/file/OpenOption;)
09-29 11:30:16.501 1111-1111/com.username.someapp I/dalvikvm﹕ Could not find method java.nio.file.Files.newInputStream, referenced from method okio.Okio.source
09-29 11:30:16.501 1111-1111/com.username.someapp W/dalvikvm﹕ VFY: unable to resolve static method 10775: Ljava/nio/file/Files;.newInputStream (Ljava/nio/file/Path;[Ljava/nio/file/OpenOption;)Ljava/io/InputStream;
09-29 11:30:16.501 1111-1111/com.username.someapp D/dalvikvm﹕ VFY: replacing opcode 0x71 at 0x000a
09-29 11:30:16.511 1111-1111/com.username.someapp D/AndroidRuntime﹕ Shutting down VM
09-29 11:30:16.511 1111-1111/com.username.someapp W/dalvikvm﹕ threadid=1: thread exiting with uncaught exception (group=0xb1aaaba8)
09-29 11:30:16.531 1111-1111/com.username.someapp E/AndroidRuntime﹕ FATAL EXCEPTION: main
Process: com.username.someapp, PID: 1111
java.lang.RuntimeException: Unable to start activity ComponentInfo{com.username.someapp/com.username.someapp.MapActivity}: java.lang.NullPointerException
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2195)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2245)
at android.app.ActivityThread.access$800(ActivityThread.java:135)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1196)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:136)
at android.app.ActivityThread.main(ActivityThread.java:5017)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:779)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:595)
at dalvik.system.NativeStart.main(Native Method)
Caused by: java.lang.NullPointerException
at com.username.someapp.MapActivity.onCreate(MapActivity.java:51)
at android.app.Activity.performCreate(Activity.java:5231)
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1087)
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2159)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2245)
at android.app.ActivityThread.access$800(ActivityThread.java:135)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1196)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:136)
at android.app.ActivityThread.main(ActivityThread.java:5017)
at java.lang.reflect.Method.invokeNative(Native Method)
I think the issue may be resolved with the instructions in here that says
Don't forget to then also include the dependencies from MapboxAndroidSDK / build.gradle in your classpath!
but I have no idea how to do that in Android Studio. Could anyone point to exactly what dependencies and how? Should I add them as File->Project Structure->Dependencies->Library dependency->? Any insight into this is deeply appreciated.
Thanks, Scott Barta.
For anybody else who comes in here with similar issues, I want to spell out how I fixed the problem. It was, as Scott points out, an issue with my code, not with the SDK. For starters, please note that mapbox-android-sdk comes with a test application which works, so in the end if you are stuck here, remember that the app is likely not the issue.
Anyway, it turned out that I had not initiated the Activity correctly. I had to instantiate it with the correct MapView and also use the appropriate mapbox:mapid (which should be declared in the *.xml file or the manifest file. Also, the example in the original page was for a fragment which looked slightly different from my code which caused some confusion. Last, but certainly not the least, I have switched so many platforms to get this to work (eclipse, gradle and now android studio) that is likely one of the biggest causes. I plan capturing these changes to my app somewhere and if I see interest, I will update this answer with more details.
We have an intermittent problem with slaves hanging AFTER the job itself is finished. In the post processing step (?) what we see is that the console log has this line:
Description set: vap_current_iter-2012_03_29_19_01_03
And then nothing. Usually, it will look like this:
Description set: prod_pull-2012_03_28_19_01_03
Notifying upstream build armada_Launch_prod_pull #13 of job completion
Project armada_Launch_prod_pull still waiting for 1 builds to complete
Notifying upstream projects of job completion
Notifying upstream of completion: armada_Launch_prod_pull #13
Finished: SUCCESS
I setup a logger for hudson.model.Run, and it currently has this :
at java.lang.Thread.run(Thread.java:619)
Mar 30, 2012 12:44:00 PM hudson.model.Run run
INFO: galleon_allUnit #1134 main build action completed: SUCCESS
Mar 30, 2012 12:44:00 PM hudson.model.Run setResult
FINE: galleon_allUnit #1134 : result is set to SUCCESS
java.lang.Exception
at hudson.model.Run.setResult(Run.java:352)
at hudson.model.Run.run(Run.java:1410)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:238)
Repeated for every hung slave.
The main hudson log doesn't have any additional information.
Disconnecting the slave has no effect.
Trying to do an orderly shutdown of Jenkins has no effect (jenkins actually appears to hang on shutdown).
The only way we have found to recover is to kill -9 the tomcat process.
The tread dump for one of the slaves (they are all the same) is:
Thread Dump
Channel reader thread: channel
"Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:199)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked java.io.BufferedInputStream#1ae615a
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1030)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Channel#e1d5ea
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Channel#e1d5ea
at java.lang.Object.wait(Object.java:485)
at hudson.remoting.Channel.join(Channel.java:766)
at hudson.remoting.Launcher.main(Launcher.java:420)
at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:366)
at hudson.remoting.Launcher.run(Launcher.java:206)
at hudson.remoting.Launcher.main(Launcher.java:168)
Ping thread for channel hudson.remoting.Channel#e1d5ea:channel
"Ping thread for channel hudson.remoting.Channel#e1d5ea:channel" Id=10 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
Pipe writer thread: channel
"Pipe writer thread: channel" Id=12 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#14263ed
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#14263ed
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
pool-1-thread-267
"pool-1-thread-267" Id=285 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374)
at hudson.Functions.getThreadInfos(Functions.java:872)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:93)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:89)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Number of locked synchronizers = 1
- java.util.concurrent.locks.ReentrantLock$NonfairSync#1186f88
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock#1798fdd
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock#1798fdd
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock#1d40442
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock#1d40442
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Signal Dispatcher
"Signal Dispatcher" Id=4 Group=system RUNNABLE
Any ideas on how to better recover or prevent this would be greatly appreciated.
We honestly just wrote a script that restarts jenkins every night at 4pm. We found that our breakages were happening at 3am give or take a half hour or so. Since restarting the server at this time, we haven't seen any further hangs. This is a way to prevent it as you asked, though it doesn't "Fix" the problem obviously!