Cassandra connection idling and timing out - docker
I'm trying to load and delete data from Cassandra using the python driver. I have tried this both using cassandra running in a docker container and again locally after the docker version was giving me problems. Here's an example of what I'm doing:
class Controller(object):
def __init__(self):
self.cluster = Cluster()
self.session = self.cluster.connect('mykeyspace')
def insert_into_cassandra(self):
query = ('INSERT INTO mytable (mykey, indexed_key) VALUES (?, ?)')
prepared = self.session.prepare(query)
prepared.consistency_level = ConsistencyLevel.QUORUM
params_gen = self.params_generator(fname)
execute_concurrent_with_args(self.session, prepared, self.parameter_generator(), concurrency=50)
def delete_param_gen(self, results):
for r in results:
yield [r.mykey]
def delete_by_index(self, value):
query = "SELECT mykey from mytable where indexed_key = '%s'" % value
res = self.session.execute(query)
delete_query = "DELETE from mytable where mykey = ?"
prepared = self.session.prepare(delete_query)
prepared.consistency_level = ConsistencyLevel.QUORUM
params_gen = self.delete_param_gen(res)
execute_concurrent_with_args(self.session, prepared, params_gen, concurrency=50)
Nothing crazy. When loading/deleting data, I frequently see the following messages:
Sending options message heartbeat on idle connection (4422117360) 127.0.0.1
Heartbeat failed for connection (4422117360) to 127.0.0.1
Here are some logs from deleting data.
[2017-02-28 08:37:20,562] [DEBUG] [cassandra.connection] Defuncting connection (4422117360) to 127.0.0.1: errors=Connection heartbeat timeout after 30 seconds, last_host=127.0.0.1
[2017-02-28 08:37:20,563] [DEBUG] [cassandra.io.libevreactor] Closing connection (4422117360) to 127.0.0.1
[2017-02-28 08:37:20,563] [DEBUG] [cassandra.io.libevreactor] Closed socket to 127.0.0.1
[2017-02-28 08:37:20,564] [DEBUG] [cassandra.pool] Defunct or closed connection (4422117360) returned to pool, potentially marking host 127.0.0.1 as down
[2017-02-28 08:37:20,566] [DEBUG] [cassandra.pool] Replacing connection (4422117360) to 127.0.0.1
[2017-02-28 08:37:20,567] [DEBUG] [cassandra.connection] Defuncting connection (4426057600) to 127.0.0.1: errors=Connection heartbeat timeout after 30 seconds, last_host=127.0.0.1
[2017-02-28 08:37:20,567] [DEBUG] [cassandra.io.libevreactor] Closing connection (4426057600) to 127.0.0.1
[2017-02-28 08:37:20,567] [DEBUG] [cassandra.io.libevreacto[2017-02-28 08:37:20,568] [ERROR] [cassandra.cluster] Unexpected exception while handling result in ResponseFuture:
Traceback (most recent call last):
File "cassandra/cluster.py", line 3536, in cassandra.cluster.ResponseFuture._set_result (cassandra/cluster.c:67556)
File "cassandra/cluster.py", line 3711, in cassandra.cluster.ResponseFuture._set_final_result (cassandra/cluster.c:71769)
File "cassandra/concurrent.py", line 154, in cassandra.concurrent._ConcurrentExecutor._on_success (cassandra/concurrent.c:3357)
File "cassandra/concurrent.py", line 203, in cassandra.concurrent.ConcurrentExecutorListResults._put_result (cassandra/concurrent.c:5539)
File "cassandra/concurrent.py", line 209, in cassandra.concurrent.ConcurrentExecutorListResults._put_result (cassandra/concurrent.c:5427)
File "cassandra/concurrent.py", line 123, in cassandra.concurrent._ConcurrentExecutor._execute_next (cassandra/concurrent.c:2369)
File "load_cassandra.py", line 148, in delete_param_gen
for r in rows:
File "cassandra/cluster.py", line 3991, in cassandra.cluster.ResultSet.next (cassandra/cluster.c:76025)
File "cassandra/cluster.py", line 4006, in cassandra.cluster.ResultSet.fetch_next_page (cassandra/cluster.c:76193)
File "cassandra/cluster.py", line 3781, in cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:73073)
cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {})r] Closed socket to 127.0.0.1
And here are some from inserting data:
[2017-02-28 16:50:25,594] [DEBUG] [cassandra.connection] Sending options message heartbeat on idle connection (140301574604448) 127.0.0.1
[2017-02-28 16:50:25,595] [DEBUG] [cassandra.cluster] [control connection] Attempting to reconnect
[2017-02-28 16:50:25,596] [DEBUG] [cassandra.cluster] [control connection] Opening new connection to 127.0.0.1
[2017-02-28 16:50:25,596] [DEBUG] [cassandra.connection] Not sending options message for new connection(140301347717016) to 127.0.0.1 because compression is disabled and a cql version was not specified
[2017-02-28 16:50:25,596] [DEBUG] [cassandra.connection] Sending StartupMessage on <AsyncoreConnection(140301347717016) 127.0.0.1:9042>
[2017-02-28 16:50:25,596] [DEBUG] [cassandra.connection] Sent StartupMessage on <AsyncoreConnection(140301347717016) 127.0.0.1:9042>
[2017-02-28 16:50:30,596] [DEBUG] [cassandra.io.asyncorereactor] Closing connection (140301347717016) to 127.0.0.1
[2017-02-28 16:50:30,596] [DEBUG] [cassandra.io.asyncorereactor] Closed socket to 127.0.0.1
[2017-02-28 16:50:30,596] [DEBUG] [cassandra.connection] Connection to 127.0.0.1 was closed during the startup handshake
[2017-02-28 16:50:30,597] [WARNING] [cassandra.cluster] [control connection] Error connecting to 127.0.0.1:
Traceback (most recent call last):
File "cassandra/cluster.py", line 2623, in cassandra.cluster.ControlConnection._reconnect_internal (cassandra/cluster.c:47899)
File "cassandra/cluster.py", line 2645, in cassandra.cluster.ControlConnection._try_connect (cassandra/cluster.c:48416)
File "cassandra/cluster.py", line 1119, in cassandra.cluster.Cluster.connection_factory (cassandra/cluster.c:15085)
File "cassandra/connection.py", line 333, in cassandra.connection.Connection.factory (cassandra/connection.c:5790)
cassandra.OperationTimedOut: errors=Timed out creating connection (5 seconds), last_host=None
[2017-02-28 16:50:39,309] [ERROR] [root] Exception inserting data into cassandra
Traceback (most recent call last):
File "load_cassandra.py", line 54, in run
controller.insert_into_cassandra(filename)
File "extract_to_cassandra.py", line 141, in insert_into_cassandra
for success, result in results:
File "cassandra/concurrent.py", line 177, in _results (cassandra/concurrent.c:4856)
File "cassandra/concurrent.py", line 186, in cassandra.concurrent.ConcurrentExecutorGenResults._results (cassandra/concurrent.c:4622)
File "cassandra/concurrent.py", line 165, in cassandra.concurrent._ConcurrentExecutor._raise (cassandra/concurrent.c:3745)
cassandra.WriteTimeout: Error from server: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'consistency': 'QUORUM', 'required_responses': 1, 'received_responses': 0}
[2017-02-28 16:50:39,465] [DEBUG] [cassandra.connection] Received options response on connection (140301574604448) from 127.0.0.1
[2017-02-28 16:50:39,466] [DEBUG] [cassandra.cluster] Shutting down Cluster Scheduler
[2017-02-28 16:50:39,467] [DEBUG] [cassandra.cluster] Shutting down control connection
[2017-02-28 16:50:39,467] [DEBUG] [cassandra.io.asyncorereactor] Closing connection (140301574604448) to 127.0.0.1
[2017-02-28 16:50:39,467] [DEBUG] [cassandra.io.asyncorereactor] Closed socket to 127.0.0.1
[2017-02-28 16:50:39,468] [DEBUG] [cassandra.pool] Defunct or closed connection (140301574604448) returned to pool, potentially marking host 127.0.0.1 as down
I tweaked with consistency and even set it to 1, but that didn't work. Inserts tend to work better when running cassandra locally as opposed to docker, but they still timeout. Deletes usually work for a couple of seconds and then hang/timeout.
edit: Here's are the logs from cassandra when things fail:
INFO 18:39:11 MUTATION messages were dropped in last 5000 ms: 4 for internal timeout and 0 for cross node timeout. Mean internal dropped latency: 2933809 ms and Mean cross-node dropped latency: 0 msINFO 18:39:11 Pool Name Active Pending Completed Blocked All Time Blocked [48/1513]
INFO 18:39:11 MutationStage 32 15 470 0 0
INFO 18:39:11 ViewMutationStage 0 0 0 0 0
INFO 18:39:11 ReadStage 0 0 59 0 0
INFO 18:39:11 RequestResponseStage 0 0 0 0 0
INFO 18:39:11 ReadRepairStage 0 0 0 0 0
INFO 18:39:11 CounterMutationStage 0 0 0 0 0
INFO 18:39:11 MiscStage 0 0 0 0 0
INFO 18:39:11 CompactionExecutor 0 0 6399 0 0
INFO 18:39:11 MemtableReclaimMemory 0 0 36 0 0
INFO 18:39:11 PendingRangeCalculator 0 0 1 0 0
INFO 18:39:11 GossipStage 0 0 0 0 0
INFO 18:39:11 SecondaryIndexManagement 0 0 0 0 0
INFO 18:39:11 HintsDispatcher 0 0 0 0 0
INFO 18:39:11 MigrationStage 0 0 2 0 0
INFO 18:39:11 MemtablePostFlush 0 0 62 0 0
INFO 18:39:11 PerDiskMemtableFlushWriter_0 0 0 36 0 0
INFO 18:39:11 ValidationExecutor 0 0 0 0 0
INFO 18:39:11 Sampler 0 0 0 0 0
INFO 18:39:11 MemtableFlushWriter 0 0 36 0 0
INFO 18:39:11 InternalResponseStage 0 0 0 0 0
INFO 18:39:11 AntiEntropyStage 0 0 0 0 0
INFO 18:39:11 CacheCleanupExecutor 0 0 0 0 0
INFO 18:39:11 Native-Transport-Requests 33 0 727 0 0
INFO 18:39:11 CompactionManager 0 0INFO 18:39:11 MessagingService n/a 0/0
INFO 18:39:11 Cache Type Size Capacity KeysToSave
INFO 18:39:11 KeyCache 1368 51380224 all
INFO 18:39:11 RowCache 0 0 all
INFO 18:39:11 Table Memtable ops,data
INFO 18:39:11 system_distributed.parent_repair_history 0,0
INFO 18:39:11 system_distributed.repair_history 0,0
INFO 18:39:11 system_distributed.view_build_status 0,0
INFO 18:39:11 system.compaction_history 1,231
INFO 18:39:11 system.hints 0,0
INFO 18:39:11 system.schema_aggregates 0,0
INFO 18:39:11 system.IndexInfo 0,0
INFO 18:39:11 system.schema_columnfamilies 0,0
INFO 18:39:11 system.schema_triggers 0,0
INFO 18:39:11 system.size_estimates 40,1255
INFO 18:39:11 system.schema_functions 0,0
INFO 18:39:11 system.paxos 0,0
INFO 18:39:11 system.views_builds_in_progress 0,0
INFO 18:39:11 system.built_views 0,0
INFO 18:39:11 system.peer_events 0,0
INFO 18:39:11 system.range_xfers 0,0
INFO 18:39:11 system.peers 0,0
INFO 18:39:11 system.batches 0,0
INFO 18:39:11 system.schema_keyspaces 0,0
INFO 18:39:11 system.schema_usertypes 0,0
INFO 18:39:11 system.local 0,0
INFO 18:39:11 system.sstable_activity 6,117
INFO 18:39:11 system.available_ranges 0,0
INFO 18:39:11 system.batchlog 0,0
INFO 18:39:11 system.schema_columns 0,0
INFO 18:39:11 system_schema.columns 0,0
INFO 18:39:11 system_schema.types 0,0
INFO 18:39:11 system_schema.indexes 0,0
INFO 18:39:11 system_schema.keyspaces 0,0
INFO 18:39:11 system_schema.dropped_columns 0,0
INFO 18:39:11 system_schema.aggregates 0,0
INFO 18:39:11 system_schema.triggers 0,0
INFO 18:39:11 system_schema.tables 0,0
INFO 18:39:11 system_schema.views 0,0
INFO 18:39:11 system_schema.functions 0,0
INFO 18:39:11 system_auth.roles 0,0
INFO 18:39:11 system_auth.role_members 0,0
INFO 18:39:11 system_auth.resource_role_permissons_index 0,0
INFO 18:39:11 system_auth.role_permissions 0,0
INFO 18:39:11 mykeyspace.mytable 430,27163514
INFO 18:39:11 system_traces.sessions 0,0
INFO 18:39:11 system_traces.events 0,0
INFO 18:39:13 ParNew GC in 261ms. CMS Old Gen: 46106544 -> 74868512; Par Eden Space: 208895224 -> 0; Par Survivor Space: 16012448 -> 26083328
I see messages like this too:
Out of 29 commit log syncs over the past 248s with average duration of 1596.14ms, 1 have exceeded the configured commit interval by an average of 18231.00ms
One thing you could try, is to reduce the idle_heartbeat_interval setting in your connection. By default it is 30 seconds, but you can configure that when instancing your Cluster class. In this example, I'll set it to 10 seconds:
def __init__(self):
self.cluster = Cluster(idle_heartbeat_interval=10)
self.session = self.cluster.connect('mykeyspace')
If that doesn't help, then it might be time to check your data model for anti-patterns.
Related
SonarQube Fetching analysis configuration settings shows the Operation has timed out
I am trying to integrate SonarQube with Jenkins Job and docker. For fetching the analysis configuration it shows Operation timed out while accessing the sonar server. Tried to curl the same url in job and it is giving success response. Please find below logs: 14:14:36 c:\workspace\testProject\sonartest>curl "https://sonar.xxx.com/api/server/version" 14:14:36 % Total % Received % Xferd Average Speed Time Time Time Current 14:14:36 Dload Upload Total Spent Left Speed 14:14:36 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 11 100 11 0 0 11 0 0:00:01 0:00:01 --:--:-- 11 14:14:37 7.9.1.27448 14:14:37 c:\workspace\testProject\sonartest>"c:\workspace\testProject\sonartest\sonar-scanner-msbuild\SonarScanner.MSBuild.exe" begin /k:"testprojectkey" 14:14:39 SonarScanner for MSBuild 4.10 14:14:39 Using the .NET Framework version of the Scanner for MSBuild 14:14:39 Pre-processing started. 14:14:39 Preparing working directories... 14:14:40 09:41:33.668 Updating build integration targets... 14:14:40 09:41:33.713 Fetching analysis configuration settings... 14:16:21 09:43:15.17 Failed to request and parse 'https://sonar.xxx.com/api/server/version': The operation has timed out 14:16:22 14:16:22 Unhandled Exception: System.Net.WebException: The operation has timed out 14:16:22 at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request) 14:16:22 at System.Net.WebClient.DownloadString(Uri address) 14:16:22 at SonarScanner.MSBuild.PreProcessor.SonarWebService.<>c__DisplayClass15_0.<DownloadServerVersion>b__0() 14:16:22 at SonarScanner.MSBuild.PreProcessor.SonarWebService.DoLogExceptions[T](Func`1 op, String url, Action`1 onError) 14:16:22 at SonarScanner.MSBuild.PreProcessor.SonarWebService.DownloadServerVersion() 14:16:22 at SonarScanner.MSBuild.PreProcessor.SonarWebService.GetServerVersion() 14:16:22 at SonarScanner.MSBuild.PreProcessor.SonarWebService.GetProperties(String projectKey, String projectBranch) 14:16:22 at SonarScanner.MSBuild.PreProcessor.TeamBuildPreProcessor.FetchArgumentsAndRulesets(ISonarQubeServer server, ProcessedArgs args, TeamBuildSettings settings, IDictionary`2& serverSettings, List`1& analyzersSettings) 14:16:22 at SonarScanner.MSBuild.PreProcessor.TeamBuildPreProcessor.DoExecute(ProcessedArgs localSettings) 14:16:22 at SonarScanner.MSBuild.BootstrapperClass.PreProcess() 14:16:22 at SonarScanner.MSBuild.BootstrapperClass.Execute() 14:16:22 at SonarScanner.MSBuild.Program.Execute(String[] args, ILogger logger) 14:16:22 at SonarScanner.MSBuild.Program.Main(String[] args)14:17:11 14:17:11 c:\workspace\testProject\sonartest>exit 255
Play a SDP file with VLC?
Goal
Confirming that VLC can play a stream from a locally given SDP file.
This appears to be possible based on testimonies:
Playing RTP using VLC - Stack Overflow
How to send SDP over RTP - Stack Overflow
Experiment
So I looked for a publicly available case and found Mobile Streaming, RTSP/RTP, Wowza Streaming Engine | Wowza Media Systems.
This works, plays color video with sound:
vlc rtsp://184.72.239.149/vod/mp4:BigBuckBunny_115k.mov
I managed to extract SDP file using trick from networking - How can I capture network traffic of a single process? - Ask Ubuntu
strace -f -e trace=network -s 10000 mplayer rtsp://184.72.239.149/vod/mp4:BigBuckBunny_115k.mov 2>&1 | grep a=
... then reformatted the file and saved to bbb.sdp
File looks like (actual line don't start with whitespace, and finish with Windows-style line ending):
o=- 785981631 785981631 IN IP4 184.72.239.149
s=BigBuckBunny_115k.mov
c=IN IP4 184.72.239.149
t=0 0
a=sdplang:en
a=range:npt=0- 596.48
a=control:*
m=audio 0 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/12000/2
a=fmtp:96 profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1490
a=control:trackID=1
m=video 0 RTP/AVP 97
a=rtpmap:97 H264/90000
a=fmtp:97 packetization-mode=1;profile-level-id=42C01E;sprop-parameter-sets=Z0LAHtkDxWhAAAADAEAAAAwDxYuS,aMuMsg==
a=cliprect:0,0,160,240
a=framesize:97 240-160
a=framerate:24.0
a=control:trackID=2
Then:
vlc bbb.sdp
shows an entry in playlist with correct duration (09:56, the information is in the SDP) but no video appears.
Trace network calls
VLC media player 2.1.6 Rincewind (revision 2.1.6-0-gea01d28)
Process 15739 attached
[0xf19118] [http] lua interface: Lua HTTP interface
Process 15740 attached
[pid 15739] bind(7, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
[pid 15739] bind(7, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)
[0xf19118] [http] main interface error: socket bind error (Permission denied)
[pid 15739] bind(7, {sa_family=AF_INET6, sin6_port=htons(8080), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRINUSE (Address already in use)
[0xf19118] [http] main interface error: socket bind error (Permission denied)
[0xf19118] [http] main interface error: cannot create socket(s) for HTTP host
[0xf19118] [http] lua interface error: Error loading script /usr/lib/vlc/lua/intf/http.luac: lua/intf/http.lua:328: Failed to create HTTP host
[pid 15739] +++ exited with 0 +++
Process 15741 attached
Process 15742 attached
Process 15743 attached
Process 15744 attached
[0xf3f9a8] dummy interface: using the dummy interface module...
Process 15745 attached
[pid 15745] bind(16, {sa_family=AF_INET, sin_port=htons(15947), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
[pid 15743] bind(17, {sa_family=AF_INET, sin_port=htons(15947), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)
[pid 15745] bind(14, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
[pid 15745] bind(16, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
[pid 15745] bind(18, {sa_family=AF_INET, sin_port=htons(49463), sin_addr=inet_addr("0.0.0.0")}, 16Unable to determine our source address: ) = 0
This computer has an invalid IP address: 0.0.0.0
[pid 15745] bind(17, {sa_family=AF_INET, sin_port=htons(15947), sin_addr=inet_addr("0.0.0.0")}, 16 <unfinished ...>
[pid 15743] bind(15, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16 <unfinished ...>
[pid 15745] <... bind resumed> ) = 0
[pid 15743] <... bind resumed> ) = 0
[pid 15743] bind(19, {sa_family=AF_INET, sin_port=htons(55667), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
[pid 15743] bind(20, {sa_family=AF_INET, sin_port=htons(15947), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
[pid 15743] bind(17, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
[pid 15743] bind(20, {sa_family=AF_INET, sin_port=htons(45441), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
[pid 15745] bind(14, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
[pid 15745] bind(22, {sa_family=AF_INET, sin_port=htons(55005), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
Process 15746 attached
[pid 15743] +++ exited with 0 +++
Process 15747 attached
Process 15748 attached
Process 15749 attached
Process 15750 attached
Process 15751 attached
[0x7f2f80c01838] live555 demux error: no data received in 10s, aborting
[pid 15746] --- SIGRTMIN {si_signo=SIGRTMIN, si_code=SI_TKILL, si_pid=15738, si_uid=1000} ---
[pid 15746] +++ exited with 0 +++
[pid 15751] --- SIGRTMIN {si_signo=SIGRTMIN, si_code=SI_TKILL, si_pid=15738, si_uid=1000} ---
[pid 15751] +++ exited with 0 +++
[pid 15747] +++ exited with 0 +++
[pid 15748] +++ exited with 0 +++
[pid 15749] +++ exited with 0 +++
[pid 15750] +++ exited with 0 +++
[pid 15745] +++ exited with 0 +++
It appears that VLC does not even try to connect to IP 184.72.239.149 .
I'm clueless. Is this a bug?
Questions
Why would playing directly from RTSP URL work, and not playing a locally saved SDP file?
What to do?
When you receive the SDP from the server during a RTSP request, it is customized by the server to tell you where it's going to send the stream it's starting. Without the RTSP control channel to tell the server you want an RTP stream, it will not stream out. Direct SDP playback is useful essentially only if you have a broadcast/multicast RTP stream, or a continuous "push" RTP stream, not if you have a VOD stream that would otherwise be issued by RTSP.
GCDWebServer:In the IOS7.0 devices can not upload the video
PC can be very good to achieve the video upload and download, but there are two problems in the use of IOS devices: 1、In the IOS7.0 devices can not upload the video, the following is log: [DEBUG] Did open connection on socket 17 [DEBUG] Did connect [DEBUG] Did start background task [DEBUG] Connection received 528 bytes on socket 17 [DEBUG] Connection received 234 bytes on socket 17 [DEBUG] Connection received 46 bytes on socket 17 [DEBUG] Connection on socket 17 preflighting request "POST /upload" with 808 bytes body [DEBUG] Connection on socket 17 processing request "POST /upload" with 808 bytes body 2015-12-14 10:58:34.523 GCDWebServer[1701:403846] [UPLOAD] /var/mobile/Containers/Data/Application/5F45B407-9DA8-43D1-AADF-07CF22B1C4B2/Documents/IMG_9675.mp4 [DEBUG] Connection sent 175 bytes on socket 17 [DEBUG] Connection sent 2 bytes on socket 17 [DEBUG] Did close connection on socket 17 [VERBOSE] [my localhost:80] iPad2(7.0) client:50692 200 "POST /upload" (808 | 177) [DEBUG] Did open connection on socket 17 2015-12-14 10:58:34.577 GCDWebServer[1701:403846] album’s name == GCDWebServer [DEBUG] Connection received 400 bytes on socket 17 [DEBUG] Connection on socket 17 preflighting request "GET /list" with 400 bytes body [DEBUG] Connection on socket 17 processing request "GET /list" with 400 bytes body [DEBUG] Connection sent 177 bytes on socket 17 [DEBUG] Connection sent 213 bytes on socket 17 [DEBUG] Did close connection on socket 17 [VERBOSE] [my localhost:80] iPad2(7.0) client's IP:50693 200 "GET /list" (400 | 390) [DEBUG] Did open connection on socket 17 [DEBUG] Connection received 453 bytes on socket 17 [DEBUG] Connection on socket 17 preflighting request "GET /download" with 453 bytes body [DEBUG] Connection on socket 17 processing request "GET /download" with 453 bytes body [DEBUG] Connection sent 182 bytes on socket 17 [DEBUG] Did close connection on socket 17 [VERBOSE] [my localhost:80] iPad2(7.0) client's IP:50694 304 "GET /download" (453 | 182) [DEBUG] Did disconnect [DEBUG] Did end background task 2、In the IPAD device can not download the video, the following is log: [DEBUG] Did open connection on socket 17 [DEBUG] Did connect [DEBUG] Did start background task [DEBUG] Connection received 427 bytes on socket 17 [DEBUG] Connection on socket 17 preflighting request "GET /download" with 427 bytes body [DEBUG] Connection on socket 17 processing request "GET /download" with 427 bytes body [DEBUG] Connection sent 398 bytes on socket 17 [DEBUG] Connection sent 32768 bytes on socket 17 [DEBUG] Connection sent 32768 bytes on socket 17 [DEBUG] Connection sent 32768 bytes on socket 17 [DEBUG] Connection sent 32768 bytes on socket 17 [DEBUG] Connection sent 32768 bytes on socket 17 [DEBUG] Connection sent 32768 bytes on socket 17 [ERROR] Error while writing to socket 17: Broken pipe (32) [DEBUG] Did close connection on socket 17 [VERBOSE] [my localhost:80] iPad2(7.0) client's IP:50690 200 "GET /download" (427 | 197006) [DEBUG] Did open connection on socket 17 [DEBUG] Connection received 346 bytes on socket 17 [DEBUG] Connection on socket 17 preflighting request "GET /download" with 346 bytes body [DEBUG] Connection on socket 17 processing request "GET /download" with 346 bytes body [DEBUG] Connection sent 398 bytes on socket 17 [DEBUG] Connection sent 32768 bytes on socket 17 [DEBUG] Connection sent 32768 bytes on socket 17 [DEBUG] Connection sent 32768 bytes on socket 17 [DEBUG] Connection sent 32768 bytes on socket 17 [ERROR] Error while writing to socket 17: Broken pipe (32) [DEBUG] Did close connection on socket 17 [VERBOSE] [my localhost:80] iPad2(7.0) client's IP:50691 200 "GET /download" (346 | 131470) [DEBUG] Did disconnect [DEBUG] Did end background task Do you know why? Thank you very much! Sorry for my ignorance.
GFS2 flags 0x00000005 blocked,join
I have cluster RHEL6, cman, corosync, pacemaker. After adding new memebers I got error in GFS mounting. GFS never mounts on servers. group_tool fence domain member count 4 victim count 0 victim now 0 master nodeid 1 wait state none members 1 2 3 4 dlm lockspaces name clvmd id 0x4104eefa flags 0x00000000 change member 4 joined 1 remove 0 failed 0 seq 1,1 members 1 2 3 4 gfs mountgroups name lv_gfs_01 id 0xd5eacc83 flags 0x00000005 blocked,join change member 3 joined 1 remove 0 failed 0 seq 1,1 members 1 2 3 In processes: root 2695 2690 0 08:03 pts/1 00:00:00 /bin/bash /etc/init.d/gfs2 start root 2702 2695 0 08:03 pts/1 00:00:00 /bin/bash /etc/init.d/gfs2 start root 2704 2703 0 08:03 pts/1 00:00:00 /sbin/mount.gfs2 /dev/mapper/vg_shared-lv_gfs_01 /mnt/share -o rw,_netdev,noatime,nodiratime fsck.gfs2 -yf /dev/vg_shared/lv_gfs_01 Initializing fsck jid=1: Replayed 0 of 0 journaled data blocks jid=1: Replayed 20 of 21 metadata blocks Recovering journals (this may take a while) Journal recovery complete. Validating Resource Group index. Level 1 rgrp check: Checking if all rgrp and rindex values are good. (level 1 passed) RGs: Consistent: 183 Cleaned: 1 Inconsistent: 0 Fixed: 0 Total: 184 2 blocks may need to be freed in pass 5 due to the cleaned resource groups. Starting pass1 Pass1 complete Starting pass1b Pass1b complete Starting pass1c Pass1c complete Starting pass2 Pass2 complete Starting pass3 Pass3 complete Starting pass4 Pass4 complete Starting pass5 Block 11337799 (0xad0047) bitmap says 1 (Data) but FSCK saw 0 (Free) Fixed. Block 11337801 (0xad0049) bitmap says 1 (Data) but FSCK saw 0 (Free) Fixed. RG #11337739 (0xad000b) free count inconsistent: is 65500 should be 65502 RG #11337739 (0xad000b) Inode count inconsistent: is 15 should be 13 Resource group counts updated Pass5 complete The statfs file is wrong: Current statfs values: blocks: 12057320 (0xb7fae8) free: 9999428 (0x989444) dinodes: 15670 (0x3d36) Calculated statfs values: blocks: 12057320 (0xb7fae8) free: 9999432 (0x989448) dinodes: 15668 (0x3d34) The statfs file was fixed. Writing changes to disk gfs2_fsck complete gfs2_edit -p 0xad0047 field di_size /dev/vg_shared/lv_gfs_01 10 (Block 11337799 is type 10: Ext. attrib which is not implemented) Howto drop flag blocked,join from GFS?
I solved it by reboot all servers which have GFS, it is one of the unpleasant behavior of GFS. GFS lock based on kernel and in the few cases it solved only with reboot. there is very usefull manual - https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Global_File_System_2/index.html
Ruby net/ssh hangs with specific host
I have an issue where if I'm trying to create a ssh connection to a specific host, the connection hangs without timing out. I have tried connecting to this host from the same machine from outside the rails console and it works so I'm assuming this shouldn't be anything related to routing/firewall. I can also confirm I have several other hosts with the exact same OS configuration in different places and they work.
This is the log when running Net::SSH.start in debug mode:
ssh = Net::SSH.start("1.2.3.4", "user", password: "password", verbose: :debug)
2014-02-27 13:17:43 +0100 [DEBUG] establishing connection to 1.2.3.4:22
2014-02-27 13:17:43 +0100 [DEBUG] connection established
2014-02-27 13:17:43 +0100 [INFO] negotiating protocol version
2014-02-27 13:17:43 +0100 [DEBUG] remote is `SSH-2.0-OpenSSH_4.3'
2014-02-27 13:17:43 +0100 [DEBUG] local is `SSH-2.0-Ruby/Net::SSH_2.6.8 i686-linux'
2014-02-27 13:17:43 +0100 [DEBUG] read 704 bytes
2014-02-27 13:17:43 +0100 [DEBUG] received packet nr 0 type 20 len 700
2014-02-27 13:17:43 +0100 [INFO] got KEXINIT from server
2014-02-27 13:17:43 +0100 [INFO] sending KEXINIT
2014-02-27 13:17:43 +0100 [DEBUG] queueing packet nr 0 type 20 len 1620
2014-02-27 13:17:43 +0100 [DEBUG] sent 1624 bytes
2014-02-27 13:17:43 +0100 [INFO] negotiating algorithms
2014-02-27 13:17:43 +0100 [DEBUG] negotiated:
* kex: diffie-hellman-group-exchange-sha1
* host_key: ssh-rsa
* encryption_server: aes128-cbc
* encryption_client: aes128-cbc
* hmac_client: hmac-sha1
* hmac_server: hmac-sha1
* compression_client: none
* compression_server: none
* language_client:
* language_server:
2014-02-27 13:17:43 +0100 [DEBUG] exchanging keys
2014-02-27 13:17:43 +0100 [DEBUG] queueing packet nr 1 type 34 len 20
2014-02-27 13:17:43 +0100 [DEBUG] sent 24 bytes
At this point the ssh connection just hangs and could stay like this for 15-30 minutes.
Unfortunately I have no error message or anything so I'm really clueless about what the problem might be.
Some specs:
ruby-2.0.0-p0
rails (3.2.13)
net-ssh-2.8.0
The IP address in the log is not a real IP on purpose.
Any suggestion about what the problem could be? Or maybe some other log or place I could check out? I found a similar problem outside SO but it didn't get a solution so I'm trying to ask here...
I solved by reducing maximum transmission unit (MTU). My environment was some specific case. I was trying ssh from VMware Virtual Machine to Openstack Instance. And openstack needed smaller packet to be connected by kitchen-openstack which is using fog, using NET::SSH. Not sure if this works for you, but have a try for these commands (assuming in Ubuntu):
check your MTU
sudo netstat -i
You'll get some output like this, in the second column, you can check MTU:
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 9658 0 0 0 308 0 0 0 BMRU
lo 16436 0 12952 0 0 0 12952 0 0 0 LRU
For network interface eth0, you can try to reduce MTU from 1500 to, for example 1400 like this:
sudo ifconfig eth0 mtu 1400
try net ssh
ssh = Net::SSH.start("1.2.3.4", "user", password: "password", verbose: :debug)