Can I know if a video is H.264? - ruby-on-rails

I'm trying to find out whether a video uploaded through Carrierwave is encoded with H.264 or not. So far I tried the MiniExiftool gem since it lets me read a file's metadata. However I don't see it specified anywhere. Perhaps it's there and I'm just missing it. Any help, suggestions, or guidance? I'd appreciate it. Here's what I get when I use MiniExiftool.new #file.path.
=> #<MiniExiftool:0x007f7f82884628 #opts={:numerical=>false, :composite=>true, :
ignore_minor_errors=>false, :replace_invalid_chars=>false, :timestamps=>Time}, #
values={"exiftoolversion"=>8.9, "filesize"=>"35 MB", "filemodifydate"=>2014-04-2
8 18:46:51 +0000, "filepermissions"=>"rw-r--r--", "filetype"=>"MP4", "mimetype"=
>"video/mp4", "majorbrand"=>"MP4 Base Media v1 [IS0 14496-12:2003]", "minorvers
ion"=>"0.0.0", "compatiblebrands"=>["isom", "3gp4"], "movieheaderversion"=>0, "c
reatedate"=>2014-04-23 14:56:19 +0000, "modifydate"=>2014-04-23 14:56:19 +0000,
"timescale"=>1000, "duration"=>"14.28 s", "preferredrate"=>1, "preferredvolume"=
>"100.00%", "previewtime"=>"0 s", "previewduration"=>"0 s", "postertime"=>"0 s",
"selectiontime"=>"0 s", "selectionduration"=>"0 s", "currenttime"=>"0 s", "next
trackid"=>3, "trackheaderversion"=>0, "trackcreatedate"=>2014-04-23 14:56:19 +00
00, "trackmodifydate"=>2014-04-23 14:56:19 +0000, "trackid"=>1, "trackduration"=
>"13.90 s", "tracklayer"=>0, "trackvolume"=>"0.00%", "imagewidth"=>1920, "imageh
eight"=>1080, "graphicsmode"=>"srcCopy", "opcolor"=>"0 0 0", "compressorid"=>"av
c1", "sourceimagewidth"=>1920, "sourceimageheight"=>1080, "xresolution"=>72, "yr
esolution"=>72, "bitdepth"=>24, "videoframerate"=>29.998, "matrixstructure"=>"1
0 0 0 1 0 0 0 1", "mediaheaderversion"=>0, "mediacreatedate"=>2014-04-23 14:56:1
9 +0000, "mediamodifydate"=>2014-04-23 14:56:19 +0000, "mediatimescale"=>48000,
"mediaduration"=>"14.28 s", "handlertype"=>"Audio Track", "handlerdescription"=>
"SoundHandle", "balance"=>0, "audioformat"=>"mp4a", "audiochannels"=>2, "audiobi
tspersample"=>16, "audiosamplerate"=>48000, "moviedatasize"=>36819481, "avgbitra
te"=>"20.6 Mbps", "imagesize"=>"1920x1080", "rotation"=>0}, #changed_values={},
#errors={}, #filename="/home/user/uploads/dbrvp74x6r.mp4", #output="[{\n \"SourceFile\": \"home/user/uploads/dbrvp74x6r.mp4\",\n \"ExifToolVersion\": 8.90,\n \"FileName\": \"dbrvp74x6r.mp4\",\n
\"Directory\": \"home/user/uploads\",\n \"FileSize\": \"35 MB\",\n \"FileModifyDate\": \"2014:0
4:28 18:46:51+00:00\",\n \"FilePermissions\": \"rw-r--r--\",\n \"FileType\": \
"MP4\",\n \"MIMEType\": \"video/mp4\",\n \"MajorBrand\": \"MP4 Base Media v1
[IS0 14496-12:2003]\",\n \"MinorVersion\": \"0.0.0\",\n \"CompatibleBrands\":
[\"isom\",\"3gp4\"],\n \"MovieHeaderVersion\": 0,\n \"CreateDate\": \"2014:04:
23 14:56:19\",\n \"ModifyDate\": \"2014:04:23 14:56:19\",\n \"TimeScale\": 100
0,\n \"Duration\": \"14.28 s\",\n \"PreferredRate\": 1,\n \"PreferredVolume\"
: \"100.00%\",\n \"PreviewTime\": \"0 s\",\n \"PreviewDuration\": \"0 s\",\n
\"PosterTime\": \"0 s\",\n \"SelectionTime\": \"0 s\",\n \"SelectionDuration\"
: \"0 s\",\n \"CurrentTime\": \"0 s\",\n \"NextTrackID\": 3,\n \"TrackHeaderV
ersion\": 0,\n \"TrackCreateDate\": \"2014:04:23 14:56:19\",\n \"TrackModifyDa
te\": \"2014:04:23 14:56:19\",\n \"TrackID\": 1,\n \"TrackDuration\": \"13.90
s\",\n \"TrackLayer\": 0,\n \"TrackVolume\": \"0.00%\",\n \"ImageWidth\": 192
0,\n \"ImageHeight\": 1080,\n \"GraphicsMode\": \"srcCopy\",\n \"OpColor\": \
"0 0 0\",\n \"CompressorID\": \"avc1\",\n \"SourceImageWidth\": 1920,\n \"Sou
rceImageHeight\": 1080,\n \"XResolution\": 72,\n \"YResolution\": 72,\n \"Bit
Depth\": 24,\n \"VideoFrameRate\": 29.998,\n \"MatrixStructure\": \"1 0 0 0 1
0 0 0 1\",\n \"MediaHeaderVersion\": 0,\n \"MediaCreateDate\": \"2014:04:23 14
:56:19\",\n \"MediaModifyDate\": \"2014:04:23 14:56:19\",\n \"MediaTimeScale\"
: 48000,\n \"MediaDuration\": \"14.28 s\",\n \"HandlerType\": \"Audio Track\",
\n \"HandlerDescription\": \"SoundHandle\",\n \"Balance\": 0,\n \"AudioFormat
\": \"mp4a\",\n \"AudioChannels\": 2,\n \"AudioBitsPerSample\": 16,\n \"Audio
SampleRate\": 48000,\n \"MovieDataSize\": 36819481,\n \"AvgBitrate\": \"20.6 M
bps\",\n \"ImageSize\": \"1920x1080\",\n \"Rotation\": 0\n}]


Yes, this video is h.264. "CompressorID": "avc1". AVC = Advanced Video Coding = h.264

Related

How to keep video quality same as it is after merge intro image beggining to video using ffmpeg

I have selected high resolution video but once i run command the video resolution quality was changed and too poor quality's video i get as output video, but i don't would like to loose my video quality. let me share full command and complete log below:
Note: libx264 encoder is not supporting iOS, I'm getting failure error so i use h264_videotoolbox so i would like to get supported command with h264_videotoolbox encoder
Command:
ffmpeg -i test.MOV -loop 1 -t 5 -i 2.jpg -f lavfi -t 5 -i anullsrc
-filter_complex "[0:v]trim=0:5,drawbox=t=fill[base];[1][base]scale2ref=iw:ih:force_original_aspect_ratio=decrease:flags=spline[2nd][base2];[base2][2nd]overlay='(W-w)/2':'(H-h)/2'[padded];[padded][2:a][0:v][0:a]concat=n=2:v=1:a=1[v][a]"
-c:v h264_videotoolbox -c:a aac -map "[v]" -map "[a]" output.mp4
Complete log
ffmpeg version 4.4 Copyright (c) 2000-2021 the FFmpeg developers
built with Apple clang version 12.0.0 (clang-1200.0.32.29)
configuration: --prefix=/usr/local/Cellar/ffmpeg/4.4_1 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.MOV':
Metadata:
major_brand : qt
minor_version : 0
compatible_brands: qt
creation_time : 2021-03-07T06:36:17.000000Z
com.apple.quicktime.location.accuracy.horizontal: 30.000000
com.apple.quicktime.location.ISO6709: +23.1141+072.5768+061.729/
com.apple.quicktime.make: Apple
com.apple.quicktime.model: iPhone 6s
com.apple.quicktime.software: 14.3
com.apple.quicktime.creationdate: 2021-03-07T12:06:17+0530
Duration: 00:00:29.79, start: 0.000000, bitrate: 15778 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080, 15643 kb/s, 29.98 fps, 29.97 tbr, 600 tbn, 1200 tbc (default)
Metadata:
creation_time : 2021-03-07T06:36:17.000000Z
handler_name : Core Media Video
vendor_id : [0][0][0][0]
encoder : H.264
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 89 kb/s (default)
Metadata:
creation_time : 2021-03-07T06:36:17.000000Z
handler_name : Core Media Audio
vendor_id : [0][0][0][0]
Stream #0:2(und): Data: none (mebx / 0x7862656D), 0 kb/s (default)
Metadata:
creation_time : 2021-03-07T06:36:17.000000Z
handler_name : Core Media Metadata
Stream #0:3(und): Data: none (mebx / 0x7862656D), 0 kb/s (default)
Metadata:
creation_time : 2021-03-07T06:36:17.000000Z
handler_name : Core Media Metadata
Stream #0:4(und): Data: none (mebx / 0x7862656D), 34 kb/s (default)
Metadata:
creation_time : 2021-03-07T06:36:17.000000Z
handler_name : Core Media Metadata
Input #1, image2, from '2.jpg':
Duration: 00:00:00.04, start: 0.000000, bitrate: 17347 kb/s
Stream #1:0: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 360x360 [SAR 72:72 DAR 1:1], 25 fps, 25 tbr, 25 tbn, 25 tbc
Input #2, lavfi, from 'anullsrc':
Duration: N/A, start: 0.000000, bitrate: 705 kb/s
Stream #2:0: Audio: pcm_u8, 44100 Hz, stereo, u8, 705 kb/s
Stream mapping:
Stream #0:0 (h264) -> trim
Stream #0:0 (h264) -> concat:in1:v0
Stream #0:1 (aac) -> concat:in1:a0
Stream #1:0 (mjpeg) -> scale2ref:default
Stream #2:0 (pcm_u8) -> concat:in0:a0
concat:out:v0 -> Stream #0:0 (h264_videotoolbox)
concat:out:a0 -> Stream #0:1 (aac)
Press [q] to stop, [?] for help
[swscaler # 0x7f976242b000] deprecated pixel format used, make sure you did set range correctly
Output #0, mp4, to 'output.mp4':
Metadata:
major_brand : qt
minor_version : 0
compatible_brands: qt
com.apple.quicktime.creationdate: 2021-03-07T12:06:17+0530
com.apple.quicktime.location.accuracy.horizontal: 30.000000
com.apple.quicktime.location.ISO6709: +23.1141+072.5768+061.729/
com.apple.quicktime.make: Apple
com.apple.quicktime.model: iPhone 6s
com.apple.quicktime.software: 14.3
encoder : Lavf58.76.100
Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080, q=2-31, 200 kb/s, 29.97 fps, 30k tbn (default)
Metadata:
encoder : Lavc58.134.100 h264_videotoolbox
Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 69 kb/s (default)
Metadata:
encoder : Lavc58.134.100 aac
frame= 1 fps=0.0 q=0.0 size= 0kB time=00:00:00.00 bitrate=N/A speed= frame= 12 fps=0.0 q=-0.0 size= 256kB time=00:00:00.32 bitrate=6452.4kbits/frame= 34 fps= 32 q=-0.0 size= 256kB time=00:00:01.02 bitrate=2053.0kbits/frame= 55 fps= 35 q=-0.0 size= 256kB time=00:00:01.74 bitrate=1204.4kbits/frame= 76 fps= 37 q=-0.0 size= 256kB time=00:00:02.46 bitrate= 852.2kbits/frame= 98 fps= 38 q=-0.0 size= 256kB time=00:00:03.18 bitrate= 659.4kbits/frame= 120 fps= 39 q=-0.0 size= 256kB time=00:00:03.90 bitrate= 537.7kbits/frame= 141 fps= 39 q=-0.0 size= 512kB time=00:00:04.62 bitrate= 907.8kbits/[out_0_0 # 0x7f975de0ae80] 100 buffers queued in out_0_0, something may be wrong.
[out_0_1 # 0x7f975de0a5c0] 100 buffers queued in out_0_1, something may be wrong.
frame= 301 fps= 67 q=-0.0 size= 768kB time=00:00:11.09 bitrate= 566.9kbits/frame= 406 fps= 81 q=-0.0 size= 1024kB time=00:00:14.60 bitrate= 574.4kbits/frame= 509 fps= 92 q=-0.0 size= 1280kB time=00:00:18.04 bitrate= 581.2kbits/frame= 604 fps=100 q=-0.0 size= 1792kB time=00:00:21.19 bitrate= 692.5kbits/frame= 705 fps=108 q=-0.0 size= 2048kB time=00:00:24.56 bitrate= 682.9kbits/frame= 809 fps=115 q=-0.0 size= 2304kB time=00:00:28.04 bitrate= 672.9kbits/frame= 909 fps=121 q=-0.0 size= 2816kB time=00:00:31.37 bitrate= 735.4kbits/frame= 1012 fps=126 q=-0.0 size= 3072kB time=00:00:34.71 bitrate= 725.0kbits/frame= 1043 fps=127 q=-0.0 Lsize= 3288kB time=00:00:34.78 bitrate= 774.3kbits/s speed=4.23x
video:2995kB audio:255kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.168448%
[aac # 0x7f9760024200] Qavg: 9569.656

The main video have a bitrate aprox of 15778K, and this is a signal for quality hight...
Duration: 00:00:29.79, start: 0.000000, bitrate: 15778 kb/s
You have to put bitrate parameter -b:v {bitrate, in this case 15778K} in your code to keep hight qualiy, some like
ffmpeg -i test.MOV -loop 1 -t 5 -i 2.jpg -f lavfi -t 5 -i anullsrc -filter_complex "[0:v]trim=0:5,drawbox=t=fill[base];[1][base]scale2ref=iw:ih:force_original_aspect_ratio=decrease:flags=spline[2nd][base2];[base2][2nd]overlay='(W-w)/2':'(H-h)/2'[padded];[padded][2:a][0:v][0:a]concat=n=2:v=1:a=1[v][a]" -c:v h264_videotoolbox -b:v 15778K -c:a aac -map "[v]" -map "[a]" output.mp4
NOTE: if have an error with "-c:a aac" parameter, remove it, because in Apple`s OS aac is for default because is an Apple codec
King Regards

IBrokers R API and same day intraday prices

I think this is be an IB API more than the IBrokers R package.
I am using reqHistoricalData to get 30 minutes intraday historical data. The market is open and I am not getting the same day's data. I only get yesterday's data.
Is it possible to get the same day intraday bar data?
here is the code I am using, it only gives data for the previous day, not same day.
library(tidyverse)
library(IBrokers)
tws = twsConnect()
contract <- twsEquity('VOD','SMART')
VOD_intraday = IBrokers::reqHistoricalData(tws, Contract = contract, endDateTime = "20210408 13:24:28", barSize = "1 min", duration = "1 D")
VOD_intraday %>% as.data.frame() %>% rownames_to_column(var = "time") %>% arrange(desc(time)) %>% head()
It's 13:27 GMT on 2021-04-08 and London is open. And here is the response - it only gives data from 2020-04-07:
> contract <- twsEquity('VOD','SMART')
> VOD_intraday = IBrokers::reqHistoricalData(tws, Contract = contract, endDateTime = "20210408 13:24:28", barSize = "1 min", duration = "1 D")
waiting for TWS reply on VOD .... done.
> VOD_intraday %>% as.data.frame() %>% rownames_to_column(var = "time") %>% arrange(desc(time)) %>% head()
time VOD.Open VOD.High VOD.Low VOD.Close VOD.Volume VOD.WAP VOD.hasGaps VOD.Count
1 2021-04-07 20:59:00 18.96 18.98 18.95 18.98 1131 18.958 0 265
2 2021-04-07 20:58:00 18.96 18.96 18.95 18.96 90 18.957 0 42
3 2021-04-07 20:57:00 18.96 18.97 18.95 18.95 258 18.960 0 72
4 2021-04-07 20:56:00 18.96 18.96 18.95 18.95 124 18.959 0 58
5 2021-04-07 20:55:00 18.96 18.96 18.95 18.96 56 18.958 0 34
6 2021-04-07 20:54:00 18.95 18.96 18.95 18.95 26 18.951 0 12
Instead of VOD, you can use SPY, MSFT or any US security while the US market is open.
Edit: It turns out you need realtime subscription to get same day data. The answer below works.

One has to specify the ending time, or leave it blank to get the most recent data available.
Try this:
VOD_intraday = IBrokers::reqHistoricalData(tws, Contract = contract, endTime = "", barSize = "1 min", duration = "1 D")
Here's the execution when I run it:
> library(tidyverse)
> library(IBrokers)
IBrokers version 0.9-10. Implementing API Version 9.64
IBrokers comes with NO WARRANTY. Not intended for production use!
See ?IBrokers for details.
> tws = twsConnect()
> contract <- twsEquity('SPY','SMART')
> VOD_intraday = IBrokers::reqHistoricalData(tws, Contract = contract, endDateTime = "", barSize = "1 min", duration = "1 D")
waiting for TWS reply on SPY ........... done.
> head(VOD_intraday)
SPY.Open SPY.High SPY.Low SPY.Close SPY.Volume SPY.WAP SPY.hasGaps SPY.Count
2021-04-08 07:30:00 407.93 407.98 407.68 407.80 5042 407.846 0 1709
2021-04-08 07:31:00 407.81 408.00 407.74 407.98 1615 407.844 0 1065
2021-04-08 07:32:00 407.99 408.05 407.81 407.90 2451 407.932 0 1560
2021-04-08 07:33:00 407.89 407.98 407.88 407.95 2353 407.932 0 1300
2021-04-08 07:34:00 407.95 407.97 407.81 407.81 1708 407.907 0 1012
2021-04-08 07:35:00 407.82 407.86 407.61 407.67 2729 407.726 0 1458
And for symbol VOD:
> contract <- twsEquity('VOD','SMART')
> VOD_intraday = IBrokers::reqHistoricalData(tws, Contract = contract, endDateTime = "", barSize = "1 min", duration = "1 D")
waiting for TWS reply on VOD .... done.
> head(VOD_intraday)
VOD.Open VOD.High VOD.Low VOD.Close VOD.Volume VOD.WAP VOD.hasGaps VOD.Count
2021-04-08 07:30:00 18.95 18.95 18.91 18.92 246 18.921 0 49
2021-04-08 07:31:00 18.91 18.91 18.90 18.90 69 18.905 0 31
2021-04-08 07:32:00 18.90 18.90 18.87 18.87 237 18.881 0 44
2021-04-08 07:33:00 18.87 18.87 18.86 18.87 45 18.870 0 20
2021-04-08 07:34:00 18.87 18.87 18.85 18.86 173 18.860 0 57
2021-04-08 07:35:00 18.86 18.87 18.85 18.86 39 18.859 0 19

xcode ios simulator The request timed out

I have an iOS 13.3 application running on the XCode iOS simulator. It communicates with a simple python tcpserver on a Raspberry PI running on a local network. The iOS code is based on //https://dev.to/bsorrentino/develop-a-textfield-validator-for-swiftui-7d3 which uses URLSession. I am getting a timeout in the iOS code after about 10 seconds. I've checked the network packets with tcpdump on the mac. The proper packets appear to be sent and received. The url used is http://10.0.0.119:60708 as the PI server does not handle https at this point. The python server receives a JWT, prepends the thread name and echos it back. The python snippet is as follows:
class ThreadedTCPRequestHandler(socketserver.BaseRequestHandler):
def handle(self):
while True:
data = self.request.recv(4096).decode('utf-8')
if len(data) == 0:
break
cur_thread = threading.current_thread()
response = "{}: {}".format(cur_thread.name, data)
print(response)
self.request.sendall(response.encode('utf-8'))
The error is:
2020-02-22 12:11:35.324596-0500 PIHome[64511:66647174] Task <2C1CE343-FB68-40CB-97C5-5E7967933838>.<2> finished with error [-1001] Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={NSUnderlyingError=0x600001046ca0 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, NSErrorFailingURLStringKey=http://10.0.0.119:60708/temperature, NSErrorFailingURLKey=http://10.0.0.119:60708/temperature, _kCFStreamErrorDomainKey=4, _kCFStreamErrorCodeKey=-2102, NSLocalizedDescription=The request timed out.}
The tcpdump is as follows:
12:11:25.305647 IP 10.0.0.89.52915 > hastings.60708: Flags [S], seq 2969915581, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2152442902 ecr 0,sackOK,eol], length 0
0x0000: b827 eb94 a718 8c85 9048 25eb 0800 4500 .'.......H%...E.
0x0010: 0040 0000 4000 4006 25e9 0a00 0059 0a00 .#..#.#.%....Y..
0x0020: 0077 ceb3 ed24 b105 50bd 0000 0000 b002 .w...$..P.......
0x0030: ffff 382f 0000 0204 05b4 0103 0306 0101 ..8/............
0x0040: 080a 804b ac16 0000 0000 0402 0000 ...K..........
12:11:25.312528 IP hastings.60708 > 10.0.0.89.52915: Flags [S.], seq 3390573497, ack 2969915582, win 28960, options [mss 1460,sackOK,TS val 3187848125 ecr 2152442902,nop,wscale 7], length 0
0x0000: 8c85 9048 25eb b827 ebc1 f24d 0800 4500 ...H%..'...M..E.
0x0010: 003c 0000 4000 4006 25ed 0a00 0077 0a00 .<..#.#.%....w..
0x0020: 0059 ed24 ceb3 ca18 0bb9 b105 50be a012 .Y.$........P...
0x0030: 7120 906f 0000 0204 05b4 0402 080a be02 q..o............
0x0040: b3bd 804b ac16 0103 0307 ...K......
12:11:25.312574 IP 10.0.0.89.52915 > hastings.60708: Flags [.], ack 1, win 2058, options [nop,nop,TS val 2152442909 ecr 3187848125], length 0
0x0000: b827 eb94 a718 8c85 9048 25eb 0800 4500 .'.......H%...E.
0x0010: 0034 0000 4000 4006 25f5 0a00 0059 0a00 .4..#.#.%....Y..
0x0020: 0077 ceb3 ed24 b105 50be ca18 0bba 8010 .w...$..P.......
0x0030: 080a 284b 0000 0101 080a 804b ac1d be02 ..(K.......K....
0x0040: b3bd ..
12:11:25.314030 IP 10.0.0.89.52915 > hastings.60708: Flags [P.], seq 1:269, ack 1, win 2058, options [nop,nop,TS val 2152442910 ecr 3187848125], length 268
0x0000: b827 eb94 a718 8c85 9048 25eb 0800 4500 .'.......H%...E.
0x0010: 0140 0000 4000 4006 24e9 0a00 0059 0a00 .#..#.#.$....Y..
0x0020: 0077 ceb3 ed24 b105 50be ca18 0bba 8018 .w...$..P.......
0x0030: 080a 597f 0000 0101 080a 804b ac1e be02 ..Y........K....
0x0040: b3bd 504f 5354 202f 7465 6d70 6572 6174 ..POST./temperat
0x0050: 7572 6520 4854 5450 2f31 2e31 0d0a 486f ure.HTTP/1.1..Ho
0x0060: 7374 3a20 3130 2e30 2e30 2e31 3139 3a36 st:.10.0.0.119:6
0x0070: 3037 3038 0d0a 436f 6e74 656e 742d 5479 0708..Content-Ty
0x0080: 7065 3a20 6170 706c 6963 6174 696f 6e2f pe:.application/
0x0090: 6a73 6f6e 0d0a 436f 6e6e 6563 7469 6f6e json..Connection
0x00a0: 3a20 6b65 6570 2d61 6c69 7665 0d0a 4163 :.keep-alive..Ac
0x00b0: 6365 7074 3a20 6170 706c 6963 6174 696f cept:.applicatio
0x00c0: 6e2f 6a73 6f6e 0d0a 5573 6572 2d41 6765 n/json..User-Age
0x00d0: 6e74 3a20 5049 486f 6d65 2f31 2043 464e nt:.PIHome/1.CFN
0x00e0: 6574 776f 726b 2f31 3132 312e 322e 3120 etwork/1121.2.1.
0x00f0: 4461 7277 696e 2f31 392e 332e 300d 0a43 Darwin/19.3.0..C
0x0100: 6f6e 7465 6e74 2d4c 656e 6774 683a 2032 ontent-Length:.2
0x0110: 3139 0d0a 4163 6365 7074 2d4c 616e 6775 19..Accept-Langu
0x0120: 6167 653a 2065 6e2d 7573 0d0a 4163 6365 age:.en-us..Acce
0x0130: 7074 2d45 6e63 6f64 696e 673a 2067 7a69 pt-Encoding:.gzi
0x0140: 702c 2064 6566 6c61 7465 0d0a 0d0a p,.deflate....
12:11:25.314317 IP 10.0.0.89.52915 > hastings.60708: Flags [P.], seq 269:488, ack 1, win 2058, options [nop,nop,TS val 2152442910 ecr 3187848125], length 219
0x0000: b827 eb94 a718 8c85 9048 25eb 0800 4500 .'.......H%...E.
0x0010: 010f 0000 4000 4006 251a 0a00 0059 0a00 ....#.#.%....Y..
0x0020: 0077 ceb3 ed24 b105 51ca ca18 0bba 8018 .w...$..Q.......
0x0030: 080a e989 0000 0101 080a 804b ac1e be02 ...........K....
0x0040: b3bd 224c 6f67 696e 2073 7563 6365 7373 .."Login.success
0x0050: 6675 6c3a 6579 4a30 6558 4169 4f69 4a71 ful:eyJ0eXAiOiJq
0x0060: 6433 5169 4c43 4a68 6247 6369 4f69 4a49 d3QiLCJhbGciOiJI
0x0070: 557a 4931 4e69 4a39 2e65 794a 7a64 5749 UzI1NiJ9.eyJzdWI
0x0080: 694f 694a 5153 5568 7662 5755 6762 4739 iOiJQSUhvbWUgbG9
0x0090: 6e61 5734 694c 434a 3163 3256 7949 6a6f naW4iLCJ1c2VyIjo
0x00a0: 6963 476c 6f62 3231 6c4c 6e52 6862 6d52 icGlob21lLnRhbmR
0x00b0: 6f51 4752 6862 4756 354c 6e64 7a49 6977 oQGRhbGV5LndzIiw
0x00c0: 6961 5746 3049 6a6f 784e 5467 794d 7a6b iaWF0IjoxNTgyMzk
0x00d0: 784e 4467 314c 434a 6e63 6d39 3163 484d xNDg1LCJncm91cHM
0x00e0: 694f 6c73 6964 4746 755a 4767 6958 5830 iOlsidGFuZGgiXX0
0x00f0: 2e53 6770 6875 4d6a 6347 6f37 6648 6a6a .SgphuMjcGo7fHjj
0x0100: 7878 4736 5569 7770 6c67 754d 645a 6268 xxG6UiwplguMdZbh
0x0110: 6374 2d35 7a44 6344 6e64 4c59 22 ct-5zDcDndLY"
12:11:25.318063 IP hastings.60708 > 10.0.0.89.52915: Flags [.], ack 269, win 235, options [nop,nop,TS val 3187848130 ecr 2152442910], length 0
0x0000: 8c85 9048 25eb b827 ebc1 f24d 0800 4500 ...H%..'...M..E.
0x0010: 0034 4928 4000 4006 dccc 0a00 0077 0a00 .4I(#.#......w..
0x0020: 0059 ed24 ceb3 ca18 0bba b105 51ca 8010 .Y.$........Q...
0x0030: 00eb 2e58 0000 0101 080a be02 b3c2 804b ...X...........K
0x0040: ac1e ..
12:11:25.318064 IP hastings.60708 > 10.0.0.89.52915: Flags [P.], seq 1:280, ack 269, win 235, options [nop,nop,TS val 3187848131 ecr 2152442910], length 279
0x0000: 8c85 9048 25eb b827 ebc1 f24d 0800 4500 ...H%..'...M..E.
0x0010: 014b 4929 4000 4006 dbb4 0a00 0077 0a00 .KI)#.#......w..
0x0020: 0059 ed24 ceb3 ca18 0bba b105 51ca 8018 .Y.$........Q...
0x0030: 00eb cccc 0000 0101 080a be02 b3c3 804b ...............K
0x0040: ac1e 5468 7265 6164 2d31 343a 2050 4f53 ..Thread-14:.POS
0x0050: 5420 2f74 656d 7065 7261 7475 7265 2048 T./temperature.H
0x0060: 5454 502f 312e 310d 0a48 6f73 743a 2031 TTP/1.1..Host:.1
0x0070: 302e 302e 302e 3131 393a 3630 3730 380d 0.0.0.119:60708.
0x0080: 0a43 6f6e 7465 6e74 2d54 7970 653a 2061 .Content-Type:.a
0x0090: 7070 6c69 6361 7469 6f6e 2f6a 736f 6e0d pplication/json.
0x00a0: 0a43 6f6e 6e65 6374 696f 6e3a 206b 6565 .Connection:.kee
0x00b0: 702d 616c 6976 650d 0a41 6363 6570 743a p-alive..Accept:
0x00c0: 2061 7070 6c69 6361 7469 6f6e 2f6a 736f .application/jso
0x00d0: 6e0d 0a55 7365 722d 4167 656e 743a 2050 n..User-Agent:.P
0x00e0: 4948 6f6d 652f 3120 4346 4e65 7477 6f72 IHome/1.CFNetwor
0x00f0: 6b2f 3131 3231 2e32 2e31 2044 6172 7769 k/1121.2.1.Darwi
0x0100: 6e2f 3139 2e33 2e30 0d0a 436f 6e74 656e n/19.3.0..Conten
0x0110: 742d 4c65 6e67 7468 3a20 3231 390d 0a41 t-Length:.219..A
0x0120: 6363 6570 742d 4c61 6e67 7561 6765 3a20 ccept-Language:.
0x0130: 656e 2d75 730d 0a41 6363 6570 742d 456e en-us..Accept-En
0x0140: 636f 6469 6e67 3a20 677a 6970 2c20 6465 coding:.gzip,.de
0x0150: 666c 6174 650d 0a0d 0a flate....
12:11:25.318096 IP 10.0.0.89.52915 > hastings.60708: Flags [.], ack 280, win 2054, options [nop,nop,TS val 2152442914 ecr 3187848131], length 0
0x0000: b827 eb94 a718 8c85 9048 25eb 0800 4500 .'.......H%...E.
0x0010: 0034 0000 4000 4006 25f5 0a00 0059 0a00 .4..#.#.%....Y..
0x0020: 0077 ceb3 ed24 b105 52a5 ca18 0cd1 8010 .w...$..R.......
0x0030: 0806 2546 0000 0101 080a 804b ac22 be02 ..%F.......K."..
0x0040: b3c3 ..
12:11:25.321081 IP hastings.60708 > 10.0.0.89.52915: Flags [P.], seq 280:510, ack 488, win 243, options [nop,nop,TS val 3187848134 ecr 2152442910], length 230
0x0000: 8c85 9048 25eb b827 ebc1 f24d 0800 4500 ...H%..'...M..E.
0x0010: 011a 492a 4000 4006 dbe4 0a00 0077 0a00 ..I*#.#......w..
0x0020: 0059 ed24 ceb3 ca18 0cd1 b105 52a5 8018 .Y.$........R...
0x0030: 00f3 b091 0000 0101 080a be02 b3c6 804b ...............K
0x0040: ac1e 5468 7265 6164 2d31 343a 2022 4c6f ..Thread-14:."Lo
0x0050: 6769 6e20 7375 6363 6573 7366 756c 3a65 gin.successful:e
0x0060: 794a 3065 5841 694f 694a 7164 3351 694c yJ0eXAiOiJqd3QiL
0x0070: 434a 6862 4763 694f 694a 4955 7a49 314e CJhbGciOiJIUzI1N
0x0080: 694a 392e 6579 4a7a 6457 4969 4f69 4a51 iJ9.eyJzdWIiOiJQ
0x0090: 5355 6876 6257 5567 6247 396e 6157 3469 SUhvbWUgbG9naW4i
0x00a0: 4c43 4a31 6332 5679 496a 6f69 6347 6c6f LCJ1c2VyIjoicGlo
0x00b0: 6232 316c 4c6e 5268 626d 526f 5147 5268 b21lLnRhbmRoQGRh
0x00c0: 6247 5635 4c6e 647a 4969 7769 6157 4630 bGV5LndzIiwiaWF0
0x00d0: 496a 6f78 4e54 6779 4d7a 6b78 4e44 6731 IjoxNTgyMzkxNDg1
0x00e0: 4c43 4a6e 636d 3931 6348 4d69 4f6c 7369 LCJncm91cHMiOlsi
0x00f0: 6447 4675 5a47 6769 5858 302e 5367 7068 dGFuZGgiXX0.Sgph
0x0100: 754d 6a63 476f 3766 486a 6a78 7847 3655 uMjcGo7fHjjxxG6U
0x0110: 6977 706c 6775 4d64 5a62 6863 742d 357a iwplguMdZbhct-5z
0x0120: 4463 446e 644c 5922 DcDndLY"
12:11:25.321101 IP 10.0.0.89.52915 > hastings.60708: Flags [.], ack 510, win 2050, options [nop,nop,TS val 2152442917 ecr 3187848134], length 0
0x0000: b827 eb94 a718 8c85 9048 25eb 0800 4500 .'.......H%...E.
0x0010: 0034 0000 4000 4006 25f5 0a00 0059 0a00 .4..#.#.%....Y..
0x0020: 0077 ceb3 ed24 b105 52a5 ca18 0db7 8010 .w...$..R.......
0x0030: 0802 245e 0000 0101 080a 804b ac25 be02 ..$^.......K.%..
0x0040: b3c6 ..
12:11:35.324471 IP 10.0.0.89.52915 > hastings.60708: Flags [F.], seq 488, ack 510, win 2050, options [nop,nop,TS val 2152452893 ecr 3187848134], length 0
0x0000: b827 eb94 a718 8c85 9048 25eb 0800 4500 .'.......H%...E.
0x0010: 0034 0000 4000 4006 25f5 0a00 0059 0a00 .4..#.#.%....Y..
0x0020: 0077 ceb3 ed24 b105 52a5 ca18 0db7 8011 .w...$..R.......
0x0030: 0802 fd64 0000 0101 080a 804b d31d be02 ...d.......K....
0x0040: b3c6 ..
12:11:35.331969 IP hastings.60708 > 10.0.0.89.52915: Flags [F.], seq 510, ack 489, win 243, options [nop,nop,TS val 3187858144 ecr 2152452893], length 0
0x0000: 8c85 9048 25eb b827 ebc1 f24d 0800 4500 ...H%..'...M..E.
0x0010: 0034 492b 4000 4006 dcc9 0a00 0077 0a00 .4I+#.#......w..
0x0020: 0059 ed24 ceb3 ca18 0db7 b105 52a6 8011 .Y.$........R...
0x0030: 00f3 dd58 0000 0101 080a be02 dae0 804b ...X...........K
0x0040: d31d ..
12:11:35.332037 IP 10.0.0.89.52915 > hastings.60708: Flags [.], ack 511, win 2050, options [nop,nop,TS val 2152452899 ecr 3187858144], length 0
0x0000: b827 eb94 a718 8c85 9048 25eb 0800 4500 .'.......H%...E.
0x0010: 0034 0000 4000 4006 25f5 0a00 0059 0a00 .4..#.#.%....Y..
0x0020: 0077 ceb3 ed24 b105 52a6 ca18 0db8 8010 .w...$..R.......
0x0030: 0802 d643 0000 0101 080a 804b d323 be02 ...C.......K.#..
0x0040: dae0 ..
Is this simply a problem with https or is it something deeper?

This might be due to ATS, try to set the following keys in your application Info.plist
NSAppTransportSecurity : Dictionary {
NSAllowsArbitraryLoads : Boolean -> YES
NSAllowsArbitraryLoadsForMedia : Boolean -> YES
NSAllowsArbitraryLoadsInWebContent : Boolean -> YES
NSAllowsLocalNetworking : Boolean -> YES
Find more details on official Preventing Insecure Network Connections

WinRM Negotiate get 200 instead of 401

I am working on a Ruby on Rails application which uses the WinRM library to access a remote Windows server. The transport supplied is :negotiate which will negotiate the authentication with the remote server.
The issue is the WinRM library expects a 401 HTTP status code so that it can send more data for authentication. However, a 200 HTTP status code is returned and the negotiate fails.
The backtrace is :
NoMethodError: undefined method `split' for nil:NilClass
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/winrm-1.7.3/lib/winrm/http/transport.rb:226:in `init_auth'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/winrm-1.7.3/lib/winrm/http/transport.rb:166:in `send_request'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/winrm-1.7.3/lib/winrm/winrm_service.rb:489:in `send_message'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/winrm-1.7.3/lib/winrm/winrm_service.rb:390:in `run_wql'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/winrm-1.7.3/lib/winrm/command_executor.rb:186:in `os_version'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/winrm-1.7.3/lib/winrm/command_executor.rb:145:in `code_page'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/winrm-1.7.3/lib/winrm/command_executor.rb:72:in `block in open'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/winrm-1.7.3/lib/winrm/command_executor.rb:218:in `retryable'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/winrm-1.7.3/lib/winrm/command_executor.rb:71:in `open'
from (irb):20
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/railties-4.2.1/lib/rails/commands/console.rb:110:in `start'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/railties-4.2.1/lib/rails/commands/console.rb:9:in `start'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/railties-4.2.1/lib/rails/commands/commands_tasks.rb:68:in `console'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/railties-4.2.1/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
from /home/cobalt/.rvm/gems/ruby-2.2.2/gems/railties-4.2.1/lib/rails/commands.rb:17:in `<top (required)>'
from bin/rails:4:in `require'
from bin/rails:4:in `<main>'2.2.2 :021 >
The TCP Dump shows the below package exchanges
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
12:04:08.372376 IP d8b5d56cba65.53166 > pokcpeusap02.corp.absc.local.wsman: Flags [S], seq 2899844066, win 29200, options [mss 1460,sackOK,TS val 1316187676 ecr 0,nop,wscale 7], length 0
0x0000: 4500 003c 75fe 4000 4006 d486 0400 0005 E..<u.#.#.......
0x0010: 0acd e165 cfae 1761 acd8 1be2 0000 0000 ...e...a........
0x0020: a002 7210 f065 0000 0204 05b4 0402 080a ..r..e..........
0x0030: 4e73 6e1c 0000 0000 0103 0307 Nsn.........
12:04:08.421019 IP pokcpeusap02.corp.absc.local.wsman > d8b5d56cba65.53166: Flags [S.], seq 3702856093, ack 2899844067, win 8192, options [mss 1351,nop,wscale 8,sackOK,TS val 79780711 ecr 1316187676],
length 0
0x0000: 4500 003c 7f04 4000 7d06 8e80 0acd e165 E..<..#.}......e
0x0010: 0400 0005 1761 cfae dcb5 199d acd8 1be3 .....a..........
0x0020: a012 2000 754e 0000 0204 0547 0103 0308 ....uN.....G....
0x0030: 0402 080a 04c1 5b67 4e73 6e1c ......[gNsn.
12:04:08.421047 IP d8b5d56cba65.53166 > pokcpeusap02.corp.absc.local.wsman: Flags [.], ack 1, win 229, options [nop,nop,TS val 1316187725 ecr 79780711], length 0
0x0000: 4500 0034 75ff 4000 4006 d48d 0400 0005 E..4u.#.#.......
0x0010: 0acd e165 cfae 1761 acd8 1be3 dcb5 199e ...e...a........
0x0020: 8010 00e5 f05d 0000 0101 080a 4e73 6e4d .....]......NsnM
0x0030: 04c1 5b67 ..[g
12:04:08.421368 IP d8b5d56cba65.53166 > pokcpeusap02.corp.absc.local.wsman: Flags [P.], seq 1:340, ack 1, win 229, options [nop,nop,TS val 1316187725 ecr 79780711], length 339
0x0000: 4500 0187 7600 4000 4006 d339 0400 0005 E...v.#.#..9....
0x0010: 0acd e165 cfae 1761 acd8 1be3 dcb5 199e ...e...a........
0x0020: 8018 00e5 f1b0 0000 0101 080a 4e73 6e4d ............NsnM
0x0030: 04c1 5b67 504f 5354 202f 7773 6d61 6e20 ..[gPOST./wsman.
0x0040: 4854 5450 2f31 2e31 0d0a 4175 7468 6f72 HTTP/1.1..Author
0x0050: 697a 6174 696f 6e3a 204e 6567 6f74 6961 ization:.Negotia
0x0060: 7465 2054 6c52 4d54 564e 5455 4141 4241 te.TlRMTVNTUAABA
0x0070: 4141 414e 3449 4934 4151 4142 4141 6741 AAAN4II4AQABAAgA
0x0080: 4141 4141 4141 4141 4351 4141 4142 4462 AAAAAAAACQAAABDb
0x0090: 334a 775a 4468 694e 5751 314e 6d4e 6959 3JwZDhiNWQ1NmNiY
0x00a0: 5459 310d 0a43 6f6e 7465 6e74 2d54 7970 TY1..Content-Typ
0x00b0: 653a 2061 7070 6c69 6361 7469 6f6e 2f73 e:.application/s
0x00c0: 6f61 702b 786d 6c3b 6368 6172 7365 743d oap+xml;charset=
0x00d0: 5554 462d 380d 0a55 7365 722d 4167 656e UTF-8..User-Agen
0x00e0: 743a 2052 7562 7920 5769 6e52 4d20 436c t:.Ruby.WinRM.Cl
0x00f0: 6965 6e74 2028 322e 372e 312c 2072 7562 ient.(2.7.1,.rub
0x0100: 7920 322e 322e 3220 2832 3031 352d 3034 y.2.2.2.(2015-04
0x0110: 2d31 3329 290d 0a41 6363 6570 743a 202a -13))..Accept:.*
0x0120: 2f2a 0d0a 4461 7465 3a20 5475 652c 2030 /*..Date:.Tue,.0
0x0130: 3720 4d61 7220 3230 3137 2031 323a 3034 7.Mar.2017.12:04
0x0140: 3a30 3820 474d 540d 0a43 6f6e 7465 6e74 :08.GMT..Content
0x0150: 2d4c 656e 6774 683a 2030 0d0a 486f 7374 -Length:.0..Host
0x0160: 3a20 706f 6b63 7065 7573 6170 3032 2e63 :.pokcpeusap02.c
0x0170: 6f72 702e 6162 7363 2e6c 6f63 616c 3a35 orp.absc.local:5
0x0180: 3938 350d 0a0d 0a 985....
12:04:08.516497 IP pokcpeusap02.corp.absc.local.wsman > d8b5d56cba65.53166: Flags [P.], seq 1:39, ack 340, win 256, options [nop,nop,TS val 79780721 ecr 1316187725], length 38
0x0000: 4500 005a 7f05 4000 7d06 8e61 0acd e165 E..Z..#.}..a...e
0x0010: 0400 0005 1761 cfae dcb5 199e acd8 1d36 .....a.........6
0x0020: 8018 0100 11f4 0000 0101 080a 04c1 5b71 ..............[q
0x0030: 4e73 6e4d 4854 5450 2f31 2e31 2032 3030 NsnMHTTP/1.1.200
0x0040: 204f 4b0d 0a43 6f6e 7465 6e74 2d4c 656e .OK..Content-Len
0x0050: 6774 683a 2030 0d0a 0d0a gth:.0....
12:04:08.516541 IP d8b5d56cba65.53166 > pokcpeusap02.corp.absc.local.wsman: Flags [.], ack 39, win 229, options [nop,nop,TS val 1316187821 ecr 79780721], length 0
0x0000: 4500 0034 7601 4000 4006 d48b 0400 0005 E..4v.#.#.......
0x0010: 0acd e165 cfae 1761 acd8 1d36 dcb5 19c4 ...e...a...6....
0x0020: 8010 00e5 f05d 0000 0101 080a 4e73 6ead .....]......Nsn.
0x0030: 04c1 5b71 ..[q
What would be the issue? Why I don't get a 401 HTTP status code?

I havae managed to found the root cause of the issue. It turns out there is another service instead of WinRM srevice is listening to port 5985. Hence when a request is sent to that port, that service responded with a respone requiring Basic Authentication and the status code 200. The issue is fixed after starting WinRM service and make it listen at port 5985.
The detailed analysis can be found at Ruby WinRM undefined method `split' for nil:NilClass. It's really a good lesson to learn. Sometimes the issue is very simple and stupid, but to find out the issue would take much effort.

logstash Twitter error when geo_enable=true

I am trying to connect logstash to twitter. All the twitter that user.geo_enabled=false parsed ok, but of the other hand if user.geo_enabled=true I receive this error:
Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"twitter", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0xb1b69ba #metadata_accessors=#<LogStash::Util::Accessors:0x74b553ef #store={}, #lut={}>, #cancelled=false, #data={"created_at"=>"Wed Jul 06 16:15:26 +0000 2016", "id"=>750724847626493953, "id_str"=>"750724847626493953", "text"=>"#HillaryClinton I would never vote for you", "source"=>"Twitter for Android", "truncated"=>false, "in_reply_to_status_id"=>nil, "in_reply_to_status_id_str"=>nil, "in_reply_to_user_id"=>1339835893, "in_reply_to_user_id_str"=>"1339835893", "in_reply_to_screen_name"=>"HillaryClinton", "user"=>{"id"=>772001468, "id_str"=>"772001468", "name"=>"charles c hutchison", "screen_name"=>"49_mail", "location"=>nil, "url"=>nil, "description"=>nil, "protected"=>false, "verified"=>false, "followers_count"=>8, "friends_count"=>99, "listed_count"=>0, "favourites_count"=>8, "statuses_count"=>176, "created_at"=>"Tue Aug 21 18:22:06 +0000 2012", "utc_offset"=>nil, "time_zone"=>nil, "geo_enabled"=>true, "lang"=>"en", "contributors_enabled"=>false, "is_translator"=>false, "profile_background_color"=>"C0DEED", "profile_background_image_url"=>"http://abs.twimg.com/images/themes/theme1/bg.png", "profile_background_image_url_https"=>"https://abs.twimg.com/images/themes/theme1/bg.png", "profile_background_tile"=>false, "profile_link_color"=>"0084B4", "profile_sidebar_border_color"=>"C0DEED", "profile_sidebar_fill_color"=>"DDEEF6", "profile_text_color"=>"333333", "profile_use_background_image"=>true, "profile_image_url"=>"http://abs.twimg.com/sticky/default_profile_images/default_profile_0_normal.png", "profile_image_url_https"=>"https://abs.twimg.com/sticky/default_profile_images/default_profile_0_normal.png", "default_profile"=>true, "default_profile_image"=>true, "following"=>nil, "follow_request_sent"=>nil, "notifications"=>nil}, "geo"=>{"type"=>"Point", "coordinates"=>[#<BigDecimal:6d26d63b,'0.384274583E2',9(12)>, #<BigDecimal:43f314b6,'-0.823958636E2',9(12)>]}, "coordinates"=>{"type"=>"Point", "coordinates"=>[#<BigDecimal:669bf464,'-0.823958636E2',9(12)>, #<BigDecimal:3d160aa5,'0.384274583E2',9(12)>]}, "place"=>{"id"=>"e4197a23034fa912", "url"=>"https://api.twitter.com/1.1/geo/id/e4197a23034fa912.json", "place_type"=>"city", "name"=>"Huntington", "full_name"=>"Huntington, WV", "country_code"=>"US", "country"=>"United States", "bounding_box"=>{"type"=>"Polygon", "coordinates"=>[[[#<BigDecimal:4feaddd3,'-0.82530433E2',8(12)>, #<BigDecimal:5438cd7c,'0.38375981E2',8(12)>], [#<BigDecimal:413b49ac,'-0.82530433E2',8(12)>, #<BigDecimal:58a6101d,'0.38439347E2',8(12)>], [#<BigDecimal:445e692e,'-0.82349236E2',8(12)>, #<BigDecimal:5f332e20,'0.38439347E2',8(12)>], [#<BigDecimal:46c19531,'-0.82349236E2',8(12)>, #<BigDecimal:71e183de,'0.38375981E2',8(12)>]]]}, "attributes"=>{}}, "contributors"=>nil, "is_quote_status"=>false, "retweet_count"=>0, "favorite_count"=>0, "entities"=>{"hashtags"=>[], "urls"=>[], "user_mentions"=>[{"screen_name"=>"HillaryClinton", "name"=>"Hillary Clinton", "id"=>1339835893, "id_str"=>"1339835893", "indices"=>[0, 15]}], "symbols"=>[]}, "favorited"=>false, "retweeted"=>false, "filter_level"=>"low", "lang"=>"en", "timestamp_ms"=>"1467821726124", "#version"=>"1", "#timestamp"=>"2016-07-06T16:15:26.000Z"}, #metadata={}, #accessors=#<LogStash::Util::Accessors:0x40cb306f #store={"created_at"=>"Wed Jul 06 16:15:26 +0000 2016", "id"=>750724847626493953, "id_str"=>"750724847626493953", "text"=>"#HillaryClinton I would never vote for you", "source"=>"Twitter for Android", "truncated"=>false, "in_reply_to_status_id"=>nil, "in_reply_to_status_id_str"=>nil, "in_reply_to_user_id"=>1339835893, "in_reply_to_user_id_str"=>"1339835893", "in_reply_to_screen_name"=>"HillaryClinton", "user"=>{"id"=>772001468, "id_str"=>"772001468", "name"=>"charles c hutchison", "screen_name"=>"49_mail", "location"=>nil, "url"=>nil, "description"=>nil, "protected"=>false, "verified"=>false, "followers_count"=>8, "friends_count"=>99, "listed_count"=>0, "favourites_count"=>8, "statuses_count"=>176, "created_at"=>"Tue Aug 21 18:22:06 +0000 2012", "utc_offset"=>nil, "time_zone"=>nil, "geo_enabled"=>true, "lang"=>"en", "contributors_enabled"=>false, "is_translator"=>false, "profile_background_color"=>"C0DEED", "profile_background_image_url"=>"http://abs.twimg.com/images/themes/theme1/bg.png", "profile_background_image_url_https"=>"https://abs.twimg.com/images/themes/theme1/bg.png", "profile_background_tile"=>false, "profile_link_color"=>"0084B4", "profile_sidebar_border_color"=>"C0DEED", "profile_sidebar_fill_color"=>"DDEEF6", "profile_text_color"=>"333333", "profile_use_background_image"=>true, "profile_image_url"=>"http://abs.twimg.com/sticky/default_profile_images/default_profile_0_normal.png", "profile_image_url_https"=>"https://abs.twimg.com/sticky/default_profile_images/default_profile_0_normal.png", "default_profile"=>true, "default_profile_image"=>true, "following"=>nil, "follow_request_sent"=>nil, "notifications"=>nil}, "geo"=>{"type"=>"Point", "coordinates"=>[#<BigDecimal:6d26d63b,'0.384274583E2',9(12)>, #<BigDecimal:43f314b6,'-0.823958636E2',9(12)>]}, "coordinates"=>{"type"=>"Point", "coordinates"=>[#<BigDecimal:669bf464,'-0.823958636E2',9(12)>, #<BigDecimal:3d160aa5,'0.384274583E2',9(12)>]}, "place"=>{"id"=>"e4197a23034fa912", "url"=>"https://api.twitter.com/1.1/geo/id/e4197a23034fa912.json", "place_type"=>"city", "name"=>"Huntington", "full_name"=>"Huntington, WV", "country_code"=>"US", "country"=>"United States", "bounding_box"=>{"type"=>"Polygon", "coordinates"=>[[[#<BigDecimal:4feaddd3,'-0.82530433E2',8(12)>, #<BigDecimal:5438cd7c,'0.38375981E2',8(12)>], [#<BigDecimal:413b49ac,'-0.82530433E2',8(12)>, #<BigDecimal:58a6101d,'0.38439347E2',8(12)>], [#<BigDecimal:445e692e,'-0.82349236E2',8(12)>, #<BigDecimal:5f332e20,'0.38439347E2',8(12)>], [#<BigDecimal:46c19531,'-0.82349236E2',8(12)>, #<BigDecimal:71e183de,'0.38375981E2',8(12)>]]]}, "attributes"=>{}}, "contributors"=>nil, "is_quote_status"=>false, "retweet_count"=>0, "favorite_count"=>0, "entities"=>{"hashtags"=>[], "urls"=>[], "user_mentions"=>[{"screen_name"=>"HillaryClinton", "name"=>"Hillary Clinton", "id"=>1339835893, "id_str"=>"1339835893", "indices"=>[0, 15]}], "symbols"=>[]}, "favorited"=>false, "retweeted"=>false, "filter_level"=>"low", "lang"=>"en", "timestamp_ms"=>"1467821726124", "#version"=>"1", "#timestamp"=>"2016-07-06T16:15:26.000Z"}, #lut={"in-reply-to"=>[{"created_at"=>"Wed Jul 06 16:15:26 +0000 2016", "id"=>750724847626493953, "id_str"=>"750724847626493953", "text"=>"#HillaryClinton I would never vote for you", "source"=>"Twitter for Android", "truncated"=>false, "in_reply_to_status_id"=>nil, "in_reply_to_status_id_str"=>nil, "in_reply_to_user_id"=>1339835893, "in_reply_to_user_id_str"=>"1339835893", "in_reply_to_screen_name"=>"HillaryClinton", "user"=>{"id"=>772001468, "id_str"=>"772001468", "name"=>"charles c hutchison", "screen_name"=>"49_mail", "location"=>nil, "url"=>nil, "description"=>nil, "protected"=>false, "verified"=>false, "followers_count"=>8, "friends_count"=>99, "listed_count"=>0, "favourites_count"=>8, "statuses_count"=>176, "created_at"=>"Tue Aug 21 18:22:06 +0000 2012", "utc_offset"=>nil, "time_zone"=>nil, "geo_enabled"=>true, "lang"=>"en", "contributors_enabled"=>false, "is_translator"=>false, "profile_background_color"=>"C0DEED", "profile_background_image_url"=>"http://abs.twimg.com/images/themes/theme1/bg.png", "profile_background_image_url_https"=>"https://abs.twimg.com/images/themes/theme1/bg.png", "profile_background_tile"=>false, "profile_link_color"=>"0084B4", "profile_sidebar_border_color"=>"C0DEED", "profile_sidebar_fill_color"=>"DDEEF6", "profile_text_color"=>"333333", "profile_use_background_image"=>true, "profile_image_url"=>"http://abs.twimg.com/sticky/default_profile_images/default_profile_0_normal.png", "profile_image_url_https"=>"https://abs.twimg.com/sticky/default_profile_images/default_profile_0_normal.png", "default_profile"=>true, "default_profile_image"=>true, "following"=>nil, "follow_request_sent"=>nil, "notifications"=>nil}, "geo"=>{"type"=>"Point", "coordinates"=>[#<BigDecimal:6d26d63b,'0.384274583E2',9(12)>, #<BigDecimal:43f314b6,'-0.823958636E2',9(12)>]}, "coordinates"=>{"type"=>"Point", "coordinates"=>[#<BigDecimal:669bf464,'-0.823958636E2',9(12)>, #<BigDecimal:3d160aa5,'0.384274583E2',9(12)>]}, "place"=>{"id"=>"e4197a23034fa912", "url"=>"https://api.twitter.com/1.1/geo/id/e4197a23034fa912.json", "place_type"=>"city", "name"=>"Huntington", "full_name"=>"Huntington, WV", "country_code"=>"US", "country"=>"United States", "bounding_box"=>{"type"=>"Polygon", "coordinates"=>[[[#<BigDecimal:4feaddd3,'-0.82530433E2',8(12)>, #<BigDecimal:5438cd7c,'0.38375981E2',8(12)>], [#<BigDecimal:413b49ac,'-0.82530433E2',8(12)>, #<BigDecimal:58a6101d,'0.38439347E2',8(12)>], [#<BigDecimal:445e692e,'-0.82349236E2',8(12)>, #<BigDecimal:5f332e20,'0.38439347E2',8(12)>], [#<BigDecimal:46c19531,'-0.82349236E2',8(12)>, #<BigDecimal:71e183de,'0.38375981E2',8(12)>]]]}, "attributes"=>{}}, "contributors"=>nil, "is_quote_status"=>false, "retweet_count"=>0, "favorite_count"=>0, "entities"=>{"hashtags"=>[], "urls"=>[], "user_mentions"=>[{"screen_name"=>"HillaryClinton", "name"=>"Hillary Clinton", "id"=>1339835893, "id_str"=>"1339835893", "indices"=>[0, 15]}], "symbols"=>[]}, "favorited"=>false, "retweeted"=>false, "filter_level"=>"low", "lang"=>"en", "timestamp_ms"=>"1467821726124", "#version"=>"1", "#timestamp"=>"2016-07-06T16:15:26.000Z"}, "in-reply-to"], "type"=>[{"created_at"=>"Wed Jul 06 16:15:26 +0000 2016", "id"=>750724847626493953, "id_str"=>"750724847626493953", "text"=>"#HillaryClinton I would never vote for you", "source"=>"Twitter for Android", "truncated"=>false, "in_reply_to_status_id"=>nil, "in_reply_to_status_id_str"=>nil, "in_reply_to_user_id"=>1339835893, "in_reply_to_user_id_str"=>"1339835893", "in_reply_to_screen_name"=>"HillaryClinton", "user"=>{"id"=>772001468, "id_str"=>"772001468", "name"=>"charles c hutchison", "screen_name"=>"49_mail", "location"=>nil, "url"=>nil, "description"=>nil, "protected"=>false, "verified"=>false, "followers_count"=>8, "friends_count"=>99, "listed_count"=>0, "favourites_count"=>8, "statuses_count"=>176, "created_at"=>"Tue Aug 21 18:22:06 +0000 2012", "utc_offset"=>nil, "time_zone"=>nil, "geo_enabled"=>true, "lang"=>"en", "contributors_enabled"=>false, "is_translator"=>false, "profile_background_color"=>"C0DEED", "profile_background_image_url"=>"http://abs.twimg.com/images/themes/theme1/bg.png", "profile_background_image_url_https"=>"https://abs.twimg.com/images/themes/theme1/bg.png", "profile_background_tile"=>false, "profile_link_color"=>"0084B4", "profile_sidebar_border_color"=>"C0DEED", "profile_sidebar_fill_color"=>"DDEEF6", "profile_text_color"=>"333333", "profile_use_background_image"=>true, "profile_image_url"=>"http://abs.twimg.com/sticky/default_profile_images/default_profile_0_normal.png", "profile_image_url_https"=>"https://abs.twimg.com/sticky/default_profile_images/default_profile_0_normal.png", "default_profile"=>true, "default_profile_image"=>true, "following"=>nil, "follow_request_sent"=>nil, "notifications"=>nil}, "geo"=>{"type"=>"Point", "coordinates"=>[#<BigDecimal:6d26d63b,'0.384274583E2',9(12)>, #<BigDecimal:43f314b6,'-0.823958636E2',9(12)>]}, "coordinates"=>{"type"=>"Point", "coordinates"=>[#<BigDecimal:669bf464,'-0.823958636E2',9(12)>, #<BigDecimal:3d160aa5,'0.384274583E2',9(12)>]}, "place"=>{"id"=>"e4197a23034fa912", "url"=>"https://api.twitter.com/1.1/geo/id/e4197a23034fa912.json", "place_type"=>"city", "name"=>"Huntington", "full_name"=>"Huntington, WV", "country_code"=>"US", "country"=>"United States", "bounding_box"=>{"type"=>"Polygon", "coordinates"=>[[[#<BigDecimal:4feaddd3,'-0.82530433E2',8(12)>, #<BigDecimal:5438cd7c,'0.38375981E2',8(12)>], [#<BigDecimal:413b49ac,'-0.82530433E2',8(12)>, #<BigDecimal:58a6101d,'0.38439347E2',8(12)>], [#<BigDecimal:445e692e,'-0.82349236E2',8(12)>, #<BigDecimal:5f332e20,'0.38439347E2',8(12)>], [#<BigDecimal:46c19531,'-0.82349236E2',8(12)>, #<BigDecimal:71e183de,'0.38375981E2',8(12)>]]]}, "attributes"=>{}}, "contributors"=>nil, "is_quote_status"=>false, "retweet_count"=>0, "favorite_count"=>0, "entities"=>{"hashtags"=>[], "urls"=>[], "user_mentions"=>[{"screen_name"=>"HillaryClinton", "name"=>"Hillary Clinton", "id"=>1339835893, "id_str"=>"1339835893", "indices"=>[0, 15]}], "symbols"=>[]}, "favorited"=>false, "retweeted"=>false, "filter_level"=>"low", "lang"=>"en", "timestamp_ms"=>"1467821726124", "#version"=>"1", "#timestamp"=>"2016-07-06T16:15:26.000Z"}, "type"]}>>], :response=>{"create"=>{"_index"=>"twitter", "_type"=>"logs", "_id"=>"AVXA_h0IgT1Xitpna0uT", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Mixing up field types: class org.elasticsearch.index.mapper.core.DoubleFieldMapper$DoubleFieldType != class org.elasticsearch.index.mapper.geo.BaseGeoPointFieldMapper$GeoPointFieldType on field coordinates.coordinates"}}}}, :level=>:warn}
I think that the relevent line is
{"create"=>{"_index"=>"twitter", "_type"=>"logs", "_id"=>"AVXA_h0IgT1Xitpna0uT", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Mixing up field types: class org.elasticsearch.index.mapper.core.DoubleFieldMapper$DoubleFieldType != class
org.elasticsearch.index.mapper.geo.BaseGeoPointFieldMapper$GeoPointFieldType on field coordinates.coordinates"}}}}, :level=>:warn}
my template is :
{
"template": "twitter",
"order": 1,
"settings": {
"number_of_shards": 1
},
"mappings": {
"tweet": {
"_all": {
"enabled": false
},
"properties": {
"coordinates": {
"properties": {
"coordinates": {
"type": "geo_point"
},
"type": {
"type": "string"
}
}
}
}
}
}
}
What can be the problem??
thanks.

The problem is that the field coordinates.coordinates in ES expect a type of data which is not the type received.
A solution would be to modify the template, removing the coordinates type. Then you delete your index and reindex your data. In that case the expected type of coordinates.coordinates will be dependent on the data inserted.
That should resolve the problem.

Resources