Appending to file not showing up on logstash or elasticsearch output - docker

Question

I spun up logstash and elasticsearch docker containers using images from elastic.co. When I append to the file which I have set as my input file I don't see any output from logstash or elasticsearch. This page didn't help much and couldn't find my exact problem on google or stackoverflow.
This is how I started my containers:
docker run \
--name elasticsearch \
-p 9200:9200 \
-p 9300:9300 \
-e "discovery.type=single-node" \
docker.elastic.co/elasticsearch/elasticsearch:6.3.1
docker run \
--name logstash \
--rm -it -v $(pwd)/elk/logstash.yml:/usr/share/logstash/config/logstash.yml \
-v $(pwd)/elk/pipeline.conf:/usr/share/logstash/pipeline/pipeline.conf \
docker.elastic.co/logstash/logstash:6.3.1
This is my pipeline configuration file:
input {
file {
path => "/pepper/logstash-tutorial.log"
}
}
output {
elasticsearch {
hosts => "http://x.x.x.x:9200/"
}
stdout {
codec => "rubydebug"
}
}
Logstash and elasticsearch started fine it seems.
Sample logstash startup output:
[INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x6dc306e7 sleep>"}
[INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:main, :".monitoring-logstash"], :non_running_pipelines=>[]}
[INFO ][org.logstash.beats.Server] Starting server on port: 5044
[INFO ][logstash.inputs.metrics ] Monitoring License OK
[INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Sample elasticsearch startup output:
[INFO ][o.e.c.r.a.AllocationService] [FJImg8Z] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-logstash-6-2018.07.10][0]] ...]).
So when I make changes to logstash-tutorial.log, I don't see any terminal output from logstash or elasticsearch. How to get output or configure logstash and elasticsearch correctly?

Answer 1

The answer is on the same page that you have referred. Take a look at start_position
Choose where Logstash starts initially reading files: at the beginning or at the end. The default behavior treats files like live streams and thus starts at the end. If you have old data you want to import, set this to beginning.
Set the start position as below:
input {
file {
path => "/pepper/logstash-tutorial.log"
start_position => "beginning"
}
}
output {
elasticsearch {
hosts => "http://x.x.x.x:9200/"
}
stdout {
codec => "rubydebug"
}
}