What is causing CouchDB changes_reader_died - erlang

Question

I'm trying to configure a filtered replication bidirectionnal between two databases. Each database has a document into the _replicator database that set the replication towards the database. Each database have the same _design document with the filter and they got the same filter parameters.
Although I have been looking over the web, I didn't find the cause of my problem. I hope you guys can help me.
Log output :
[Fri, 26 Aug 2016 19:36:31 GMT] [error] [<0.22247.80>] ** Generic server <0.22247.80> terminating
** Last message in was {'EXIT',<0.22219.80>,changes_reader_died}
** When Server state == {state,<0.22219.80>,<0.22249.80>,20,
{httpdb,
"REPLACEDFORSECURITYREASONS",
nil,
[{"Accept","application/json"},
{"User-Agent","CouchDB/1.6.1"}],
30000,
[{socket_options,
[{keepalive,true},{nodelay,false}]}],
10,250,<0.22065.80>,20},
{httpdb,
"http:REPLACEDFORSECURITYREASONS",
nil,
[{"Accept","application/json"},
{"User-Agent","CouchDB/1.6.1"}],
30000,
[{socket_options,
[{keepalive,true},{nodelay,false}]}],
10,250,<0.22223.80>,20},
[],nil,nil,nil,
{rep_stats,0,0,0,0,0},
nil,nil,
{batch,[],0}}
** Reason for termination ==
** changes_reader_died
[Fri, 26 Aug 2016 19:36:31 GMT] [error] [<0.22243.80>] {error_report,<0.34.0>,
{<0.22243.80>,crash_report,
[[{initial_call,
{couch_replicator_worker,init,['Argument__1']}},
{pid,<0.22243.80>},
{registered_name,[]},
{error_info,
{exit,changes_reader_died,
[{gen_server,terminate,6,
[{file,"gen_server.erl"},{line,744}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}},
{ancestors,
[<0.22219.80>,couch_replicator_job_sup,
couch_primary_services,couch_server_sup,
<0.35.0>]},
{messages,[]},
{links,[<0.22244.80>]},
{dictionary,
[{last_stats_report,{1472,240191,741233}}]},
{trap_exit,true},
{status,running},
{heap_size,233},
{stack_size,27},
{reductions,158}],
[{neighbour,
[{pid,<0.22244.80>},
{registered_name,[]},
{initial_call,{erlang,apply,2}},
{current_function,
{couch_replicator_worker,queue_fetch_loop,5}},
{ancestors,[]},
{messages,[]},
{links,[<0.22243.80>]},
{dictionary,[]},
{trap_exit,false},
{status,waiting},
{heap_size,610},
{stack_size,10},
{reductions,4}]}]]}}
[Fri, 26 Aug 2016 19:36:31 GMT] [error] [<0.22247.80>] {error_report,<0.34.0>,
{<0.22247.80>,crash_report,
[[{initial_call,
{couch_replicator_worker,init,['Argument__1']}},
{pid,<0.22247.80>},
{registered_name,[]},
{error_info,
{exit,changes_reader_died,
[{gen_server,terminate,6,
[{file,"gen_server.erl"},{line,744}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}},
{ancestors,
[<0.22219.80>,couch_replicator_job_sup,
couch_primary_services,couch_server_sup,
<0.35.0>]},
{messages,[]},
{links,[<0.22249.80>]},
{dictionary,
[{last_stats_report,{1472,240191,741235}}]},
{trap_exit,true},
{status,running},
{heap_size,233},
{stack_size,27},
{reductions,162}],
[{neighbour,
[{pid,<0.22249.80>},
{registered_name,[]},
{initial_call,{erlang,apply,2}},
{current_function,
{couch_replicator_worker,queue_fetch_loop,5}},
{ancestors,[]},
{messages,[]},
{links,[<0.22247.80>]},
{dictionary,[]},
{trap_exit,false},
{status,waiting},
{heap_size,610},
{stack_size,10},
{reductions,4}]}]]}}
Best regards.

Answer 1

While doing a deep analyze of the log, I found that there was a timeout error before the changes_read_died.
The error was pretty explicit :
Fri, 26 Aug 2016 20:01:06 GMT] [info] [<0.2238.0>] Replication `"c48f6c26aa44689de43ee5ffaa18c7ad+continuous"` is using:
4 worker processes
a worker batch size of 500
20 HTTP connections
a connection timeout of 30000 milliseconds
10 retries per request
socket options are: [{keepalive,true},{nodelay,false}]
[Fri, 26 Aug 2016 20:01:06 GMT] [info] [<0.2204.0>] XXX.XXX.X.XX- - GET /akpaper/_changes?filter=global%2FbySite&IDSITE=MILLS2&feed=continuous&style=all_docs&since=0&heartbeat=10000 200
[Fri, 26 Aug 2016 20:01:06 GMT] [error] [<0.2204.0>] OS Process Error <0.204.0> :: {<<"compilation_error">>,
<<"Expression does not eval to a function. (ffunction(doc, req) { return doc._deleted || doc.IDSITE == req.query.IDSITE;})">>}
[Fri, 26 Aug 2016 20:01:06 GMT] [info] [<0.2204.0>] XXX.XXX.X.XX - - GET /akpaper/_changes?filter=global%2FbySite&IDSITE=MILLS2&feed=continuous&style=all_docs&since=0&heartbeat=10000 500
[Fri, 26 Aug 2016 20:01:06 GMT] [error] [<0.2204.0>] httpd 500 error response:
{"error":"compilation_error","reason":"Expression does not eval to a function. (ffunction(doc, req) { return doc._deleted || doc.IDSITE == req.query.IDSITE;})"}
All I did to correct the problem was just to correct my filter function wich could not be eval.

Answer 2

for me, the "changes" parser was failing, hence the message: changes_reader_dies
Looking at the replication doc, it seems that it setting the Content-Type to JSON and somehow failing to parse changes. Removing that line fixed it for me:
Bad:
{
"_id": "repl/...",
"_rev": "8-76d8a4dd87f5911b5dccb2527b2304",
"source": {
"url": "
"headers": {
"Authorization": "Basic FaBNwB...",
"Content-Type": "application/json"
}
"target": ...
...
}
Good:
{
"_id": "repl/...",
"_rev": "8-76d8a4dd87f5911b5dccb2527b2304",
"source": {
"url": "
"headers": {
"Authorization": "Basic FaBNwB..."
}
"target": ...
...
}