facet.pivot 4+ fields pivot can kill cluster [JIRA: RIAK-3051]

[mogadanez]

single node installation ( dev )
docs in index: 2300

query: some-index?q=:&facet=true&rows=0&facet.pivot=field1,field2,field3
queryTime: 8-9sec

query: some-index?q=:&facet=true&rows=0&facet.pivot=field1,field2,field3,filed4

Internal Server Error

The server encountered an error while processing this request:
{error,
    {throw,
        {"Failed to search",
         "http://localhost:8093/internal_solr/some-index/select",
         {error,req_timedout}},
        [{yz_solr,search,3,[{file,"src/yz_solr.erl"},{line,343}]},
         {yz_wm_search,search,3,[{file,"src/yz_wm_search.erl"},{line,134}]},
         {webmachine_resource,resource_call,3,
             [{file,"src/webmachine_resource.erl"},{line,186}]},
         {webmachine_resource,do,3,
             [{file,"src/webmachine_resource.erl"},{line,142}]},
         {webmachine_decision_core,resource_call,1,
             [{file,"src/webmachine_decision_core.erl"},{line,48}]},
         {webmachine_decision_core,decision,1,
             [{file,"src/webmachine_decision_core.erl"},{line,562}]},
         {webmachine_decision_core,handle_request,2,
             [{file,"src/webmachine_decision_core.erl"},{line,33}]},
         {webmachine_mochiweb,loop,2,
             [{file,"src/webmachine_mochiweb.erl"},{line,72}]}]}}

Problem 1.
after that Solr still run something in background with 100% CPU usage

Problem 2.
restart riak not helps, solr still continue running and start attempt failed

2016-11-23 13:07:16.996 [info] <0.19886.0> Administrative stop
2016-11-23 13:07:17.005 [info] <0.603.0>@yz_app:prep_stop:74 Stopping application yokozuna.
2016-11-23 13:07:33.133 [notice] <0.19249.0>@yz_entropy:iterate_entropy_data:43 Can't ping Solr index <<"test-map-index">> to start iterating over entropy data
2016-11-23 13:07:46.600 [error] <0.6519.0>@yz_solrq_helper:send_solr_ops_for_entries:301 Updating a batch of Solr operations failed for index <<"test-index">> with error {error,{othe
r,{error,req_timedout}}}
.....
2016-11-23 13:08:17.214 [notice] <0.14994.0>@yz_exchange_fsm:handle_info:99 YZ Exchange FSM received a DOWN message from a process it was monitoring.  The received message is: {'DOWN',
#Ref<0.0.7.71704>,process,<0.6580.0>,shutdown}
2016-11-23 13:08:22.484 [error] <0.1967.0> Supervisor yz_general_sup had child yz_events started with yz_events:start_link() at <0.6578.0> exit with reason killed in context shutdown_e
rror
2016-11-23 13:08:22.484 [error] <0.171.0> Supervisor riak_core_eventhandler_sup had child {yz_events,yz_events} started with riak_core_eventhandler_guard:start_link(yz_events, yz_event
s, [], undefined) at <0.6593.0> exit with reason killed in context child_terminated
2016-11-23 13:08:22.485 [error] <0.19999.0> CRASH REPORT Process <0.19999.0> with 0 neighbours exited with reason: no such process or port in call to gen_server:init_it/6 line 328
2016-11-23 13:08:22.485 [error] <0.171.0> Supervisor riak_core_eventhandler_sup had child {yz_events,yz_events} started with riak_core_eventhandler_guard:start_link(yz_events, yz_event
s, [], undefined) at <0.6593.0> exit with reason noproc in context start_error
2016-11-23 13:08:22.486 [error] <0.171.0> Supervisor riak_core_eventhandler_sup had child {yz_events,yz_events} started with riak_core_eventhandler_guard:start_link(yz_events, yz_event
s, [], undefined) at {restarting,<0.6593.0>} exit with reason noproc in context start_error
2016-11-23 13:08:22.486 [error] <0.20007.0> CRASH REPORT Process <0.20007.0> with 0 neighbours exited with reason: no such process or port in call to gen_server:init_it/6 line 328
....
2016-11-23 13:08:27.536 [info] <0.603.0>@yz_app:stop:88 Stopped application yokozuna.
2016-11-23 13:08:27.536 [error] <0.1964.0> Supervisor yz_sup had child yz_solr_sup started with yz_solr_sup:start_link() at <0.1965.0> exit with reason killed in context shutdown_error
2016-11-23 13:08:27.543 [info] <0.338.0>@riak_kv_app:prep_stop:228 Stopping application riak_kv - marked service down.
....

2016-11-23 13:08:47.495 [info] <0.6627.0>@yz_solr_proc:init:119 Starting solr: "/usr/bin/java" ["-Djava.awt.headless=true","-Djetty.home=/usr/lib/riak/lib/yokozuna-2.1.7-0-g6cf80ad/pri
2016-11-23 13:08:47.495 [info] <0.208.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_control,member_info_version} = v1
2016-11-23 13:08:47.571 [info] <0.6627.0>@yz_solr_proc:handle_info:184 solr stdout/err: OpenJDK 64-Bit Server VM warning: ignoring option UseStringCache; support was removed in 8.0

2016-11-23 13:08:47.803 [info] <0.6627.0>@yz_solr_proc:handle_info:184 solr stdout/err: Error: Exception thrown by the agent
2016-11-23 13:08:47.803 [info] <0.6627.0>@yz_solr_proc:handle_info:184 solr stdout/err: : java.rmi.server.ExportException: Port already in use: 8985; nested exception is:
        java.net.BindException: Address already in use (Bind failed)
2016-11-23 13:08:47.804 [info] <0.6627.0>@yz_solr_proc:handle_info:184 solr stdout/err:

2016-11-23 13:08:48.122 [warning] <0.6.0> lager_error_logger_h dropped 15 messages in the last second that exceeded the limit of 100 messages/sec
2016-11-23 13:08:48.122 [error] <0.6627.0> gen_server yz_solr_proc terminated with reason: {"solr OS process exited",1}
....
2016-11-23 13:08:48.183 [info] <0.606.0>@yz_app:stop:88 Stopped application yokozuna.
....
2016-11-23 13:08:48.202 [info] <0.340.0>@riak_kv_app:stop:250 Stopped  application riak_kv.
...
2016-11-23 13:08:48.490 [info] <0.166.0>@riak_core_app:stop:116 Stopped  application riak_core.

only way is hard kill yokozuna process