datastax enterprise - New Solr node in "Active - Joining" state for several days -
we trying add new solr node our cluster:
dc cassandra
- cassandra node 1
dc solr
- solr node 1 <-- new node (actually, replacement old node; followed steps "replacing dead node")
- solr node 2
- solr node 3
- solr node 4
- solr node 5
our cassandra data approximately 962gb. replication factor 1 both dcs. normal new node in "active - joining" state several days? there way know progress?
last week, there time when had kill , restart dse process because began throwing "too many open files" exception. right now, system log full of messages completed compaction/flushing tasks (no errors far).
edit:
the node still in "active - joining" state of moment. it's been week since restarted dse process in node. started monitoring size of solr.data directory yesterday , far haven't seen increase. system.log still filled compacting/flushing messages.
one thing bothers me in opscenter nodes screen (ring/list view), node shown under "cassandra" dc though node solr node. in nodetool status, nodetool ring, , dsetool ring, node listed under correct dc.
edit:
we decided restart bootstrap process scratch deleting data , commitlog directories. unfortunately, during subsequent bootstrap attempt:
- the stream node 3 node 1 (the new node) failed exception: error [stream-out-/] 2014-04-01 01:14:40,887 cassandradaemon.java (line 196) exception in thread thread[stream-out-/,5,main]
- the stream node 4 node 1 never started. last relevant line in node 4's system.log is: streamresultfuture.java (line 116) received streaming plan bootstrap. should have been followed by: prepare completed. receiving 0 files(0 bytes), sending x files(y bytes)
how can force streams retried?
Comments
Post a Comment