fix: allow read-only operations when all masters are down#497
Conversation
p0rtale
force-pushed
the
fix/read-only-operations-with-dead-masters
branch
3 times, most recently
from
a489998 to
427fc6a
Compare
p0rtale
force-pushed
the
fix/read-only-operations-with-dead-masters
branch
from
427fc6a to
c93121e
Compare
p0rtale
force-pushed
the
fix/read-only-operations-with-dead-masters
branch
from
c93121e to
3dd8fbe
Compare
| end | ||
|
|
||
| local results, err = call.map(vshard_router, CRUD_LEN_FUNC_NAME, {space_name}, { | ||
| mode = 'write', |
There was a problem hiding this comment.
Otherwise len() will fail if masters are unavailable
There was a problem hiding this comment.
Otherwise
len()will fail if masters are unavailable
may be it's ok? We need to know exact number of tuples on master. No masters -> no result.
you may add ticket for some flag for len method to get len from replicas when master is absent
There was a problem hiding this comment.
Added the read only options (mode, request_timeout, prefer_replica, balance) to allow users to explicitly fetch the length from replicas if needed.
The default behavior is unchanged (mode = 'write'), so len will still fail without masters to guarantee exact data.
There was a problem hiding this comment.
Can you add this change as a separate commit? Because it changes the method options rather than fixing a bug.
Or would it be better to make a separate MR?
There was a problem hiding this comment.
Removed len-related changes from this commit. I'll create a separate PR for them
p0rtale
force-pushed
the
fix/read-only-operations-with-dead-masters
branch
from
3dd8fbe to
6a1fada
Compare
|
|
||
| if res == box.NULL then | ||
| return nil | ||
| for replicaset_id, replicaset in pairs(replicasets) do |
There was a problem hiding this comment.
If we send requests to all replicaset(s), then those requests need to share a common overall timeout. Otherwise, the effective timeout for a request right now is the number of replicasets multiplied by the per-request timeout.
In general, how necessary is it to try to go through all replicasets? What do you think - maybe just callro would be enough?
There was a problem hiding this comment.
also there is a 'request_timeout' option, may be it might be useful
There was a problem hiding this comment.
In general, how necessary is it to try to go through all replicasets? What do you think - maybe just
callrowould be enough?
We iterate through all replicasets to provide a fallback mechanism in case the whole replicaset is down.
| end | ||
|
|
||
| local results, err = call.map(vshard_router, CRUD_LEN_FUNC_NAME, {space_name}, { | ||
| mode = 'write', |
There was a problem hiding this comment.
Can you add this change as a separate commit? Because it changes the method options rather than fixing a bug.
Or would it be better to make a separate MR?
p0rtale
force-pushed
the
fix/read-only-operations-with-dead-masters
branch
2 times, most recently
from
7862077 to
e285f73
Compare
| for replicaset_id, replicaset in pairs(replicasets) do | ||
| local wait_timeout = deadline - fiber_clock() | ||
| if wait_timeout < 0 then | ||
| wait_timeout = 0 |
Read-only operations (get, select, pairs, count, min, max) used to fail with connection errors if all masters in the cluster were unavailable, even when healthy replicas were up and failover hadn't processed yet. To resolve this, the following improvements were made: - Introduced a `read_only` flag to `utils.get_space[s]` to fetch cluster schema from any healthy replica if masters are down. - Updated `get`, `select`, `pairs`, `count`, `min`, `max` to use this new flag. - Rewrote `call.any` to iterate through all replicasets and utilize vshard's `callro` instead of `call` to fetch metadata from replicas.
p0rtale
force-pushed
the
fix/read-only-operations-with-dead-masters
branch
from
e285f73 to
130050e
Compare
3 participants
Read-only operations (get, select, pairs, count, min, max) used to fail with connection errors if all masters in the cluster were unavailable, even when healthy replicas were up and failover hadn't processed yet.
To resolve this, the following improvements were made:
read_onlyflag toutils.get_space[s]to fetch cluster schema from any healthy replica if masters are down.get,select,pairs,count,min,maxto use this new flag.call.anyto iterate through all replicasets and utilize vshard'scallroinstead ofcallto fetch metadata from replicas.Closes TNTP-7102