If the tower returns CreateSessionCodeAlreadyExists in response to the
CreateSession message from the client, then skip forward a few key
indices until we find one that the server does not return the error
for. This will allow a client to recover after a data loss incident.
This commit adds a forceNext boolean parameter to NextSessionKeyIndex.
Setting this param to true will force the index to cycle over 1000 key
indices before returning the new key.
In this commit, the bug demonstrated in the previous commit is fixed.
The locking capabilities of the AddressIterator are used to lock
addresses if they are being used for session negotiation. So now, when a
request comes through to remove a tower address then a check is first
done to ensure that the address is not currently in use. If it is not,
then the request can go through.
This commit upgrades the wtclient package to make use of the new
`AddressIterator`. It does so by first creating new `Tower` and
`ClientSession` types. The new `Tower` type has an `AddressIterator`
instead of a list of addresses. The `ClientSession` type contains a
`Tower`.
This commit introduces a change in the key format used to reserve/lookup
session-key-indexes. Currently the reservations are stored under the
tower id, however this creates issues when multiple clients are using
the same database since only one reservation is permitted per tower.
We fix this by appending the blob type to the session-key-index locator.
This allows multiple clients to reserve keys for the same tower, but
still limits each client to one outstanding reservation. The changes are
made in a way such that we fall back to the legacy format if the a
reservation under the new format is not found, but only if the blob type
matches blob.TypeAltruistCommit, which is so far the only actively
deployed blob type.
Currently if the tower hangs up during session negotiation there is no
backoff applied. We add backoff here to avoid excessive CPU/network
utilization during unexpected failures.
