* Initialize the database outside the node param constructor
* Do not create folders during StartupSpec
* Simplify syntax for instantiating test Databases
* Rework parameter passing to database initialization
* Force UTF-8 file encoding on all platform.
* Use bitcoin-lib 0.11, which embeds libsecp256k1
* Unit tests: generate dummy sig from 32 random bytes
We now use a version of bitcoin-lib which embeds JNI bindings for libsecp256k1,
and it will only sign data that is 32 bytes long (in Bitcoin and LN you always
sign data hashes, not the actual data).
* Use maven 3.6.0 and a different mirror
* RoutingSyncSpec: don't create databases at init time
We called nodeParams which created a new in-memory sqlite database everytime we created "fake" routing info
* Bitcoin tests: generate 150 blocks instead of 500
We don't need to generate 432 blocks to activate segwit but we still need to have
spendable coins and coinbase maturity is 100 blocks even on regtest.
* Electrum client: test against mainnet Electrum servers
Previous test against testnet servers was flaky because testnet Electrum
servers are unrelable. Here we test against our own server on mainnet (and
2 servers from our list for the pool test).
Bitcoin Core 0.18 is about to enter RC cycle and should be release soon (initial target was April). It is not compatible with 0.16 (some of the RPC calls that we use have been removed. They're still available in 0.17 but tagged as deprecated).
With this PR, eclair will be compatible with 0.17 and the upcoming 0.18, but not with 0.16 any more so it will be a breaking change for some of our users. Supporting the last 2 versions is the right option and we should be ready before 0.18 is actually released (its initial target was April).
* don't spam with channel_updates at startup
Previous logic was very simple but naive:
- every time a channel_update changed we would send it out
- we would always make a new channel_update with the disabled flag set
at startup.
In case our node was simply restarted, this resulted in us re-sending a
channel_update with the disabled flag set, then a second one with the
disabled flag unset a few seconds later, for each public channel.
On top of that, this opened way to a bug: if reconnection is very fast,
then the two successive channel_update will have the same timestamp,
causing the router to not send the second one, which means that the
channel would be considered disabled by the network, and excluded from
payments.
The new logic is as follows:
- when we do NORMAL->NORMAL or NORMAL->OFFLINE or OFFLINE->NORMAL, we
send out the new channel_update if it has changed
- in all other case (e.g. WAIT_FOR_INIT_INTERNAL->OFFLINE) we do nothing
As a side effect, if we were connected to a peer, then we shut down
eclair, then the peer goes down, then we restart eclair: we will make a
new channel_update with the disabled flag set but we won't broadcast it.
If someone tries to make a payment to that node, we will return the
new channel_update with disabled flag set (and maybe the payer will then
broadcast that channel_update). So even in that corner case we are good.
* quick reconnection: bump channel_update timestamp
In case of a disconnection-reconnection, we first generate a
channel_update with disabled bit set, then after we reconnect we
generate a second channel_update with disabled bit not set.
If this happens very quickly, then both channel_updates will have the
same timestamp, and the second one will get ignored by the network.
A simple fix is to bump the second timestamp in this case.
* set channel_update refresh timer at reconnection
We only care about this timer when connected anyway. We also cancel it
when disconnecting.
This has several advantages:
- having a static task resulted in unnecessary refresh if the channel
got disconnected/reconnected in between 2 weeks
- better repartition of the channel_update refresh over time because at
startup all channels were generated at the same time causing all refresh
tasks to be synchronized
- less overhead for the scheduler (because we cancel refresh task for
offline channels (minor, but still)
Use bitcoin-lib v0.10 which has finally been synced to maven central.
Fix transactions unit test (the check in the test was using the whole locktime and not
the last 24 bits).
See https://github.com/ACINQ/bitcoin-lib/pull/31.
We still have to use `Array[Byte]` for low-level cryptographic primitives, and `akka.util.ByteBuffer` for tcp connections. In order to reduce unnecessary copies, we used `ByteVector.view(...)` as much as possible.
Took the opportunity to do a project-wide optimize imports. We might as well do it now since pretty much all files have been touched already.
NB: temporarily use bitcoin-lib 0.10.1-SNAPSHOT because maven central is very slow and we can't access the recently release 0.10 for now.
Add methods to delete channels and tags channels as pruned in batch which is much
more efficient in sqlite.
* Network db: minor changes in unit tests
Test pruning a few 1000s channels at once.
* NetworkDb API: use Iterators and not Seq
It's more consistent with our code base.
There are several separate but related changes in this PR:
(a) Fast close on scenarii where we have nothing at stake (instead of going to `CLOSING` state). The previous process was not only slower (we had to wait for confirmations), but it never resolved when the funding tx hadn't been confirmed. Note that there is still an edge case where the funding tx never gets confirmed, we are fundee and we have something at stake (`push_msat` > 0).
(b) When *fundee*: after a timeout (5 days), if the funding tx hasn't reached `min_depth`, we cancel the channel.
(c) When *funder*: there is no timeout on the funding tx: however on restart, if we detect that our funding tx was doublespent, then we cancel the channel. Just because there is a doublespend doesn't mean that something malicious is going on: e.g. fee was to low, the tx was eventually removed from mempools and we just spent the inputs on something else).
Commits:
* set proper channelid in logs on restore
* fast close if we have nothing at stake
* added fundingTx and timestamp to DATA_WAIT_FOR_FUNDING_CONFIRMED
Also added migration codecs and tests
* implemented funding timeout for fundee
After a given delay, fundee will consider that the funding tx will never
confirm and cancels the channel.
Note that this doesn't apply to the funder, because our implementation
guarantees that we have sent out a funding tx, and the only way to be
sure that it will never be confirmed is that we double spend it. We just
can't rely on a timeout if we want to be safe.
* Electrum: detect if a wallet transaction has been double-spent
If it's in the mempool, or if it's been confirmed, then it's not double spent.
If it's not confirmed and not the mempool, we check if we have a transaction in
our wallet that sspends one of the inputs of our tx. If we find one, then it's been
double spent.
This will work with our funding txs, but not with their funding txs.
* fix regression with dataloss protection
The fast close causes a regression with dataloss protection, because
if we have nothing at stake we won't publish anything in case of
error (even if our peer asks us to).
This fixes#854.
* Add route-weight-ratios to SendPayment/RouteRequest
* Update test channel_update with real world fee values
* Add maxFeeBase and maxFeePct to SendCommand, use high fee tolerance in integration test
* Expose randomized route selection feature in SendPayment, used in integration test too
* Add maxCltv to SendPayment/RouteRequest
* Implement boundaries for graph searching with cost, cltv, and size
* Enable searching for routes with size/CLTV/fee limits
* Expose RouteParams in RouteRequest and SendPayment
* If we couldn't find a route on the first attempt, retry relaxing the restriction on the route size
* Avoid returning an empty path, collapse the route not found cases into one
* When retrying to search for a route, relax 'maxCltv'
* Group search params configurations into a block
* Use the returning edges in 'ignoredEdges' when looking for a spur path
* Log path-finding params when receiving a route request
* Enforce weight ratios to be between (0,1]
* Make path-finding heuristics optional
* Upgrade to JDK11
Eclair can be built and used on Oracle JDK 1.8 or OpenJDK 11.
JavaFX is now embedded in eclair-node-gui and does not need to be installed separately.
* Install: update java download links
OpenJDK 11 is now our recommendation. Tell users to download java from https://jdk.java.net/11
* README: Rewrite installation instructions
* Electrum: don't ask for merkle proofs for unconfirmed txs
* Electrum: clear status when we get disconnected and still have pending history transactions
When we get disconnected and have script hashes for which we still have pending connections,
clear the script hash status. When we reconnect we will ask for its history again, this way we
won't miss anything. Since we rotate keys it should not result in heavy traffic (script hashes have
few history items).
* Electrum: represent and persist block heights as 32 bits signed ints
Int.MaxValue is about 40,000 years of block which should be enough, and it will fix the encoding
problem users on testnet when there's a reorg and one of their txs has a height of -1.
Side-note: changing the wallet codec can be done easily: if we cannot read and decode persisted data
we just start with an empty wallet and retrieve all wallet data from electrum servers, and once it's
ready it will be encoded with the new codec and saved.
* Electrum persistence: include a version number
It provides a clean way, when upgrading the app, of choosing whether to keep the same version and start from the
last persisted wallet (if the persistence format has not been changed or is compatible with the old one), or to
change the version and force starting from an empty wallet and downloading all wallet items from Electrum servers.
* ElectrumClient: remove useless buffer
The GUI took a very long time to startup because nodes and channels were
stored in javafx `ObervableList` which doesn't allow random access. We
can't easily use `ObservableMap` because `TableView` do not support
them.
As a workaround, created an `IndexedObservableList` which keeps track of
indices in `ObservableList` so that we can still have fast random
access.
Also, now grouping announcements in `NetworkEvent` instead of sending
them one by one.
* Enable searching for routes with size/CLTV/fee limits
* Expose the RouteParams in RouteRequest
* Expose the RouteParams in SendPayment
* Rename DEFAULT_ROUTE_MAX_LENGTH
* If we couldn't find a route on the first attempt, retry relaxing the restriction on the route size
* Avoid returning an empty path, collapse the route not found cases into one
* When retrying to search for a route, relax 'maxCltv'
* Move the default params for route searching in the conf, refactor together router params into a config class
* Remove max-payment-fee in favor of router.search-max-fee-pct
* Group search params configurations into a block
* Rename config keys for router path-finding
When we are disconnected and we don't have any channel with that peer,
we can stop it and remove its address from db.
We also need to handle the case where we are in `DISCONNECTED` or
`INITIALIZING` state and we are notified that the last channel we had
with that peer just got closed.
Note that this opens a race condition in the switchboard, if we receive
an incoming connection from that same peer right after we stopped it,
and before the switchboard received the `Terminated` event. If that
happens, then `Peer.Connect` or `Peer.OpenChannel` will timeout. We
could also have the switchboard listens to deadletter events, but that
seems a bit over the top.
Also, removed switchboard's map of peers. Instead, we use the actor's
children list, and access peers using the recommended `child()` method.
Now the 'peers request only returns an `Iterable` instead of a `Map`. This
removes the need to watch child actors, and thus removes the race
condition when peers were stopped. As a trade-off, peer lookup is now
in O(log(N)) instead of O(1) (with N being the number of peers), but this
seems acceptable.
Persist a partial view of our wallet to enable faster startup time.
Users will be able to create transactions as soon as we've checked that we're connected to a "good" electrum server. In the unlikely event where they were able to create and try to publish a transaction during the few seconds it took to finish syncing and their utxo was spent (because they were trying to use an unconfirmed utxo that got double spent, or if they spent their utxo themselves from another wallet initialized with the same seed while his one was offline), publishing the tx will fail.
* Electrum: persist wallet state
Wallet state is persisted and reloaded on startup.
* Electrum wallet: fix handling of headers chunk
When we receive a tx that is older than our checkpoints, we download and check
the header chunk that it's in, check it and connect it to our chain.
* Electrum: advertise wallet transactions
Send notifications for all wallet transactions once we're connected and synced
* Electrum: add timestamp to wallet events
Add an optional timestamp to TransactionReceived and TransactionConfidenceChanged, which is the timestamp of the block the tx was confirmed in (if any).
* Electrum client: use a Close message
This will fix concurrency issues where handlers are called when the actor
is already stopped.
Note that balance events are logged at most once every 30s, and only when
the balance actually changes (e.g. won't log if a payment is failed).
Also, only send `AvailableBalanceChanged` when needed.
We were sending this event everytime we sent a `commit_sig`, which is
incorrect because our balance doesn't change if, say, we are signing an
incoming htlc.
Note that we only send this event in `NORMAL` state, not in `SHUTDOWN`
state, because the balance is not really _available_ in the latter.
This includes support for hosting onion services, and connecting to them, which are two separate things:
- Opening an onion service implie interacting with the tor daemon controller, which requires authentication. We support both `SAFECOOKIE` and `HASHEDPASSWORD` authentication mechanisms, with a default to `SAFECOOKIE`. We support v2 and v3 services, with a default to v3 as recommended by the tor project.
- Connecting to onion services requires tunnelling through tor's local SOCKS5 proxy.
Incoming and outgoing tor connections are thus separate matters that needs to be configured independently. A specific documentation has been added to guide users through these steps.
Big thanks to @rorp for doing the heavy lifting on all this!
While it makes sense to exclude from the routing table channels for
which there is a spending tx in the mempool, we shouldn't blame our
peers for considering the channel still opened if the spending tx hasn't
yet been confirmed.
Also, reworked `ValidateResult` with better types. It appears that the
`NonexistingChannel` error wasn't really useful (a malicious peer would
probably point to an existing funding tx, so there is no real difference
between "txid not found" and "invalid script"), so it was replaced by
`InvalidAnnouncement` error which is a more serious offense (punished
by a disconnection, and probably a ban when we implement that sort of
things).
* Make route randomization optional (enabled by default), option is exposed in SendPayment/RouteRequest
* Fix non deterministic behavior in IntegrationTest
We previously assumed that the `gossip_timestamp_filter` only applied to
future messages, so we set `first_timestamp` to 0 in order to have a
pass-all filter.
But BOLT 7 actually says that the remote peer:
> SHOULD send all gossip messages whose timestamp is greater or equal to
first_timestamp, and less than first_timestamp plus timestamp_range
Which means that the way our filter was set, the remote peer would dump
the entire routing table on us.
By setting `first_timestamp` to the current time, we achieve what we
want. The synchronization of older messages is done by sending a
`query_channel_range` and then one or several `query_short_channel_ids`.
* relay to channel with lowest possible balance
Our current channel selection is very simplistic: we relay to the
channel with the largest balance. As time goes by, this leads to all
channels having the same balance.
A better strategy is to relay to the channel which has the smallest
balance but has enough to process the payment. This way we save larger
channels for larger payments, and also on average channels get depleted
one after the other.
* added tests...
...and found bugs!
Note that there is something fishy in BOLT 4, filed a PR:
https://github.com/lightningnetwork/lightning-rfc/pull/538
Also, first try of softwaremill's quicklens lib (in scope test for now)
* minor: fixed typo (h/t @btcontract)
This is a simple mechanism to test our direct peers by sending fake
small payments to them. A probe is considered successful if the peer
replies with an `UnknownPaymentHash` error.
Probing is configurable and disabled by default.
