Commit graph

1338 commits

Author SHA1 Message Date
Rusty Russell
008bd74c93 renepay: use more formal allocator pattern.
The general pattern for xxx_new is that it should populate all
fields, for encapsulation and so you never can have a half-formed
object.

This means a fair bit of work for now, but it pays off in the next
patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell
ca72a230f3 renepay: fix localmods.
You cannot refresh the gossmap with localmods applied, nor apply localmods
when others have applied localmods in the same process.

There are optimizations we could do, but for now always apply/unapply before
querying gossmap.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell
0a9b3d6740 renepay: remove attempt limit.
Time is what users care about, so remove this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell
b5273610b2 renepay: remove unused all_flows field.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-12 09:41:24 +09:30
Rusty Russell
5f8b77480c plugins: fix error report from bitcoin-cli exec failure.
We've stomped errno, so if exec fails we don't get a reliable result:

```
2023-08-07T17:58:45.713Z **BROKEN** plugin-bcli: bitcoin-cli exec failed: Bad file descriptor
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-09 19:53:47 +09:30
junderw
32b88a2340 Fix: Remove Sync requirements on Futures returned in the Rust plugin library.
See: https://github.com/bitcoindevkit/bdk/issues/1047#issuecomment-1660645669

In general, futures produced by most libraries in the ecosystem of Rust, and bounds placed
on users of famous runtimes like tokio and its spawn method all lack Sync requirements.

Because of this, anyone who creates a callback using any sort of library that returns a
non-Sync future (which most libraries fit this description) inside of it will get some
cryptic error messages (async error messages still leave a lot to be desired).

Removing these Sync requirements will make the library more useful.
2023-08-08 09:45:42 +09:30
Rusty Russell
54bcb10227 lightningd: fix bolt11 parsing in preapproveinvocie, sendonion, listsendpays and renepay
Since bolt11_decode now insists that any `lightning:` prefix be removed, we need
to make sure to use param_invstring not param_string for all bolt11 parameters:

```
2023-08-07T05:55:32.515Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v23.08rc1-21-g0bf5ee6)
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x55dd94934154
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: common/daemon.c:75 (crashdump) 0x55dd949342e6
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x7f5cf5a3bcef
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:44 (__pthread_kill_implementation) 0x7f5cf5a9226b
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:78 (__pthread_kill_internal) 0x7f5cf5a9226b
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:89 (__GI___pthread_kill) 0x7f5cf5a9226b
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ../sysdeps/posix/raise.c:26 (__GI_raise) 0x7f5cf5a3bc45
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./stdlib/abort.c:79 (__GI_abort) 0x7f5cf5a227fb
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./assert/assert.c:92 (__assert_fail_base) 0x7f5cf5a2271a
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: ./assert/assert.c:101 (__GI___assert_fail) 0x7f5cf5a33595
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: common/bolt11.c:734 (bolt11_decode_nosig) 0x55dd94929967
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: common/bolt11.c:953 (bolt11_decode) 0x55dd9492a44f
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: lightningd/pay.c:1730 (json_listsendpays) 0x55dd948d7d72
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:658 (command_exec) 0x55dd948b525b
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:786 (rpc_command_hook_final) 0x55dd948b5876
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:285 (plugin_hook_call_) 0x55dd948f6446
2023-08-07T05:55:32.515Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:874 (plugin_hook_call_rpc_command) 0x55dd948b5c77
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:984 (parse_request) 0x55dd948b6234
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:1090 (read_json) 0x55dd948b670f
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x55dd94ac9bf4
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x55dd94aca823
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x55dd94aca865
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x55dd94accbff
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x55dd948b33c4
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1332 (main) 0x55dd948ba429
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x7f5cf5a2350f
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: ../csu/libc-start.c:381 (__libc_start_main_impl) 0x7f5cf5a235c8
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x55dd94881e74
2023-08-07T05:55:32.516Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
```

Fixes: #6524
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: broken in master since last release.
2023-08-07 18:46:48 +09:30
Rusty Russell
2042b50978 plugins/bcli: update minimum required bitcoind version.
Less than 22 is obsolete anyway, so we should increment this from 16.0 at least!

Closes: #6234
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-07 16:04:14 +09:30
Rusty Russell
84fe3dc940 build: more 32-bit printf fixes.
Reported-by: Shahana
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-07 11:52:40 +09:30
ShahanaFarooqui
18b928046b plugin/clnrest: Downgrading pyln-proto versino from 23.08 to 23.5.2
Bug fix: pip install from plugins/clnrest/requirements.txt was unable to find "not yet released" pyln-proto version 23.08.
2023-08-06 12:34:19 +09:30
Dusty Daemon
7c9f5e9847 renepay: Use macros instead of manual format types
Fixes 32-bit builds:

```
In file included from plugins/renepay/pay.c:5:
./plugins/renepay/pay_flow.h: In function 'fmt_payflow_key':
./plugins/renepay/pay_flow.h:54:17: error: format '%ld' expects argument of type 'long int', but argument 4 has type 'u64' {aka 'long long unsigned int'} [-Werror=format=]
   54 |                 "key: groupid=%ld, partid=%ld, payment_hash=%s",
      |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   55 |                 k->groupid,k->partid,
      |                 ~~~~~~~~~~
      |                  |
      |                  u64 {aka long long unsigned int}
```

etc
2023-08-04 20:33:20 +09:30
Rusty Russell
67e43ea868 renepay: clean up ARC_ADDITIONAL_BITS.
It's expressed in bits, but really it's clearer as a quantity, given
how it's used.

Suggested-by: @Lagrang3
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 11:40:52 +09:30
Rusty Russell
15c8a6f6fe renepay: switch from arc_t to struct arc.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 11:40:52 +09:30
Rusty Russell
b793dc9224 renepay: implement an alternative to a union with bitsets.
This was actually relying on little-endian; if we want this, we need to be explicit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 11:40:52 +09:30
Shahana Farooqui
c96d0c2643 plugin/clnrest: Updated pyln version to 23.08 in requirements.txt 2023-08-01 11:35:19 +09:30
Dusty Daemon
4628e3ace8 channeld: Code to implement splicing
Update the lightningd <-> channeld interface with lots of new commands to needed to facilitate spicing.

Implement the channeld splicing protocol leveraging the interactivetx protocol.

Implement lightningd’s channel_control to support channeld in its splicing efforts.

Changelog-Added: Added the features to enable splicing & resizing of active channels.
2023-07-31 21:00:22 +09:30
Dusty Daemon
50fe819f47 splicing: Add channel state AWAITING_SPLICE
Update gossip routiens and various other hecks on the channel state to consider AWAITING_SPLICE to be routable and treated similar to CHANNELD_NORMAL.

Small updates to psbt interface

Changelog-None
2023-07-31 21:00:22 +09:30
Lagrang3
3024afe524 renepay: more cleanups
- adopt "const <type> *"convention
- remove use_shadow option for some pyln tests
- show prob. information of flows into paynotes
- show prob. of success of entire payment flow in paynotes
- minflow: We were not releasing the memory of flow arrays when replacing
  them with a new canditate.
- use memleak_scan_obj in memleak_check
- replace u64 with size_t

Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Rusty Russell
b5da85e85d plugins/renepay/dijkstra: improve API to remove global.
The global is an *internal* hack because dijkstra_item_mover doesn't
take a context arg!  It should be used with care.

Easy, since all the accessors exist: we just hand in the struct dijkstra.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-31 12:58:56 +09:30
Lagrang3
79486c1e3b renepay: fixups after comments
- remove internal gheap checks
- add check for arc_t.chanidx overflow
- remove outdated comments
- check the delta flow bounds before augmenting along a path
- get_flow_paths uses a dynamic tal array instead of a list.
- fix a unit test that depended on the order of returned flows
- fix bug: lightnind doesn't like if I reuse the partid of a failed
  flow, therefore use a higher partid than any of the previous attempts.
- plugin_err instead of LOG_BROKEN if sendpay fails and we cannot get a
  an error code.
- fix wrong comments.
- remove the background timer.
- This is a bugfix. Previous to this the MCF network was built using the
knowledge of the min and max liquidity but it didn't take into account
pending HTLCs.
- Also remove the min_prob_success option but hardcode a 90% value.

Removing some options that are not relevant to the user, they're kept
for developer mode only:
- base_fee_penalty
- min_prob_success
- prob_cost_factor
- remove heap.h, not used

Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Lagrang3
d46990d301 renepay: add information decay linear in time
Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
2023-07-31 12:58:56 +09:30
Rusty Russell
b8ca193606 renepay: an experimental payment plugin
Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
Changelog-Added: Plugins: `renepay`: an experimental pay plugin implementing Pickhardt payments (`renepay` and `renepaystatus`).
2023-07-31 12:58:56 +09:30
Shahana Farooqui
637f314bc6 plugin/clnrest: Improved readme and comments
- Adding runes info in readme
- Added extra comments in the code
- System exit on shutdown
- Replaced single quotes with double quotes

Changelog-None
2023-07-30 16:50:04 +09:30
Shahana Farooqui
3fab4cc4f3 plugin/clnrest: Updating package
- Adding package initializer for future proofing
- Updated Pyln version
2023-07-30 16:50:04 +09:30
niftynei
73a67fbb3e funder: don't re-reserve utxos on retries
This way unreserving the PSBT will work as intended, and we don't have
to keep track of how many times we've called reserved for any one input.

Technically we're supposed to not reserve inputs at *all* while doing
opens, this moves us slightly closer to that.
2023-07-30 15:20:04 +09:30
Christian Decker
5f2072869a pay: Fix a null-dereference introduced in #6428
The alias may not be set for non-alias channels after they
confirm. The other branch is safe because we only consider active
channels.

Changelog-None
Fixes #6450
2023-07-28 10:06:43 +09:30
Rusty Russell
b2c1ef73cb build: avoid printing PLUGIN_BASES on every build.
Debugging left over in 94205c094f.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-28 09:44:58 +09:30
Rusty Russell
0a727ae6c5 plugins/bcli: plug temporary leak on retry.
Caught by leak detection, we just re-assigned this when we retried: sure,
it's temporary, but it's technically a leak.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:58:38 +09:30
Rusty Russell
7ddebada90 plugins/bcli: fix leak report when bitcoind goes away.
I shut down bitcoind during a test, and bcli leak reports flooded in.
They're all temporary, but this fixes them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:58:38 +09:30
Rusty Russell
620135335d libplugin/pay: allow shortcut for self-pay.
This is the simplest solution, not the best, but there's significant risk in try to remove the "we have a path" assumption in the code pay code.

Includes removing a `tal_steal` which was incorrect: the buffer has the same lifetime as the plugin, so if we steal it then things get messy when we free the  struct payment.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `pay` will now pay your own invoices if you try.
2023-07-26 06:55:48 +09:30
Rusty Russell
981d82c406 common/utils: add tal_strdup_or_null helper.
It's not that uncommon to want to pass NULL through, for optional strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:55:48 +09:30
Rusty Russell
b6d347a6d3 db: add optional column string helper, and make db_col amounts return sat/msat.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-26 06:55:48 +09:30
Shahana Farooqui
911d2c117e plugins/clnrest: Websocket Server
- Added Dependencies
- Updated WS client instructions
- WS Server
2023-07-25 19:20:16 +09:30
Shahana Farooqui
94205c094f plugins/clnrest: Update clnrest dependencies in scripts and Dockerfiles
Adding python3-json5 python3-flask python3-gunicorn flask_restx pyln-client
Also run `pip3 install -r plugins/clnrest/requirements.txt`
2023-07-25 19:20:16 +09:30
Shahana Farooqui
5e3ffc468b plugins/clnrest: removing notifications queue in favor of websocket server 2023-07-25 19:20:16 +09:30
Rusty Russell
2ab0b4af3e plugins/clnrest: disable ourselves unless rest-port is set.
This mirrors grpc's behavior, and avoids listening on ports without
explicit user permission.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 19:20:16 +09:30
Shahana Farooqui
abcfda133f plugin: On-boarding default python plugins
plugin: On-boarding default python plugins
2023-07-25 19:20:16 +09:30
Rusty Russell
a6701bf180 plugins/clnrest: fail gracefully if dependencies not installed.
json and sys are Python built-in packages, so we can do this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 19:20:16 +09:30
Shahana Farooqui
21160aa6a7 plugin: clnrest
plugin: clnrest
2023-07-25 19:20:16 +09:30
Rusty Russell
a219c99691 commando: pass through commands to lightningd.
The in-core commands are much more helpful with details on why runes failed, too!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell
1f791e29eb commando: remove now-unused internal checking routines.
Separate patch to make the previous diff smaller.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell
38075a95d4 commando: use CLN's checkrune() instead of our own for for rune validation.
This means (temporarily) that blacklisting won't work (fix later), and
means that old-style (commando.py) master-secret-override doesn't work.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Plugins: `commando` no longer allows datastore ['commando', 'secret'] to override master secret (re-issue runes if you were using that!).
2023-07-25 14:47:12 +09:30
Rusty Russell
3e4c0103a5 plugins/commando: make struct cond_info self-contained, split try_command()
In preparation for going async:
1. Split try_command's tail into a new function called execute_command() after
   the rune checks have succeeded.
2. Put all the info execute_command() needs into struct cond_info, to make it
   a simple callback style.

So we create new_cond_info() which dynamically allocates `struct cond_info`
and sets the destructor.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell
b16932697d plugins/commando: try_command can reuse incoming struct commando.
We would create a `struct commando` to marshal our incoming messages,
then try_command would create a *new* one.  We can simply reuse, but
when I did I noticed a trick: the new one was not in the `incomings`
array, so didn't work towards the ratelimit.  So we need to remove it
from `incomings` in `try_command`, but at least it's now explicit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell
8db148871d plugins/commando: create new_commando() allocator, use it.
We allocate one in three places, so at least a partial constructor
is a nice pattern to have.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Rusty Russell
cd16690af8 commando: always active.
We used to activate on the first rune creation, but we're no longer in charge
of runes, so we can't make that call.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-25 14:47:12 +09:30
Christian Decker
2549dd6ea5 pay: Annotate both local alias and real scid with channel hints
Debugging a number of payments showed that we sometimes waste a number
of attempts routing through a channel via its alias, rather than its
scid. This is because while we annotate the scid when it has been set,
we do not do so for the alias. The alias then is picked for routing
despite not having enough capacity, failing the attempt locally.

It can also happen that we alternate between scid and alias, doubling
the number of failed attempts before we can make progress.

This patch sets the hint for the alias to a capacity of 0 and disables
it as if the peer were offline. This means when available we'll always
use the scid, which is also far easier to read in the logs.

Changelog-Fixed: pay: We now track spendable amounts when routing on both the local alias as well as the short channel ID
2023-07-24 09:05:40 +09:30
Christian Decker
22462e1d91 libplugin: Remove presplitter from the API 2023-07-21 07:13:48 +09:30
Christian Decker
b768804734 pay: Remove the presplitter modifier
The presplitter modifier would split a payment before trying the first
attempt based on some common sizes. Its goal was to have smaller parts
in flight over different paths, in order to make it more difficult for
a forwarding node to learn payment amount. However it was causing some
issues for direct payments, and estimates on spendable amounts which
considers only the first HTLC being added, but presplitter would
always cause multiple HTLCs to be kicked off, causing the estimate to
be off.

Removing the presplitter fixes this, making draining channels easier,
and worse success rates, due to more HTLCs in flight directly
impacting the changes of getting stuck.

Changelog-Removed: pay: `pay` no longer splits based on common size, as it was causing issues in various scenarios.
2023-07-21 07:13:48 +09:30
Shahana Farooqui
93a3d7f632 commando: added check for empty params
Changelog-Changed: Protocol: commando commands now allow a missing params field, instead of requiring an empty field.

No-schema-diff-check
2023-07-19 05:40:30 +09:30