Commit Graph

58 Commits

Author SHA1 Message Date
yyforyongyu
4db6617d45
multi: skip checking long lines for config files 2022-10-20 23:59:09 +08:00
Eng Zer Jun
889cb0af47
kvdb/etcd: replace defer cleanup with t.Cleanup
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-10-13 17:47:03 +08:00
Oliver Gugger
ee9d6f21bd
etcd: add SequenceKey function 2022-10-13 09:37:52 +02:00
Oliver Gugger
d576184fdb
etcd: extract NewEtcdClient into exported function
In order for us to be able to re-use the code for initializing an etcd
client connection, we extract and export the NewEtcdClient function.
2022-10-13 09:37:52 +02:00
Oliver Gugger
98f359c5b7
etcd: fix typos 2022-10-13 09:37:52 +02:00
Eng Zer Jun
0edc16aa01
kvdb: use T.TempDir to create temporary test directory
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-08-24 09:03:04 +08:00
Andras Banki-Horvath
1093683b47
etcd: set embedded etcd default max msg size and up txn limit 2022-02-28 12:03:05 +01:00
Andras Banki-Horvath
5e90193180
etcd: fix config clone 2022-02-28 12:03:05 +01:00
Andras Banki-Horvath
40af029413
etcd: extend kvdb_etcd with a real ForAll implementation 2022-02-28 12:03:04 +01:00
Joost Jager
b8408a1484
kvdb: add ForAll
A new method to allow efficient range queries for backends that support
it.
2022-01-12 19:11:53 +01:00
Joost Jager
99566b768e
kvdb/postgres: use readonly db transaction if possible 2022-01-12 13:38:56 +01:00
Joost Jager
7cd7cef6c6
htlcswitch: fix duplicate close
The decayed log database opening and closing is managed at a higher
level in config_builder.go.
2022-01-10 08:42:32 +01:00
Joost Jager
0cae55c162
kvdb/postgres: fix context cancellation 2021-12-22 14:54:43 +01:00
Joost Jager
62a2f3c809
kvdb/postgres: fix tests 2021-12-22 14:54:41 +01:00
Andras Banki-Horvath
1ccc5e8014
etcd: make max message size configurable 2021-12-02 21:07:49 +01:00
Jamie Turley
fb43843253 kvdb/bolt_compact: have log message take account space multiplier
[skip ci]
2021-11-21 21:52:30 -08:00
Joost Jager
274faff980
postgres: add connection limit 2021-11-17 11:08:43 +01:00
Oliver Gugger
41c579fd09
multi: run go fmt with go 1.17 2021-11-08 22:39:41 +01:00
Oliver Gugger
97786c72fc
Merge pull request #5878 from halseth/filter-ascii-bbolt-name
kvdb/bolt_compact: filter out non-ASCII from bucket name before printing
2021-11-02 10:22:46 +01:00
Johan T. Halseth
6be6585643
kvdb/bolt_compact: filter out non-ASCII from bucket name before printing
Printing the raw bucket name lead to weird characters and even sounds
being printed.
2021-11-02 08:54:35 +01:00
Johan T. Halseth
10aaa35db5
kvdb: add loggableKeyName method
Safely logs the bucket key name as hex if it includes special
characters.

Co-authored-by: Oliver Gugger <gugger@gmail.com>
2021-11-02 08:54:35 +01:00
Joost Jager
d997bbf6b3
channeldb/test: test with postgres 2021-10-29 10:54:44 +02:00
Joost Jager
7c048efa21
kvdb/postgres/test: single instance embedded postgres database
Prepare for parallel tests that use a postgres backend. We don't want a
high number of embedded postgres instances running simultaneously.
2021-10-29 10:54:42 +02:00
Joost Jager
bd291286f7
kvdb/postgres: convert all types of panic data to error
Previously the assumption existed that panic data was already an error.
If that wasn't the case for example because panic("string") was used,
the transaction wasn't unlocked.
2021-10-29 10:54:40 +02:00
naveen
06433d351e
Upgrade to go 1.16 for the left over packages
Upgraded the go to 1.16 for the rest of the packages.
2021-10-19 03:55:13 +00:00
naveen
d5e3302265
mod : replace gogo/protobuf for GO-2021-0053
The gogo/protobuf has OSV GO-2021-0053
https://deps.dev/advisory/OSV/GO-2021-0053?from=%2Fgo%2Fgithub.com%252Flightningnetwork%252Flnd

This is used indirectly and has to be replaced in go.mod to avoid the
direct reference to that version.

The OSV has been addressed in v1.3.2
2021-10-01 13:07:52 +00:00
Olaoluwa Osuntokun
8ba68ca59f
Merge pull request #5789 from naveensrinivasan/naveen/feat/fix-CVE-2021-29482
mod: Upgraded xz library to FIX the CVE-2021-29482
2021-09-30 20:16:13 -07:00
Oliver Gugger
0b4e03f5fc
multi: add golang 1.17 compatible build tags
With go 1.17 a change to the build flags was implemented:
https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md

The formatter now automatically adds the forward-compatible build tag
format and the linter checks for them, so we need to include them in our
code.
2021-09-29 17:31:37 -07:00
naveen
04f57c6f57
mod: Upgraded xz library to FIX the CVE-2021-29482
Included a replace directive to avoid using an high severity CVE
https://github.com/advisories/GHSA-25xm-hr59-7c27

This library is indirectly referenced and cannot be upgraded directly.

The https://github.com/fergusstrange/embedded-postgres/pull/42 was
merged to fix the CVE issue.
2021-09-29 15:16:26 +00:00
ErikEk
d5463818a8 bbolt sync-freelist ignored 2021-09-27 06:49:56 +01:00
Joost Jager
3eb80cac97
kvdb: add postgres 2021-09-21 10:44:18 +02:00
Oliver Gugger
29a8661517
Merge pull request #5640 from bhandras/kvdb-prefetch
kvdb+channeld: extend `kvdb` with `Prefetch` for prefetching buckets in one go and speed up payment control by prefetching payments on hot paths
2021-09-20 09:42:18 +02:00
naveen
8b4bbfbf61 mod : Replace for OSV GO-2020-0017 JWT
The github.com/dgrijalva/jwt-go is no longer maintained
has this CVE https://github.com/advisories/GHSA-w73w-5m7g-f7qc

But it is being used by https://github.com/etcd-io/etcd/issues/13254
which `lnd` uses.

The https://github.com/golang-jwt/jwt/releases/tag/v3.2.1 is a fork of
the same version which was 3.2.0 with the security fix.
2021-09-16 11:23:28 +00:00
Andras Banki-Horvath
152d1dc36a
kvdb: extra test coverage for kvdb.Prefetch 2021-09-14 13:12:23 +02:00
Andras Banki-Horvath
256b62e0d5
etcd: add kvdb.Prefetch
This commit extends the kvdb interface in a backwards compatible way
such that we'll be able to prefetch all keys in a bucket in one go reducing the
number of roundtrips.
2021-09-14 13:12:22 +02:00
Andras Banki-Horvath
daec66a4e4
etcd: STM to allow prefetching of keys and full ranges
This commit adds more full range caching to the STM such that we'll
be able to prefetch keys and ranges on the kvdb level. This will allow
us to reduce roundtrips to etcd do fast iteration of small ranges.
2021-09-14 13:12:19 +02:00
Andras Banki-Horvath
6c2d8bb176
etcd: enable optional log file for embedded etcd log output
In this commit we add an extra config for enabling logging to an
external file when using embedded etcd. This can be useful when running
integration tests to see more details about etcd related issues.
2021-09-10 14:40:54 +02:00
Andras Banki-Horvath
b29ae94e10
etcd: redesign commit queue to make it more robust and scalable
This commit builds on the ideas of @cfromknecht in lnd/5153. The
addition is that the design is now simpler and more robust by queueing
up everything, but allowing maximal parallelism where txns don't block.
Furthermore the commit makes CommitQueue.Done() private essentially
removing the need to understand the queue externally.
2021-08-06 07:53:13 +02:00
Andras Banki-Horvath
02aa77261d
etcd: fix dereferncing issue in etcd.CommitQueue causing contention
This commit fixes an issue where subsequent transaction retries may have
changed the read/write sets inside the STM which in turn left junk
references to these keys in the CommitQueue. The left keys potentially
conflicted with subsequent transactions, queueing them up causing
througput degradation.
2021-08-06 07:53:10 +02:00
Oliver Gugger
fe931d179f
etcd+channeldb: fix linter issues, rename receiver
With this commit we address some issues the linter picked up after
touching older code.
2021-08-04 14:55:56 +02:00
Oliver Gugger
482f76a0f4
mod+kvdb+channeldb: use btcwallet new DB interface
Depends on btcsuite/btcwallet#757.
Pulls in the updated version of btcwallet and walletdb that have the DB
interface enhanced by their own View() and Update() methods with the
reset callback/closure supported out of the box. That way the global
package-level View() and Update() functions now become pure redirects.
2021-08-04 14:55:55 +02:00
Oliver Gugger
1e84b52fee
lncfg+etcd: add namespace support for etcd databases
Since we're now storing the content of multiple previously distinct
database files in etcd, we want to properly namespace them as not to
provoke any key collisions. We append a sub namespace to the given
global namespace in order to still support multiple lnd nodes using the
same etcd database simultaneously.

Because the btcwallet code uses the legacy walletdb interface we must
assume it is not fully concurrency safe. Therefore we make sure only a
single writer can be active at any given time for the wallet DB backend
when using etcd.
2021-08-04 14:55:54 +02:00
Oliver Gugger
b1b0aac643
kvdb+lncfg: fix some typos in comments 2021-08-04 14:55:48 +02:00
Oliver Gugger
76e1223bf2
mod+kvdb: update etcd to v3.5.0 2021-07-27 13:09:59 +02:00
Oliver Gugger
724ca7a358
kvdb: exclude anything bbolt related from JS builds
Since bbolt uses syscalls for memory mapping that aren't available in
JS/WASM builds, we need to make sure we don't reference that code at
all. Otherwise we can't use parts of lnd as a library in projects that
are being compiled down to a WASM binary.
2021-07-22 09:47:37 +02:00
Oliver Gugger
73aa40f00e
lnwallet+kvdb: remove duplicate bdb backend imports
We only want to register the bbolt DB backend ("bdb") when we're not
compiling for a JS/WASM build targets. That's why we want to have that
import in only one file that we can add a build tag to. We remove it in
two other places since only one import is enough anyway.
2021-07-22 09:47:36 +02:00
Andras Banki-Horvath
bc4ffb489d
etcd: optionally reduce concurrency to single writer for legacy code 2021-07-20 21:02:11 +02:00
Andras Banki-Horvath
96caa6f242
etcd: remove assertion when creating bucket and value with the same key
This commit removes an assertion which is not needed because with etcd
we can safely create keys and values with the same key since they are
stored under different keys in the DB. This saves us one unnecessary Get
on every Put.
2021-07-16 16:06:32 +02:00
Andras Banki-Horvath
11a44a94b1
etcd: remove unnecessary iterator step from cursor Delete
The etcd cursor Delete stepped to the next item in the range before
Delete to not invalidate the iteation. This is unnecessary and not
compatible with bbolt, resulting in an extra fetch too.
2021-07-16 15:21:14 +02:00
Oliver Gugger
8acbe177fa
Merge pull request #5525 from bhandras/etcd_failover_itest_flake_fix
harness: remove killed nodes from active nodes
2021-07-15 17:27:07 +02:00