In this commit, we fix a race in the `TestPackager` series on channeldb.
A few tests were sharing the same global variable of the set of log
updates, which includes a pointer to an HTLC. The `ExtraData` value of
the HTLC would then be mutated once we go to encode the message on disk.
To fix this, we the global with a function that returns a new instance
of all the test data.
```
==================
WARNING: DATA RACE
Write at 0x0000021b0a48 by goroutine 2896:
github.com/lightningnetwork/lnd/lnwire.(*ExtraOpaqueData).PackRecords()
/home/runner/work/lnd/lnd/lnwire/extra_bytes.go:74 +0x546
github.com/lightningnetwork/lnd/lnwire.EncodeMessageExtraData()
/home/runner/work/lnd/lnd/lnwire/extra_bytes.go:121 +0x4d
github.com/lightningnetwork/lnd/lnwire.(*UpdateAddHTLC).Encode()
/home/runner/work/lnd/lnd/lnwire/update_add_htlc.go:164 +0x5af
github.com/lightningnetwork/lnd/lnwire.WriteMessage()
/home/runner/work/lnd/lnd/lnwire/message.go:330 +0x351
github.com/lightningnetwork/lnd/channeldb.WriteElement()
/home/runner/work/lnd/lnd/channeldb/codec.go:186 +0x1975
github.com/lightningnetwork/lnd/channeldb.WriteElements()
/home/runner/work/lnd/lnd/channeldb/codec.go:247 +0x14f
github.com/lightningnetwork/lnd/channeldb.serializeLogUpdate()
/home/runner/work/lnd/lnd/channeldb/channel.go:2529 +0x3c
github.com/lightningnetwork/lnd/channeldb.putLogUpdate()
/home/runner/work/lnd/lnd/channeldb/forwarding_package.go:525 +0xae
github.com/lightningnetwork/lnd/channeldb.(*ChannelPackager).AddFwdPkg()
/home/runner/work/lnd/lnd/channeldb/forwarding_package.go:489 +0x684
github.com/lightningnetwork/lnd/channeldb_test.TestPackagerOnlyAdds.func1()
/home/runner/work/lnd/lnd/channeldb/forwarding_package_test.go:283 +0x4c
github.com/btcsuite/btcwallet/walletdb/bdb.(*db).Update()
/home/runner/go/pkg/mod/github.com/btcsuite/btcwallet/walletdb@v1.4.2/bdb/db.go:429 +0xe5
github.com/lightningnetwork/lnd/kvdb.Update()
/home/runner/go/pkg/mod/github.com/lightningnetwork/lnd/kvdb@v1.4.6/interface.go:16 +0x258
github.com/lightningnetwork/lnd/channeldb_test.TestPackagerOnlyAdds()
/home/runner/work/lnd/lnd/channeldb/forwarding_package_test.go:282 +0x17b
testing.tRunner()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:1595 +0x238
testing.(*T).Run.func1()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:1648 +0x44
Previous write at 0x0000021b0a48 by goroutine 2898:
github.com/lightningnetwork/lnd/lnwire.(*ExtraOpaqueData).PackRecords()
/home/runner/work/lnd/lnd/lnwire/extra_bytes.go:74 +0x546
github.com/lightningnetwork/lnd/lnwire.EncodeMessageExtraData()
/home/runner/work/lnd/lnd/lnwire/extra_bytes.go:121 +0x4d
github.com/lightningnetwork/lnd/lnwire.(*UpdateAddHTLC).Encode()
/home/runner/work/lnd/lnd/lnwire/update_add_htlc.go:164 +0x5af
github.com/lightningnetwork/lnd/lnwire.WriteMessage()
/home/runner/work/lnd/lnd/lnwire/message.go:330 +0x351
github.com/lightningnetwork/lnd/channeldb.WriteElement()
/home/runner/work/lnd/lnd/channeldb/codec.go:186 +0x1975
github.com/lightningnetwork/lnd/channeldb.WriteElements()
/home/runner/work/lnd/lnd/channeldb/codec.go:247 +0x14f
github.com/lightningnetwork/lnd/channeldb.serializeLogUpdate()
/home/runner/work/lnd/lnd/channeldb/channel.go:2529 +0x3c
github.com/lightningnetwork/lnd/channeldb.putLogUpdate()
/home/runner/work/lnd/lnd/channeldb/forwarding_package.go:525 +0xae
github.com/lightningnetwork/lnd/channeldb.(*ChannelPackager).AddFwdPkg()
/home/runner/work/lnd/lnd/channeldb/forwarding_package.go:489 +0x684
github.com/lightningnetwork/lnd/channeldb_test.TestPackagerAddsThenSettleFails.func1()
/home/runner/work/lnd/lnd/channeldb/forwarding_package_test.go:490 +0x4c
github.com/btcsuite/btcwallet/walletdb/bdb.(*db).Update()
/home/runner/go/pkg/mod/github.com/btcsuite/btcwallet/walletdb@v1.4.2/bdb/db.go:429 +0xe5
github.com/lightningnetwork/lnd/kvdb.Update()
/home/runner/go/pkg/mod/github.com/lightningnetwork/lnd/kvdb@v1.4.6/interface.go:16 +0x2cd
github.com/lightningnetwork/lnd/channeldb_test.TestPackagerAddsThenSettleFails()
/home/runner/work/lnd/lnd/channeldb/forwarding_package_test.go:489 +0x1e7
testing.tRunner()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:1595 +0x238
testing.(*T).Run.func1()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:1648 +0x44
Goroutine 2896 (running) created at:
testing.(*T).Run()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:1648 +0x82a
testing.runTests.func1()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:2054 +0x84
testing.tRunner()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:1595 +0x238
testing.runTests()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:2052 +0x896
testing.(*M).Run()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:1925 +0xb57
github.com/lightningnetwork/lnd/kvdb.RunTests()
/home/runner/go/pkg/mod/github.com/lightningnetwork/lnd/kvdb@v1.4.6/test_utils.go:23 +0x26
github.com/lightningnetwork/lnd/channeldb.TestMain()
/home/runner/work/lnd/lnd/channeldb/setup_test.go:10 +0x308
main.main()
_testmain.go:321 +0x303
Goroutine 2898 (running) created at:
testing.(*T).Run()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:1648 +0x82a
testing.runTests.func1()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:2054 +0x84
testing.tRunner()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:1595 +0x238
testing.runTests()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:2052 +0x896
testing.(*M).Run()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.4.linux-amd64/src/testing/testing.go:1925 +0xb57
github.com/lightningnetwork/lnd/kvdb.RunTests()
/home/runner/go/pkg/mod/github.com/lightningnetwork/lnd/kvdb@v1.4.6/test_utils.go:23 +0x26
github.com/lightningnetwork/lnd/channeldb.TestMain()
/home/runner/work/lnd/lnd/channeldb/setup_test.go:10 +0x308
main.main()
_testmain.go:321 +0x303
==================
```
* mod: bump btcwallet version to accept db timeout
* btcwallet: add DBTimeOut in config
* kvdb: add database timeout option for bbolt
This commit adds a DBTimeout option in bbolt config. The relevant
functions walletdb.Open/Create are updated to use this config. In
addition, the bolt compacter also applies the new timeout option.
* channeldb: add DBTimeout in db options
This commit adds the DBTimeout option for channeldb. A new unit
test file is created to test the default options. In addition,
the params used in kvdb.Create inside channeldb_test is updated
with a DefaultDBTimeout value.
* contractcourt+routing: use DBTimeout in kvdb
This commit touches multiple test files in contractcourt and routing.
The call of function kvdb.Create and kvdb.Open are now updated with
the new param DBTimeout, using the default value kvdb.DefaultDBTimeout.
* lncfg: add DBTimeout option in db config
The DBTimeout option is added to db config. A new unit test is
added to check the default DB config is created as expected.
* migration: add DBTimeout param in kvdb.Create/kvdb.Open
* keychain: update tests to use DBTimeout param
* htlcswitch+chainreg: add DBTimeout option
* macaroons: support DBTimeout config in creation
This commit adds the DBTimeout during the creation of macaroons.db.
The usage of kvdb.Create and kvdb.Open in its tests are updated with
a timeout value using kvdb.DefaultDBTimeout.
* walletunlocker: add dbTimeout option in UnlockerService
This commit adds a new param, dbTimeout, during the creation of
UnlockerService. This param is then passed to wallet.NewLoader
inside various service calls, specifying a timeout value to be
used when opening the bbolt. In addition, the macaroonService
is also called with this dbTimeout param.
* watchtower/wtdb: add dbTimeout param during creation
This commit adds the dbTimeout param for the creation of both
watchtower.db and wtclient.db.
* multi: add db timeout param for walletdb.Create
This commit adds the db timeout param for the function call
walletdb.Create. It touches only the test files found in chainntnfs,
lnwallet, and routing.
* lnd: pass DBTimeout config to relevant services
This commit enables lnd to pass the DBTimeout config to the following
services/config/functions,
- chainControlConfig
- walletunlocker
- wallet.NewLoader
- macaroons
- watchtower
In addition, the usage of wallet.Create is updated too.
* sample-config: add dbtimeout option
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
The explicit `bbolt` dep is gone, as we depend on `kvdb`, which is
actually `walletdb`, which has its own module that defines the proper
`bbolt` version.
In this commit, we migrate all the code in `channeldb` to only reference
the new `kvdb` package rather than `bbolt` directly.
In many instances, we need to add two version to fetch a bucket as both
read and write when needed. As an example, we add a new
`fetchChanBucketRw` function. This function is identical to
`fetchChanBucket`, but it will be used to fetch the main channel bucket
for all _write_ transactions. We need a new method as you can pass a
write transaction where a read is accepted, but not the other way around
due to the stronger typing of the new `kvdb` package.
