## How to _clusterize_ your Eclair node Eclair allows you to scale up one _logical_ lightning node across multiple servers. Front servers take care of routing table related gossip and syncing requests from peers, which is cpu/bandwidth intensive. The backend server can focus on core channel management. So, BOLT 1&7 messages are handled in the frontend, while BOLT 2 messages go through and are processed in the backend. Front servers are stateless, they can be stopped/killed at will. The node will remain operational and reachable as long as there is at least one `frontend` available. ``` +---+ +-----------+ | | | +-------+ | | |-----|-| | | P -----| L |-----|-| FRONT |-|---, U | O |-----|-| | | \ B ---| A | | +-------+ | \ L | D | | | \ I | | | +-------+ | \ +------+ C -----| B |-----|-| | | `| | | A |-----|-| FRONT |-|---------| BACK |<-- channels management N | L |-----|-| | | ,| | relay logic E ----| A | | +-------+ | / +------+ T | N | | | / W | C | | +-------+ | / O | E |-----|-| | | / R -----| R |-----|-| FRONT |-|---' K | |-----|-| |<------- connection management | | | +-------+ | routing table sync +---+ +-----------+ ``` The goal is to offload your node from connection and routing table management: - incoming connections - outgoing connections - gossip queries + pings - incoming gossip aggregation - outgoing gossip dispatch (rebroadcast) ### Prerequisite You already have a lightning node up and running in a standalone setup (with Bitcoin Core properly configured, etc.). You know what your `node id` is. Conventions used in this document: - what we previously called `eclair-node` will be called `backend`. It is to be launched, configured and backed-up exactly like in a standalone setup. - `node` refer to *akka cluster nodes*, not to be confused with lightning nodes. Together, all *cluster nodes* form a single logical *lighting node*. ### Minimal/Demo setup Use this if you want to experiment with the cluster mode on a single local server. Set the following values in `.eclair/eclair.conf`: ``` akka.actor.provider = cluster akka.extensions = ["akka.cluster.pubsub.DistributedPubSub"] // replace this with your node id // if you don't know what your node id is, you should probably stop right here eclair.front.pub = 03............... ``` Start the `backend`: ```shell $ ./eclair-node.sh ``` Then run an instance of `frontend`: ```shell $ ./eclair-front.sh -Dakka.remote.artery.canonical.port=25521 -Declair.server.port=9736 ``` NB: we override the ports, otherwise they would conflict since in this example everything runs on the same server. You can run multiple `frontend`s on the same server, just make sure to change the ports. ### Production setup In production you should: - run multiple `frontend` servers - run one app per server - enable `tcp-tls` to encrypt communications between members of the cluster with your own generated certificate (see below) - use a load balancer to hide all your `frontend` servers under the same ip address - set firewall rules to disable lightning connections (port 9735) on your `backend` server, so all connections go through the `frontend` - enable [monitoring](Monitoring.md) - on AWS, use AWS Secrets Manager (see [AWS deployment](#aws-deployment)) #### Enable encrypted communication for the cluster We use a self-signed certificate, which offers a good compromise. More advanced options are available, see [akka doc](https://doc.akka.io/docs/akka/current/remoting-artery.html#remote-security). > Have a single set of keys and a single certificate for all nodes and disable hostname checking > - The single set of keys and the single certificate is distributed to all nodes. The certificate can be self-signed as it is distributed both as a certificate for authentication but also as the trusted certificate. > - If the keys/certificate are lost, someone else can connect to your cluster. > - Adding nodes to the cluster is simple as the key material can be deployed / distributed to the new cluster node. Generate a self-signed certificate (set a strong password): ```shell $ keytool -genkeypair -v \ -keystore akka-cluster-tls.jks \ -dname "O=ACME, C=FR" \ -keypass \ -storepass \ -keyalg RSA \ -keysize 4096 \ -validity 9999 ``` Copy the resulting certificate to the `.eclair` directory on your backend node and all your frontend nodes: ```shell $ cp akka-cluster-tls.jks ~/.eclair ``` Add this to `eclair.conf` on all your frontend nodes: ``` akka.remote.artery.transport = "tls-tcp" ``` #### Run cluster nodes on separate servers Start all your frontend nodes with the following environment variables: * BACKEND_IP set to the IP address of your backend node * LOCAL_IP set to the IP address of this frontend node (this is typically a private IP address, reachable from your backend node) * NODE_PUB_KEY set to your node public key * AKKA_TLS_PASSWORD set to the password of your Akka certificate Add this to `eclair.conf` on your backend node: ``` akka.remote.artery.transport = "tls-tcp" akka.remote.artery.canonical.hostname="ip-of-this-backend-node" akka.cluster.seed-nodes=["akka://eclair-node@ip-of-this-backend-node:25520"] ``` Start your backend node with the following environment variables: * AKKA_TLS_PASSWORD set to the password of your Akka certificate ### AWS Deployment For convenience, we provide a prebuilt AWS Beanstalk bundle for the `frontend` (choose a WebServer environment type, and Java platform). You can run it as-is for testing. #### TLS encryption If you intend to use it in production, you need to enable encryption with your own certificate: 1. Follow the procedure above to generate your `akka-tls.jks` 2. We recommend forking the project and building your own bundle: ```shell $ git clone git@github.com:ACINQ/eclair.git $ vi eclair-core/src/main/reference.conf # set akka.remote.artery.transport = "tls-tcp" $ cp akka-cluster-tls.jks eclair-front/modules/awseb/ # copy the file you generated $ vi eclair-front/modules/awseb.xml # uncomment the relevant parts $ mvn package -DskipTests ``` Alternatively, you can also edit the existing bundle and manually add the `akka-cluster-tls.jks` file to the root of the zip archive. You will also need to set `akka.remote.artery.transport=tls-tcp` at runtime. #### Private key In production, we highly recommend using AWS Secrets manager to provide the node private key. This is done by setting `eclair.front.priv-key-provider=aws-sm`. Default secret name is "node-priv-key", but it is configurable with `eclair.front.aws-sm.priv-key-name` #### Configuration We recommend using Beanstalk environment variables for `AKKA_TLS_PASSWORD`, `BACKEND_IP`, and `NODE_PUB_KEY`. Other configuration keys should be set in the `AKKA_CONF` environment variable, semicolon separated. Example: - `AKKA_CONF`: `eclair.enable-kamon=true; akka.remote.artery.transport=tls-tcp; eclair.front.priv-key-provider=aws-sm...` - `AKKA_TLS_PASSWORD`: `xxxxxxxx` - `BACKEND_IP`: `1.2.3.4` - `NODE_PUB_KEY`: `03933884aaf1d6b108397e5efe5c86bcf2d8ca8d2f700eda99db9214fc2712b134` Port `25520` needs to be open, within the Beanstalk security group, and between the Beanstalk security group and your `backend` node. ### Tor We recommend running your Tor hidden service on a separate server, and use `eclair.tor.targets` to redirect clearnet(*) connections to your `frontend` servers. (*) Clearnet for Tor, but BOLT 8 encrypted. Here is the resulting architecture: ``` +---+ +-----------+ | | | +-------+ | | |-----|-| | | P -----| L |-----|-| FRONT |-|---, U | O |-----|-| | | \ B ---| A | | +-------+ | \ L | D | | | \ I | | | +-------+ | \ +------+ C -------| B |-----|-| | | `| | | A |-----|-| FRONT |-|---------| BACK |<-- channels management N | L |-----|-| | | ,| | relay logic E ----| A | | +-------+ | / +------+ T | N | | | / W +-------+ | C | | +-------+ | / O | | | E |-----|-| | | / R ---| Tor |------| R |-----|-| FRONT |-|---' K | | | |-----|-| |<------- connection management +-------+ | | | +-------+ | routing table sync +---+ +-----------+ ```