core-lightning/doc/dev/contributors/codegen.md

Code Generation

The CLN project has a multitude of interfaces, most of which are generated from an abstract schema:

• Wire format for peer-to-peer communication: this is the binary format that is specific by the LN spec. It uses the generate-wire.py script to parse the (faux) CSV files that are automatically extrated from the specification and writes C source code files that are then used internally to encode and decode messages, as well as provide print functions for the messages.

• Wire format for inter-daemon communication: CLN follows a multi-daemon architecture, making communication explicit across daemons. For this inter-daemon communication we use a slightly altered message format from the LN spec. The changes are 1) addition of FD passing semantics to allow establishing a new connection between daemons (communication uses socketpairs, so no connect), and 2) change the message length prefix from u16 to u32, allowing for messages larger than 65Kb. The CSV files are with the respective sub-daemon and also use generate-wire.py to generate encoding, decoding and printing functions.

• We describe the JSON-RPC using JSON Schema in the doc/schemas directory. Each method has a .request.json for the request message, and a .schema.json for the response (the mismatch is historical and will eventually be addressed). During tests the pytest target will verify responses, however the JSON-RPC methods are not generated (yet?). We do generate various client stubs for languages, using the msggen tool. More on the generated stubs and utilities below.

Man pages

The manpages are partially generated from the JSON schemas using the fromschema tool. It reads the request schema and fills in the manpage between two markers:

[comment]: # (GENERATE-FROM-SCHEMA-START)
...
[comment]: # (GENERATE-FROM-SCHEMA-END)

!!! note

Some of this functionality overlaps with [`msggen`][msggen] (parsing the Schemas)
and [blockreplace.py][blockreplace.py] (filling in the template). It
is likely that this will eventually be merged.

msggen

msggen is used to generate JSON-RPC client stubs, and converters between in-memory formats and the JSON format. In addition, by chaining some of these we can expose a grpc interface that matches the JSON-RPC interface. This conversion chain is implemented in the grpc-plugin

```mermaid graph LR A[JSON schema]; A --> B[cln-rpc]; B --> B1[Request Structs]; B --> B2[Response Structs]; B --> B3[Method stubs];

A --> C[cln-grpc];
C --> C1[Protobuf File];
C --> C2[In-memory conversion];
C --> C3[Service Implementation];

<figcaption>Artifacts generated from the JSON Schemas using `msggen`</figcaption>
</figure>

### `cln-rpc`

We use `msggen` to generate the Rust bindings crate
[`cln-rpc`][cln-rpc]. These bindings contain the stubs for the
JSON-RPC methods, as well as types for the request and response
structs. The [generator code][cln-rpc-gen] maps each abstract JSON-RPC
type to a Rust type, minimizing size (e.g., binary data is
hex-decoded).

The calling pattern follows the `call(req_obj) -> resp_obj` format,
and the individual arguments are not expanded. For more ergonomic
handling of generic requests and responses we also define the
`Request` and `Response` enumerations, so you can hand them to a
generic function without having to resort to dynamic dispatch.

The remainder of the crate implements an async/await JSON-RPC client,
that can deal with the Unix Domain Socket [transport][man:json-rpc]
used by CLN.

### `cln-grpc`

The `cln-grpc` crate is mostly used to provide the primitives to build
the `grpc-plugin`. As mentioned above, the grpc functionality relies on a chain of generated parts:

 - First `msggen` is used to generate the [protobuf file][proto],
   containing the service definition with the method stubs, and the types
   referenced by those stubs.
 - Next it generates the `convert.rs` file which is used to convert
   the structs for in-memory representation from `cln-rpc` into the
   corresponding protobuf structs.
 - Finally `msggen` generates the `server.rs` file which can be bound
   to a grpc endpoint listening for incoming grpc requests, and it
   will convert the request and forward it to the JSON-RPC. Upon
   receiving the response it gets converted back into a grpc response
   and sent back.

```mermaid
graph LR
	A[grpc client] --> B[grpc server] -->|convert.rs| C[cln-rpc] --> D[lightningd];
	D --> C -->|convert.rs| B --> A;