mirror of
https://github.com/ElementsProject/lightning.git
synced 2025-01-18 21:35:11 +01:00
doc: Schema generation instructions update
This commit is contained in:
ShahanaFarooqui
Rusty Russell
parent
2c99f2e100
commit
7e33a10425
@ -3,7 +3,7 @@ title: "Code Generation"
|
||||
slug: "code-generation"
|
||||
hidden: false
|
||||
createdAt: "2023-04-22T12:29:01.116Z"
|
||||
updatedAt: "2023-04-22T12:44:47.814Z"
|
||||
updatedAt: "2024-01-18T12:44:47.814Z"
|
||||
---
|
||||
The CLN project has a multitude of interfaces, most of which are generated from an abstract schema:
|
||||
|
||||
@ -13,19 +13,11 @@ The CLN project has a multitude of interfaces, most of which are generated from
|
||||
1. addition of FD passing semantics to allow establishing a new connection between daemons (communication uses [socketpair](https://man7.org/linux/man-pages/man2/socketpair.2.html), so no `connect`)
|
||||
2. change the message length prefix from `u16` to `u32`, allowing for messages larger than 65Kb. The CSV files are with the respective sub-daemon and also use [generate-wire.py](https://github.com/ElementsProject/lightning/blob/master/tools/generate-wire.py) to generate encoding, decoding and printing functions
|
||||
|
||||
- We describe the JSON-RPC using [JSON Schema](https://json-schema.org/) in the [`doc/schemas`](https://github.com/ElementsProject/lightning/tree/master/doc/schemas) directory. Each method has a `.request.json` for the request message, and a `.schema.json` for the response (the mismatch is historical and will eventually be addressed). During tests the `pytest` target will verify responses, however the JSON-RPC methods are _not_ generated (yet?). We do generate various client stubs for languages, using the `msggen`][msggen] tool. More on the generated stubs and utilities below.
|
||||
- We describe the JSON-RPC using [JSON Schema](https://json-schema.org/) in the [`doc/schemas`](https://github.com/ElementsProject/lightning/tree/master/doc/schemas) directory. Each method has a `lightning-*.json` for request and response. During tests the `pytest` target will verify responses, however the JSON-RPC methods are _not_ generated (yet?). We do generate various client stubs for languages, using the `msggen`[msggen] tool. More on the generated stubs and utilities below.
|
||||
|
||||
## Man pages
|
||||
|
||||
The manpages are partially generated from the JSON schemas using the [`fromschema`](https://github.com/ElementsProject/lightning/blob/master/tools/fromschema.py) tool. It reads the request schema and fills in the manpage between two markers:
|
||||
|
||||
```markdown
|
||||
[comment]: # (GENERATE-FROM-SCHEMA-START)
|
||||
...
|
||||
[comment]: # (GENERATE-FROM-SCHEMA-END)
|
||||
```
|
||||
|
||||
|
||||
The manpages are generated from the JSON schemas using the [`fromschema`](https://github.com/ElementsProject/lightning/blob/master/tools/fromschema.py) tool. It reads the request and response schema from `lightning-*.json` and generates markdown contents and manpages:
|
||||
|
||||
> 📘
|
||||
>
|
||||
@ -51,10 +43,6 @@ The manpages are partially generated from the JSON schemas using the [`fromschem
|
||||
}
|
||||
[/block]
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### `cln-rpc`
|
||||
|
||||
We use `msggen` to generate the Rust bindings crate [`cln-rpc`](https://github.com/ElementsProject/lightning/tree/master/cln-rpc). These bindings contain the stubs for the JSON-RPC methods, as well as types for the request and response structs. The [generator code](https://github.com/ElementsProject/lightning/blob/master/contrib/msggen/msggen/gen/rust.py) maps each abstract JSON-RPC type to a Rust type, minimizing size (e.g., binary data is hex-decoded).
|
||||
|
||||
@ -3,36 +3,24 @@ title: "Writing JSON Schemas"
|
||||
slug: "writing-json-schemas"
|
||||
hidden: false
|
||||
createdAt: "2023-01-25T05:46:43.718Z"
|
||||
updatedAt: "2023-01-30T15:36:28.523Z"
|
||||
updatedAt: "2024-01-18T15:36:28.523Z"
|
||||
---
|
||||
A JSON Schema is a JSON file which defines what a structure should look like; in our case we use it in our testsuite to check that they match command responses, and also use it to generate our documentation.
|
||||
A JSON Schema is a JSON file which defines what a structure should look like; in our case we use it in our testsuite to check that they match command requests and responses, and also use it to generate our documentation.
|
||||
|
||||
Yes, schemas are horrible to write, but they're damn useful. We can only use a subset of the full [JSON Schema Specification](https://json-schema.org/), but if you find that limiting it's probably a sign that you should simplify your JSON output.
|
||||
|
||||
## Updating a Schema
|
||||
|
||||
If you add a field, you should add it to the schema, and you must add "added": "VERSION" (where VERSION is the next release version!).
|
||||
If you add a field, you should add it to the field schema, and you must add "added": "VERSION" (where VERSION is the next release version!).
|
||||
|
||||
Similarly, if you deprecate a field, add "deprecated": "VERSION" (where VERSION is the next release version). They will be removed two versions later.
|
||||
Similarly, if you deprecate a field, add "deprecated": "VERSION" (where VERSION is the next release version) to the field. They will be removed two versions later.
|
||||
|
||||
## How to Write a Schema
|
||||
|
||||
Name the schema doc/schemas/`command`.schema.json: the testsuite should pick it up and check all invocations of that command against it.
|
||||
Name the schema doc/schemas/lightning-`command`.json: the testsuite should pick it up and check all invocations of that command against it.
|
||||
The core lightning RPC commands use custom schema specification defined in [rpc-schema-draft](https://github.com/ElementsProject/lightning/doc/rpc-schema-draft.json).
|
||||
|
||||
I recommend copying an existing one to start.
|
||||
|
||||
You will need to put the magic lines in the manual page so `make doc-all` will fill it in for you:
|
||||
|
||||
```json
|
||||
[comment]: # (GENERATE-FROM-SCHEMA-START)
|
||||
[comment]: # (GENERATE-FROM-SCHEMA-END)
|
||||
```
|
||||
|
||||
|
||||
|
||||
If something goes wrong, try tools/fromscheme.py doc/schemas/`command`.schema.json to see how far it got before it died.
|
||||
|
||||
You should always use `"additionalProperties": false`, otherwise your schema might not be covering everything. Deprecated fields simply have `"deprecated": true` in their properties, so they are allowed by omitted from the documentation.
|
||||
I recommend copying an existing one to start. If something goes wrong, try tools/fromscheme.py doc/schemas/lightning-`command`.json to see how far it got before it died.
|
||||
|
||||
You should always list all fields which are _always_ present in `"required"`.
|
||||
|
||||
@ -57,6 +45,23 @@ To add conditional fields:
|
||||
6. Inside the "then", use `"additionalProperties": false` and place empty `{}` for all the other possible properties.
|
||||
7. If you haven't covered all the possibilties with `if` statements, add an `else` with `"additionalProperties": false` which simply mentions every allowable property. This ensures that the fields can _only_ be present when conditions are met.
|
||||
|
||||
### Exceptions in dynamic schema generation
|
||||
|
||||
- If response (`RETURN VALUE`) should not be generated dynamically and you want it to be a custom text message instead. You can use `return_value_notes` to add custom text with empty `properties`. Examples: `setpsbtversion`, `commando`, `recover`.
|
||||
- If only one of multiple request parameters can be provided then utilize `oneOfMany`
|
||||
key with condition defining arrays. For example, `plugin` command defines it as
|
||||
`"oneOfMany": [["plugin", "directory"]]` and it prints the parameter output as
|
||||
`[*plugin|directory*]`.
|
||||
- If request parameters are paired with other parameter and either all of them can be passed
|
||||
to the command or none of them; then utilize `pairedWith` key with condition defining arrays.
|
||||
For example, `delpay` command defines it as `"pairedWith": [["partid", "groupid"]]`
|
||||
and it prints the parameter output as `[*partid* *groupid*]`.
|
||||
- - If some of the optional request parameters are dependent upon other optional parameters,
|
||||
use `dependentUpon` key where object key can be mapped with the array of dependent params.
|
||||
For example, `listforwards` command has `start` and `limit` params dependent upon `index` and
|
||||
it can be defined as `"dependentUpon": { "index": ["start", "limit"] }` in the json and it will
|
||||
generate the markdown syntax as `[*index* [*start*] [*limit*]]`.
|
||||
|
||||
### JSON Drinking Game!
|
||||
|
||||
1. Sip whenever you have an additional comma at the end of a sequence.
|
||||
|
||||
@ -78,7 +78,7 @@ With the default configurations, the Swagger user interface will be available at
|
||||
The POST method requires `rune` header for authorization.
|
||||
|
||||
- A new `rune` can be created via [createrune](https://docs.corelightning.org/reference/lightning-createrune) or the list of
|
||||
existing runes can be retrieved with [listrunes](https://docs.corelightning.org/reference/lightning-commando-listrunes) command.
|
||||
existing runes can be retrieved with [showrunes](https://docs.corelightning.org/reference/lightning-showrunes) command.
|
||||
|
||||
Note: in version v23.08, a parameter `Nodeid` was required to be the id of the node we're talking to (see `id (pubkey)` received
|
||||
from [getinfo](https://docs.corelightning.org/reference/lightning-getinfo)). You can still send this for backwards compatibility,
|
||||
|
||||
@ -68,8 +68,15 @@ The `getmanifest` method is required for all plugins and will be called on start
|
||||
"disconnect"
|
||||
],
|
||||
"hooks": [
|
||||
{ "name": "openchannel", "before": ["another_plugin"] },
|
||||
{ "name": "htlc_accepted" }
|
||||
{
|
||||
"name": "openchannel",
|
||||
"before": [
|
||||
"another_plugin"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "htlc_accepted"
|
||||
}
|
||||
],
|
||||
"featurebits": {
|
||||
"node": "D0000000",
|
||||
@ -79,11 +86,12 @@ The `getmanifest` method is required for all plugins and will be called on start
|
||||
},
|
||||
"notifications": [
|
||||
{
|
||||
"method": "mycustomnotification"
|
||||
"method": "mycustomnotification"
|
||||
}
|
||||
],
|
||||
"custommessages": [
|
||||
11008, 11010
|
||||
11008,
|
||||
11010
|
||||
],
|
||||
"nonnumericids": true,
|
||||
"dynamic": true
|
||||
@ -129,6 +137,7 @@ Nota bene: if a `flag` type option is not set, it will not appear in the options
|
||||
Here's an example option set, as sent in response to `getmanifest`
|
||||
|
||||
```json
|
||||
{
|
||||
"options": [
|
||||
{
|
||||
"name": "greeting",
|
||||
@ -158,9 +167,10 @@ Here's an example option set, as sent in response to `getmanifest`
|
||||
"type": "int",
|
||||
"default": 0,
|
||||
"description": "Another number to add",
|
||||
"multi": true
|
||||
"multi": true
|
||||
}
|
||||
],
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Custom notifications
|
||||
@ -175,9 +185,9 @@ When forwarding a custom notification `lightningd` will wrap the payload of the
|
||||
"method": "mycustomnotification",
|
||||
"params": {
|
||||
"key": "value",
|
||||
"message": "Hello fellow plugin!"
|
||||
"message": "Hello fellow plugin!"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
is delivered as
|
||||
@ -194,7 +204,6 @@ is delivered as
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
The notification topic (`method` in the JSON-RPC message) must not match one of the internal events in order to prevent breaking subscribers that expect the existing notification format. Multiple plugins are allowed to emit notifications for the same topics, allowing things like metric aggregators where the aggregator subscribes to a common topic and other plugins publish metrics as notifications.
|
||||
@ -207,7 +216,9 @@ The `init` method is required so that `lightningd` can pass back the filled comm
|
||||
{
|
||||
"options": {
|
||||
"greeting": "World",
|
||||
"number": [0]
|
||||
"number": [
|
||||
0
|
||||
]
|
||||
},
|
||||
"configuration": {
|
||||
"lightning-dir": "/home/user/.lightning/testnet",
|
||||
@ -215,15 +226,15 @@ The `init` method is required so that `lightningd` can pass back the filled comm
|
||||
"startup": true,
|
||||
"network": "testnet",
|
||||
"feature_set": {
|
||||
"init": "02aaa2",
|
||||
"node": "8000000002aaa2",
|
||||
"channel": "",
|
||||
"invoice": "028200"
|
||||
"init": "02aaa2",
|
||||
"node": "8000000002aaa2",
|
||||
"channel": "",
|
||||
"invoice": "028200"
|
||||
},
|
||||
"proxy": {
|
||||
"type": "ipv4",
|
||||
"address": "127.0.0.1",
|
||||
"port": 9050
|
||||
"type": "ipv4",
|
||||
"address": "127.0.0.1",
|
||||
"port": 9050
|
||||
},
|
||||
"torv3-enabled": true,
|
||||
"always_use_proxy": false
|
||||
|
||||
@ -177,11 +177,13 @@ A notification for topic `invoice_creation` is sent every time an invoice is cre
|
||||
"invoice_creation": {
|
||||
"label": "unique-label-for-invoice",
|
||||
"preimage": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"amount_msat": 10000
|
||||
"msat": 10000
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Before version `23.11` the `msat` field was a string with msat-suffix, e.g: `"10000msat"`.
|
||||
|
||||
### `warning`
|
||||
|
||||
A notification for topic `warning` is sent every time a new `BROKEN`/`UNUSUAL` level(in plugins, we use `error`/`warn`) log generated, which means an unusual/borken thing happens, such as channel failed, message resolving failed...
|
||||
|
||||
@ -1,92 +0,0 @@
|
||||
# Writing JSON Schemas
|
||||
|
||||
A JSON Schema is a JSON file which defines what a structure should
|
||||
look like; in our case we use it in our testsuite to check that they
|
||||
match command responses, and also use it to generate our
|
||||
documentation.
|
||||
|
||||
Yes, schemas are horrible to write, but they're damn useful. We can only
|
||||
use a subset of the full [JSON Schema Specification](https://json-schema.org/),
|
||||
but if you find that limiting it's probably a sign that you should simplify
|
||||
your JSON output.
|
||||
|
||||
## Updating a Schema
|
||||
|
||||
If you add a field, you should add it to the schema, and you must add
|
||||
"added": "VERSION" (where VERSION is the next release version!).
|
||||
|
||||
Similarly, if you deprecate a field, add "deprecated": "VERSION" (where
|
||||
VERSION is the next release version). They will be removed two versions
|
||||
later.
|
||||
|
||||
## How to Write a Schema
|
||||
|
||||
Name the schema doc/schemas/`command`.schema.json: the testsuite should
|
||||
pick it up and check all invocations of that command against it.
|
||||
|
||||
I recommend copying an existing one to start.
|
||||
|
||||
You will need to put the magic lines in the manual page so `make doc-all`
|
||||
will fill it in for you:
|
||||
|
||||
```
|
||||
[comment]: # (GENERATE-FROM-SCHEMA-START)
|
||||
[comment]: # (GENERATE-FROM-SCHEMA-END)
|
||||
```
|
||||
|
||||
If something goes wrong, try tools/fromscheme.py
|
||||
doc/schemas/`command`.schema.json to see how far it got before it died.
|
||||
|
||||
You should always use `"additionalProperties": false`, otherwise
|
||||
your schema might not be covering everything. Deprecated fields
|
||||
simply have `"deprecated": true` in their properties, so they
|
||||
are allowed by omitted from the documentation.
|
||||
|
||||
You should always list all fields which are *always* present in
|
||||
`"required"`.
|
||||
|
||||
We extend the basic types; see [fixtures.py][fixtures].
|
||||
|
||||
[fixtures]: https://github.com/ElementsProject/lightning/blob/master/contrib/pyln-testing/pyln/testing/fixtures.py
|
||||
|
||||
In addition, before committing a new schema or a new version of it, make sure that it
|
||||
is well formatted. If you don't want do it by hand, use `make fmt-schema` that uses
|
||||
jq under the hood.
|
||||
|
||||
### Using Conditional Fields
|
||||
|
||||
Sometimes one field is only sometimes present; if you can, you should make
|
||||
the schema know when it should (and should not!) be there.
|
||||
|
||||
There are two kinds of conditional fields expressable: fields which
|
||||
are only present if another field is present, or fields only present
|
||||
if another field has certain values.
|
||||
|
||||
To add conditional fields:
|
||||
|
||||
1. Do *not* mention them in the main "properties" section.
|
||||
2. Set `"additionalProperties": true` for the main "properties" section.
|
||||
3. Add an `"allOf": [` array at the same height as `"properties"'`. Inside
|
||||
this place one `if`/`then` for each conditional field.
|
||||
4. If a field simply requires another field to be present, use the pattern
|
||||
`"required": [ "field" ]` inside the "if".
|
||||
5. If a field requires another field value, use the pattern
|
||||
`"properties": { "field": { "enum": [ "val1", "val2" ] } }` inside
|
||||
the "if".
|
||||
6. Inside the "then", use `"additionalProperties": false` and place
|
||||
empty `{}` for all the other possible properties.
|
||||
7. If you haven't covered all the possibilties with `if` statements,
|
||||
add an `else` with `"additionalProperties": false` which simply
|
||||
mentions every allowable property. This ensures that the fields
|
||||
can *only* be present when conditions are met.
|
||||
|
||||
### JSON Drinking Game!
|
||||
|
||||
1. Sip whenever you have an additional comma at the end of a sequence.
|
||||
2. Sip whenever you omit a comma in a sequence because you cut & paste.
|
||||
3. Skull whenever you wish JSON had comments.
|
||||
|
||||
Good luck!
|
||||
Rusty.
|
||||
|
||||
[contrib/pyln-testing/pyln/testing/fixtures.py]: https://github.com/ElementsProject/lightning/tree/master/contrib/pyln-testing/pyln/testing/fixtures.py
|
||||
Loading…
Reference in New Issue
Block a user