mirrors/bisq

Fork 0

mirror of https://github.com/bisq-network/bisq.git synced 2024-11-19 09:52:23 +01:00

Manfred Karrer f268abe2e1 Merge DAO branch to Development branch

2017-06-27 01:30:24 +02:00

13 KiB

Raw Blame History

Protobuffer FAQ

Why protobuffer?

There are a number of reasons why protobuffer was chosen, here are some of them:

avoids java serialisation security issues
smaller in size than java serialisation (less network usage)
All P2P network messages are described in a clear protobuffer schema
allows to evolve your schema in a backward compatible way
can generate code in many languages, making alternative bisq clients or monitoring tools easier

Which classes are transformed to protobuffer?

All classes in the 'wire' module. This module contains the following classes:

classes sent over the wire (P2P network)
classes serialized to disk

The protobuffer schema file(s), generated classes and domain classes are in the 'wire' module. Look for *.proto for the schema files.

How is serialisation done (Java -> Protobuffer)

Some interfaces have a 'toProtobuf' method to force all extending classes to implement that method.

How is deserialisation done (Java -> Protobuffer)

Some interfaces have a 'toProtobuf' method to force all extending classes to implement that method.

If fields are not filled in, what are protobuffer's default values?

Read this very carefully:

https://developers.google.com/protocol-buffers/docs/proto3#default

How to handle Enums

For Java -> Protobuffer, you should extract the name from the Java enum:

.setContext(PB.AddressEntry.Context.valueOf(context.name()))

For Protobuffer -> Java, use the ProtoUtil helper method 'enumFromProto' to avoid crashes:

ProtoUtil.enumFromProto(OfferPayload.Direction.class, direction.name());

How to handle Date

For Java -> Protobuffer, you should extract the name from the Java enum:

.setDate(date.getTime())

For Protobuffer -> Java, do the opposite:

Date date = new Date(PB.bla.getDate());

Other Stuff

Frameworks

Drop-in replacement for serialization, fixes size concerns but not backward compat: https://ruedigermoeller.github.io/fast-serialization/

Protobuffer Setup

installing

Install the latest protobuffer release on your machine (3.3.0 at the time of writing): https://github.com/google/protobuf/releases
Increase the Intellij Idea Code insight limit, because it breaks on the generated protobuffer files: Go to Help > Edit custom properties => paste the following line: idea.max.intellisense.filesize=12500 Source: https://stackoverflow.com/questions/23057988/file-size-exceeds-configured-limit-2560000-code-insight-features-not-availabl

In IntelliJ 14 you need to edit the idea.properties in the app container: /Applications/IntelliJ\ IDEA\ 14\ CE.app/Contents/bin/idea.properties

maven plugin vs ant plugin

Bitcoinj uses an ant plugin to call the 'protoc' executable. This was tried but didn't work, although command-line invocation with the same params worked.
There is also a protobuf maven plugin This worked immediately + is executed automatically when doing 'mvn clean install'. Output is in target/generated-sources which avoids the temptation of checking in generated classes.

multiple different messages in one stream

In order to support this, we need to use .writeDelimitedTo(outputstream) and parseDelimitedFrom(inputstream). The writeDelimited writes a length varint before the object, allowing the parseDelimited to know the extent of the message.

Checklist and conventions

Code style

Use line breaks after each param in constructor and PB methods
Use line breaks at PB builders for best readability
Use after the constructor a comment separator with the PB stuff and after the PB stuff a comment separator with API
Use same order of fields as in main constructor and follow that order in the PB methods to make it easier to spot missing fields

Conventions

Try to use value objects for data which gets serialized with PB
Use toProto and a static fromProto in the class which gets serialized
Use proto as name for the PB param in the fromProto method
If a constructor is used only for PB make it private and put it to the PB section, so its clear it is used only in that context
Use same name for classes and fields in PB definition as in the java code base
Use final
Use Lombok Annotations
Annotate all nullable fields with @Nullable
If nullable fields must not be null at time of serialisation use checkNotNull with comment inside
Use ProtoUtil convenience methods (e.g. stringOrNullFromProto, byteArrayOrNullFromProto, collectionToProto, enumFromProto,..)
Use ProtoUtil.enumFromProto for all enums
Enum in PB definition file needs an additional first entry with PB_ERROR in case of multiple enums in one message add postfix of enum (PB_ERROR_REASON = 0;)
When serializing a custom type use the toProto and fromProto methods in the class of that type.
Use the ProtoResolver classes for switching between different message cases. Pass the reference to the resolver if needed in fromProto
For abstract super classes use a getBuilder method for handling the fields in that super class (e.g. PaymentAccountPayload)
Use Payload as postfix for objects which are used in Envelopes or other Payloads if it helps to distinguish between the domain object and the value object (e.g. UserPayload)
Separate network and persistable domains if possible

Architecture

Envelope is the base interface for all objects carrying PB data (messages, persisted objects)
Payload is used inside Envelopes or other Payloads
Interface structure: Proto: base has Message toProtoMessage(); Envelope extends Proto: Marker interface for objects carrying PB data NetworkEnvelope extends Envelope: Marker interface, has getDefaultBuilder for P2PMessageVersion PersistableEnvelope extends Envelope: Marker interface Payload extends Proto: Marker interface for objects used inside other Payload or Envelope objects NetworkPayload extends Payload: Marker interface PersistablePayload extends Payload: Marker interface ProtoResolver: Base for resolver (switch message cases) NetworkProtoResolver extends ProtoResolver: Marker interface CoreNetworkProtoResolver implements NetworkProtoResolver: Implements switches for network messages PersistableProtoResolver extends ProtoResolver: Marker interface CorePersistableProtoResolver implements NetworkProtoResolver: Implements switches for persistable messages

Check list

Treat nullable fields correctly in the toProto and fromProto methods. PB does not support null values but use default implementation for not set fields. Depending on the type there is: isEmpty (string, collections) or has[Propertyname]
If using @EqualsAndHashCode or @Data/@Value make sure to use callSuper=true if the class is extending another class
If collections are modifiable take care to wrap the result of PB to a modifiable collection. PB delivers unmodifiable collections
For network envelopes use NetworkEnvelope.getDefaultBuilder() which includes the P2PMessageVersion. Store the messageVersion in all NetworkEnvelope instances

// TODO update, outdated...

Actually transformed subtypes of Message

I Message (io.bisq.p2p)
I AnonymousMessage (io.bisq.p2p.network.messages)
+ PreliminaryGetDataRequest (io.bisq.p2p.peers.getdata.messages)
C BroadcastMessage (io.bisq.p2p.storage.messages)
+ AddDataMessage (io.bisq.p2p.storage.messages)
+ RefreshTTLMessage (io.bisq.p2p.storage.messages)
+ RemoveDataMessage (io.bisq.p2p.storage.messages)
+ RemoveMailboxDataMessage (io.bisq.p2p.storage.messages)
+ CloseConnectionMessage (io.bisq.p2p.network.messages)
I DirectMessage (io.bisq.p2p.messaging)
I MailboxMessage (io.bisq.p2p.messaging)
+ DepositTxPublishedMessage (io.bisq.trade.protocol.trade.messages)
C DisputeMessage (io.bisq.arbitration.messages)
+ DisputeCommunicationMessage (io.bisq.arbitration.messages)
+ DisputeResultMessage (io.bisq.arbitration.messages)
+ OpenNewDisputeMessage (io.bisq.arbitration.messages)
+ PeerOpenedDisputeMessage (io.bisq.arbitration.messages)
+ PeerPublishedPayoutTxMessage (io.bisq.arbitration.messages)
+ FiatTransferStartedMessage (io.bisq.trade.protocol.trade.messages)
+ FinalizePayoutTxRequest (io.bisq.trade.protocol.trade.messages)
+ MockMailboxPayload (io.bisq.p2p.mocks)
+ PayDepositRequest (io.bisq.trade.protocol.trade.messages)
+ PayoutTxFinalizedMessage (io.bisq.trade.protocol.trade.messages)
+ PrefixedSealedAndSignedMessage (io.bisq.p2p.messaging)
+ PrivateNotificationMessage (io.bisq.alert)
T StressTestMailboxMessage (io.bisq.p2p.network)
T TestMessage (io.bisq.crypto)
C OfferMessage (io.bisq.trade.protocol.availability.messages)
+ OfferAvailabilityRequest (io.bisq.trade.protocol.availability.messages)
+ OfferAvailabilityResponse (io.bisq.trade.protocol.availability.messages)
T StressTestDirectMessage (io.bisq.p2p.network)
C TradeMessage (io.bisq.trade.protocol.trade.messages)
+ DepositTxPublishedMessage (io.bisq.trade.protocol.trade.messages)
D FiatTransferStartedMessage (io.bisq.trade.protocol.trade.messages)
D FinalizePayoutTxRequest (io.bisq.trade.protocol.trade.messages)
D PayDepositRequest (io.bisq.trade.protocol.trade.messages)
D PayoutTxFinalizedMessage (io.bisq.trade.protocol.trade.messages)
+ PublishDepositTxRequest (io.bisq.trade.protocol.trade.messages)
I GetDataRequest (io.bisq.p2p.peers.getdata.messages)
+ GetUpdatedDataRequest (io.bisq.p2p.peers.getdata.messages)
D PreliminaryGetDataRequest (io.bisq.p2p.peers.getdata.messages)
C KeepAliveMessage (io.bisq.p2p.peers.keepalive.messages)
+ Ping (io.bisq.p2p.peers.keepalive.messages)
+ Pong (io.bisq.p2p.peers.keepalive.messages)
T MockPayload (io.bisq.p2p.mocks)
C PeerExchangeMessage (io.bisq.p2p.peers.peerexchange.messages)
+ GetPeersRequest (io.bisq.p2p.peers.peerexchange.messages)
+ GetPeersResponse (io.bisq.p2p.peers.peerexchange.messages)
I SendersNodeAddressMessage (io.bisq.p2p.network.messages)
D GetPeersRequest (io.bisq.p2p.peers.peerexchange.messages)
D GetUpdatedDataRequest (io.bisq.p2p.peers.getdata.messages)
D PrefixedSealedAndSignedMessage (io.bisq.p2p.messaging)
I SupportedCapabilitiesMessage (io.bisq.p2p.messaging)
+ GetDataResponse (io.bisq.p2p.peers.getdata.messages)
D GetPeersRequest (io.bisq.p2p.peers.peerexchange.messages)
D GetPeersResponse (io.bisq.p2p.peers.peerexchange.messages)
D OfferAvailabilityRequest (io.bisq.trade.protocol.availability.messages)
D OfferAvailabilityResponse (io.bisq.trade.protocol.availability.messages)
D PreliminaryGetDataRequest (io.bisq.p2p.peers.getdata.messages)

== Actually transformed subtypes of Payload

+ Attachment (io.bisq.arbitration.payload)
I CapabilityRequiringPayload (io.bisq.p2p.storage.payload)
+     TradeStatistics (io.bisq.trade.statistics)
+ Contract (io.bisq.trade)
+ Dispute (io.bisq.arbitration)
+ DisputeResult (io.bisq.arbitration)
I ExpirablePayload (io.bisq.p2p.storage.payload)
I     StoragePayload (io.bisq.p2p.storage.payload)
+        Alert (io.bisq.alert)
+        Arbitrator (io.bisq.arbitration)
+        Filter (io.bisq.filter)
I        LazyProcessedStoragePayload (io.bisq.p2p.storage.payload)
+            CompensationRequestPayload (io.bisq.dao.compensation)
D            TradeStatistics (io.bisq.trade.statistics)
+        MailboxStoragePayload (io.bisq.p2p.storage.payload)
+        Offer (io.bisq.trade.offerPayload)
I        PersistedStoragePayload (io.bisq.p2p.storage.payload)
D            CompensationRequestPayload (io.bisq.dao.compensation)
D            TradeStatistics (io.bisq.trade.statistics)
+ NodeAddress (io.bisq.p2p)
C+PaymentAccountContractData (io.bisq.payment)
+    AliPayAccountContractData (io.bisq.payment)
+    ChaseQuickPayAccountContractData (io.bisq.payment)
+    ClearXchangeAccountContractData (io.bisq.payment)
C+   CountryBasedPaymentAccountContractData (io.bisq.payment)
C+       BankAccountContractData (io.bisq.payment)
+            NationalBankAccountContractData (io.bisq.payment)
+            SameBankAccountContractData (io.bisq.payment)
+            SpecificBanksAccountContractData (io.bisq.payment)
+        CashDepositAccountContractData (io.bisq.payment)
+        SepaAccountContractData (io.bisq.payment)
+    CryptoCurrencyAccountContractData (io.bisq.payment)
+    FasterPaymentsAccountContractData (io.bisq.payment)
+    InteracETransferAccountContractData (io.bisq.payment)
+    OKPayAccountContractData (io.bisq.payment)
+    PerfectMoneyAccountContractData (io.bisq.payment)
+    SwishAccountContractData (io.bisq.payment)
+    USPostalMoneyOrderAccountContractData (io.bisq.payment)
+ Peer (io.bisq.p2p.peers.peerexchange)
+ PrivateNotification (io.bisq.alert)
+ ProtectedStorageEntry (io.bisq.p2p.storage.storageentry)
+    ProtectedMailboxStorageEntry (io.bisq.p2p.storage.storageentry)
+ PubKeyRing (io.bisq.common.crypto)
+ RawTransactionInput (io.bisq.btc.data)
I RequiresOwnerIsOnlinePayload (io.bisq.p2p.storage.payload)
D    Offer (io.bisq.trade.offerPayload)
+ SealedAndSigned (io.bisq.common.crypto)

13 KiB Raw Blame History