Protobuffer migration

classes sent over the wire (P2P network)
classes serialized to disk

If possible we'll start with the P2P network because this has wider backward compatibility impact.

Frameworks

Drop-in replacement for serialization, fixes size concerns but not backward compat: https://ruedigermoeller.github.io/fast-serialization/

Protobuffer Setup

installing

Install the latest protobuffer release on your machine (3.1.0 at the time of writing): https://github.com/google/protobuf/releases
Increase the Intellij Idea Code insight limit, because it breaks on the generated protobuffer files: Go to Help > Edit custom properties => paste the following line: idea.max.intellisense.filesize=12500 Source: https://stackoverflow.com/questions/23057988/file-size-exceeds-configured-limit-2560000-code-insight-features-not-availabl

maven plugin vs ant plugin

Bitcoinj uses an ant plugin to call the 'protoc' executable. This was tried but didn't work, although command-line invocation with the same params worked.
There is also a protobuf maven plugin This worked immediately + is executed automatically when doing 'mvn clean install'. Output is in target/generated-sources which avoids the temptation of checking in generated classes.

multiple different messages in one stream

In order to support this, we need to use .writeDelimitedTo(outputstream) and parseDelimitedFrom(inputstream). The writeDelimited writes a length varint before the object, allowing the parseDelimited to know the extent of the message.

P2P Network

Extends Payload search results

public final class PubKeyRing implements Payload {
public final class SealedAndSigned implements Payload {
public final class PrivateNotification implements Payload {
public final class Dispute implements Payload {
public final class DisputeResult implements Payload {
public final class Attachment implements Payload {
public final class RawTransactionInput implements Payload {
public abstract class PaymentAccountContractData implements Payload {
public final class Contract implements Payload {
public final class NodeAddress implements Persistable, Payload {
public final class Peer implements Payload, Persistable {
public interface CapabilityRequiringPayload extends Payload {
public interface ExpirablePayload extends Payload {
public interface RequiresOwnerIsOnlinePayload extends Payload {
public class ProtectedStorageEntry implements Payload {

Messages

public interface DirectMessage extends Message {
public interface SupportedCapabilitiesMessage extends Message {
public final class MockPayload implements Message, ExpirablePayload {
public interface AnonymousMessage extends Message {
public final class CloseConnectionMessage implements Message {
public interface SendersNodeAddressMessage extends Message {
public interface GetDataRequest extends Message {
public abstract class KeepAliveMessage implements Message {
abstract class PeerExchangeMessage implements Message {
public abstract class BroadcastMessage implements Message {

Disk serialization

extends Serializable

private static class MockMessage implements Serializable {
public interface Persistable extends Serializable {
public class Tuple2<A, B> implements Serializable {
public class Tuple3<A, B, C> implements Serializable {
public class Tuple4<A, B, C, D> implements Serializable {
public static <T extends Serializable> T deserialize(byte[] data) {
public interface Payload extends Serializable {
public class PlainTextWrapper implements Serializable {
public class Storage<T extends Serializable> {
public abstract class HttpClientProvider implements Serializable {
public class PaymentAccountFilter implements Serializable {
public class CurrencyTuple implements Serializable {
public class ProcessModel implements Model, Serializable {
public final class Altcoin implements Monetary, Comparable<Altcoin>, Serializable {
public class AltcoinExchangeRate implements Serializable {
public interface Message extends Serializable {
public interface Message extends Serializable {
public static final class DataAndSeqNrPair implements Serializable {

Actually transformed subtypes of Message

I Message (io.bitsquare.p2p)
I AnonymousMessage (io.bitsquare.p2p.network.messages)
+ PreliminaryGetDataRequest (io.bitsquare.p2p.peers.getdata.messages)
C BroadcastMessage (io.bitsquare.p2p.storage.messages)
+ AddDataMessage (io.bitsquare.p2p.storage.messages)
+ RefreshTTLMessage (io.bitsquare.p2p.storage.messages)
+ RemoveDataMessage (io.bitsquare.p2p.storage.messages)
+ RemoveMailboxDataMessage (io.bitsquare.p2p.storage.messages)
+ CloseConnectionMessage (io.bitsquare.p2p.network.messages)
I DirectMessage (io.bitsquare.p2p.messaging)
I MailboxMessage (io.bitsquare.p2p.messaging)
+ DepositTxPublishedMessage (io.bitsquare.trade.protocol.trade.messages)
C DisputeMessage (io.bitsquare.arbitration.messages)
+ DisputeCommunicationMessage (io.bitsquare.arbitration.messages)
+ DisputeResultMessage (io.bitsquare.arbitration.messages)
+ OpenNewDisputeMessage (io.bitsquare.arbitration.messages)
+ PeerOpenedDisputeMessage (io.bitsquare.arbitration.messages)
+ PeerPublishedPayoutTxMessage (io.bitsquare.arbitration.messages)
+ FiatTransferStartedMessage (io.bitsquare.trade.protocol.trade.messages)
+ FinalizePayoutTxRequest (io.bitsquare.trade.protocol.trade.messages)
+ MockMailboxPayload (io.bitsquare.p2p.mocks)
+ PayDepositRequest (io.bitsquare.trade.protocol.trade.messages)
+ PayoutTxFinalizedMessage (io.bitsquare.trade.protocol.trade.messages)
+ PrefixedSealedAndSignedMessage (io.bitsquare.p2p.messaging)
+ PrivateNotificationMessage (io.bitsquare.alert)
T StressTestMailboxMessage (io.bitsquare.p2p.network)
T TestMessage (io.bitsquare.crypto)
C OfferMessage (io.bitsquare.trade.protocol.availability.messages)
+ OfferAvailabilityRequest (io.bitsquare.trade.protocol.availability.messages)
+ OfferAvailabilityResponse (io.bitsquare.trade.protocol.availability.messages)
T StressTestDirectMessage (io.bitsquare.p2p.network)
C TradeMessage (io.bitsquare.trade.protocol.trade.messages)
+ DepositTxPublishedMessage (io.bitsquare.trade.protocol.trade.messages)
D FiatTransferStartedMessage (io.bitsquare.trade.protocol.trade.messages)
D FinalizePayoutTxRequest (io.bitsquare.trade.protocol.trade.messages)
D PayDepositRequest (io.bitsquare.trade.protocol.trade.messages)
D PayoutTxFinalizedMessage (io.bitsquare.trade.protocol.trade.messages)
+ PublishDepositTxRequest (io.bitsquare.trade.protocol.trade.messages)
I GetDataRequest (io.bitsquare.p2p.peers.getdata.messages)
+ GetUpdatedDataRequest (io.bitsquare.p2p.peers.getdata.messages)
D PreliminaryGetDataRequest (io.bitsquare.p2p.peers.getdata.messages)
C KeepAliveMessage (io.bitsquare.p2p.peers.keepalive.messages)
+ Ping (io.bitsquare.p2p.peers.keepalive.messages)
+ Pong (io.bitsquare.p2p.peers.keepalive.messages)
T MockPayload (io.bitsquare.p2p.mocks)
C PeerExchangeMessage (io.bitsquare.p2p.peers.peerexchange.messages)
+ GetPeersRequest (io.bitsquare.p2p.peers.peerexchange.messages)
+ GetPeersResponse (io.bitsquare.p2p.peers.peerexchange.messages)
I SendersNodeAddressMessage (io.bitsquare.p2p.network.messages)
D GetPeersRequest (io.bitsquare.p2p.peers.peerexchange.messages)
D GetUpdatedDataRequest (io.bitsquare.p2p.peers.getdata.messages)
D PrefixedSealedAndSignedMessage (io.bitsquare.p2p.messaging)
I SupportedCapabilitiesMessage (io.bitsquare.p2p.messaging)
+ GetDataResponse (io.bitsquare.p2p.peers.getdata.messages)
D GetPeersRequest (io.bitsquare.p2p.peers.peerexchange.messages)
D GetPeersResponse (io.bitsquare.p2p.peers.peerexchange.messages)
D OfferAvailabilityRequest (io.bitsquare.trade.protocol.availability.messages)
D OfferAvailabilityResponse (io.bitsquare.trade.protocol.availability.messages)
D PreliminaryGetDataRequest (io.bitsquare.p2p.peers.getdata.messages)

== Actually transformed subtypes of Payload

+ Attachment (io.bitsquare.arbitration.payload)
I CapabilityRequiringPayload (io.bitsquare.p2p.storage.payload)
+     TradeStatistics (io.bitsquare.trade.statistics)
+ Contract (io.bitsquare.trade)
+ Dispute (io.bitsquare.arbitration)
+ DisputeResult (io.bitsquare.arbitration)
I ExpirablePayload (io.bitsquare.p2p.storage.payload)
I     StoragePayload (io.bitsquare.p2p.storage.payload)
+        Alert (io.bitsquare.alert)
+        Arbitrator (io.bitsquare.arbitration)
+        Filter (io.bitsquare.filter)
I        LazyProcessedStoragePayload (io.bitsquare.p2p.storage.payload)
+            CompensationRequestPayload (io.bitsquare.dao.compensation)
D            TradeStatistics (io.bitsquare.trade.statistics)
+        MailboxStoragePayload (io.bitsquare.p2p.storage.payload)
+        Offer (io.bitsquare.trade.offer)
I        PersistedStoragePayload (io.bitsquare.p2p.storage.payload)
D            CompensationRequestPayload (io.bitsquare.dao.compensation)
D            TradeStatistics (io.bitsquare.trade.statistics)
+ NodeAddress (io.bitsquare.p2p)
C+PaymentAccountContractData (io.bitsquare.payment)
+    AliPayAccountContractData (io.bitsquare.payment)
+    ChaseQuickPayAccountContractData (io.bitsquare.payment)
+    ClearXchangeAccountContractData (io.bitsquare.payment)
C+   CountryBasedPaymentAccountContractData (io.bitsquare.payment)
C+       BankAccountContractData (io.bitsquare.payment)
+            NationalBankAccountContractData (io.bitsquare.payment)
+            SameBankAccountContractData (io.bitsquare.payment)
+            SpecificBanksAccountContractData (io.bitsquare.payment)
+        CashDepositAccountContractData (io.bitsquare.payment)
+        SepaAccountContractData (io.bitsquare.payment)
+    CryptoCurrencyAccountContractData (io.bitsquare.payment)
+    FasterPaymentsAccountContractData (io.bitsquare.payment)
+    InteracETransferAccountContractData (io.bitsquare.payment)
+    OKPayAccountContractData (io.bitsquare.payment)
+    PerfectMoneyAccountContractData (io.bitsquare.payment)
+    SwishAccountContractData (io.bitsquare.payment)
+    USPostalMoneyOrderAccountContractData (io.bitsquare.payment)
+ Peer (io.bitsquare.p2p.peers.peerexchange)
+ PrivateNotification (io.bitsquare.alert)
+ ProtectedStorageEntry (io.bitsquare.p2p.storage.storageentry)
+    ProtectedMailboxStorageEntry (io.bitsquare.p2p.storage.storageentry)
+ PubKeyRing (io.bitsquare.common.crypto)
+ RawTransactionInput (io.bitsquare.btc.data)
I RequiresOwnerIsOnlinePayload (io.bitsquare.p2p.storage.payload)
D    Offer (io.bitsquare.trade.offer)
+ SealedAndSigned (io.bitsquare.common.crypto)

10 KiB Raw Blame History